Video-teleconferencing system with eye-gaze correction
Patent 6771303 Issued on August 3, 2004. Estimated Expiration Date: April 23, 2022. Estimated Expiration Date is calculated based on simple USPTO term provisions. It does not account for terminal disclaimers, term adjustments, failure to pay maintenance fees, or other factors which might affect the term of a patent.
348/14.16, User positioning (e.g., parallax)348/14.08, Conferencing (e.g., loop)382/118, Using a facial characteristic348/14.06, Answering machine348/14.04, Operating with other appliance (e.g., TV, VCR, FAX, etc.)348/14.05, Remote control348/14.03, User interface (e.g., touch screen menu)348/14.07, Display arrangement (e.g., multiscreen display)348/14.09, Conferencing with multipoint control unit348/14.02, Over wireless communication348/14.01, TWO-WAY VIDEO AND VOICE COMMUNICATION (E.G., VIDEOPHONE)348/14.1, Motion image conferencing348/14.11, Switching348/14.12, Transmission control (e.g., resolution or quality)348/14.13, Compression or decompression382/103, Target tracking or detecting382/117Using a characteristic of the eye
Correcting for eye-gaze in video communication devices is accomplished by blending information captured from a stereoscopic view of the conferee and generating a virtual image of the conferee. A personalized face model of the conferee is captured to track head position of the conferee. First and second video images representative of a first conferee taken from different views are concurrently captured. A head position of the first conferee is tracked from the first and second video images. Matching features and contours from the first and second video images are ascertained. The head position as well as the matching features and contours from the first and second video images are synthesized to generate a virtual image video stream of the first conferee that makes the first conferee appear to be making eye contact with a second conferee who is watching the virtual image video stream.
Other References
U.S. patent application Ser. No. 09/528,827, Microsoft Corporation, filed Mar. 20, 2000.
Sumit Basu, Irfan Essa, Alex Pentland, “Motion Regularization for Model-Based Head Tracking.” In Proceedings of International Conference on Pattern Recognition, Wien, Austria, 1996 IEEE, pp. 611-616.
Michael J. Black, Yaser Yacoob, “Tracking and Recognizing Rigid and Non-Rigid Facial Motions using Local Parametric Models of Image Motion.” In Proceedings of International Conference on Computer Vision, pp. 374-381, Cambridge, MA, 1995 IEEE.
C. Choi, K. Aizawa, H. Harashima & T. Takebe, “Analysis and Synthesis of Facial Image Sequences in Model-Based Image Coding,” IEEE Circuits and Systems for Video Technology, vol. 4, No. 3, Jun. 1994, pp. 257-275.
T. Darrell, B. Moghaddam & A. Pentland, “Active Face Tracking and Pose Estimation in an Interactive Room.” In IEEE Computer Vision and Pattern Recognition, pp. 67-72, 1996.
D. Decarlo, D. Metaxas, “Optical Flow Constraints on Deformable Models with Applications to Face Tracking,” International Journal of Computer Vision 38(2), 99-127, 2000.
T. Horprasert, Y. Yacoob & L. S. Davis, “Computing 3-D Head Orientation from a Monocular Image Sequence.” In International Conference Automatic Face and Gesture Recognition, pp. 242-247, 1996.
Z. Liu, Z. Zhang, C. Jacobs, M. Cohen, “Rapid Modeling of Animated Faces From Video.” In the Third International Conference on Visual Comupting (Visual 2000), pp. 58-67, Mexico City, Sep. 2000. Also available as Technical Report MSR-TR-99-21.
R. Newman, Y. Matsumoto, S. Rougeaux & A. Zelinsky, “Real-Time Stereo Tracking for Head Pose and Gaze Estimation.” In Proceedings of the Fourth IEEE International Conference on Automatic Face and Gesture Recognition (FG 2000), pp. 122-128, Grenoble, France, 2000.
J. Shi & C. Tomasi, “Good Features to Track.” In the IEEE Conf. on Computer Vision and Pattern Recognition, pp. 593-600, Washington, Jun. 1994.
H. Li, P. Roivainen, & R. Forchheimer, “3-D Motion Estimation in Model-Based Facial Image Coding,” IEEE Pattern Analysis and Machine Intelligence, 15(6):545-555, Jun. 1993.
Chen, E. and Williams, L., “View Interpolation for Image Synthesis,” in Siggraph, 1993, pp. 1-7.
Cox, Ingemar J. et al., “A Maximum Likelihood Stereo Algorithm,” Computer Vision and Image Understanding, 63:3, May 1996, pp. 542-567 (1-47).
Gemmell, Jim et al., “Gaze Awareness for Video-conferencing: A Software Approach,” IEEE MultiMedia, Oct.-Dec., 2000, pp. 26-35.
Kolmogorov, Vladimir et al., “Multi-camera Scene Reconstruction via Graph Cuts, ” In Proc. Europ. Conf. Computer Vision, Copenhagen, Denmark, May 2002, pp. 1-16.
Ohta, Yuichi et al., “Stereo by Intra-and Inter-Scanline Search Using Dynamic Programming,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. PAMI-7, No. 2, Mar. 1985, pp. 139-154.
Roy, Sebastien et al., “A Maximum-Flow Formulation of the N-camera Stereo Correspondence Problem,” IEEE Proc. of Int. Conference on Computer Vision, Bombai, Jan. 1998, pp. 492-499.
Scharstein, Daniel et al., “A Taxonomy and Evaluation of Dense Two-Frame Stereo Correspondence Algorithms,” Int. J. Computer Vision, 47(1-3); 7-42, 2002, (10 pages).
Vetter, Thomas, “Synthesis of novel views from a single face image,” Max-Planck-Institut, Germany, Technical Report No. 26, Feb. 1996, pp. 1-13.
Sun, Jian et al., “Stereo Matching Using Belief Propagation,”A. Heyden et al. (Eds): ECCV 2002, LNCS 2351, pp. 510-524, 2002.
Belhumeur, Peter N. et al., “A Bayesian Treatment of the Stereo Correspondence Problem Using Half-Occluded Regions,” in IEEE Conf. On Computer Vision and Pattern Recognition, 1992, 8 pages.
Ishikawa, Hiroshi et al., “Occlusions, Discontinuities, and Epipolar Lines in Stereo, ” in the Fifth European Conference on Computer Vision (ECCV '98), 2-6 Jun. 1998, Freiburg, Germany, pp. 1-14.
Szeliski, Richard, “Prediction Error as a Quality Metric for Motion and Stereo,” Vision Technology Group, Microsoft Research, Sep. 20-25, 1999, Proceeding of the Int'l. Conf. on Computer Vision, vol. 2, Corfu, Greece, 8 pages.
Bobick, Aaron F. et al., “Large Occlusion Stereo,” HCV 1999, pp. 1-25.
Loop, Charles et al., “Computing Rectifying Homographies for Stereo Vision,” Apr. 8, 1999, Technical Report MSR-TR-99-21, 14 pages.
Zhang, Zhengyou et al., “Robust and Rapid Generation of Animated Faces from Video Images: A Model-Based Modeling Approach,” Technical Report, MSR-TR-2001-101, 32 pages.
LU, Le et al., “Model-and Exemplar-based Robust Head Pose Tracking Under Occlusion and Varying Expression,” Microsoft Research China, pp. 1-8.
April 23, 2022. Estimated Expiration Date is calculated based on simple USPTO term provisions. It does not account for terminal disclaimers, term adjustments, failure to pay maintenance fees, or other factors which might affect the term of a patent.
