Abstract

A method and apparatus of converting input text into an audio-visual speech stream resulting in a talking face image enunciating the text. This method of converting input text into an audio-visual speech stream comprises the steps of: recording a visual corpus of a human-subject, building a viseme interpolation database, and synchronizing the talking face image with the text stream. In a preferred embodiment, viseme transitions are automatically calculated using optical flow methods, and morphing techniques are employed to result in smooth viseme transitions. The viseme transitions are concatenated together and synchronized with the phonemes according to the timing information. The audio-visual speech stream is then displayed in real time, thereby displaying a photo-realistic talking face.

Other References

• T. Beier and S. Neely, "Feature-Based Image Metamorphosis", Siggraph '92 Proceedings, p. 35-42, Chicago, 1992
• JR Bergen and R. Hingorami, "Hierarchial Motion-Based Frame Rate Conversion", Technical Report, David Sarnoff Research Center,1990
• D. Beymer, et al., "Example Based Image Analysis and Synthesis", Technical Report 1431, MIT AT Lab, 1993
• A. Black and P. Taylor, The Festival Speech Sysnthesis System,University of Edinburgh, 1997
• C. Bregler, et al., "Video Rewrite: Driving Visual Speech with Audio", SIGGRAPH '97 Proceedings, Los Angeles, 1997,
• M.M. Cohen and D. W. Massaro, "Modeling Coarticulation in Synthetic Visual Speech", Models and Techniques in Computer Animation, p. 139-156, Springer-Verlag, Tokyo, 1993
• T. Ezzat, "Example-Based Analysis and Synthesis for Images of Human Faces", Master's Thesis, Massachusetts Institute of Technology, 1996
• B.K.P. Horn and B.G. Schnuck, "Determining Optical Flow", Artifical Intelligence, 17:185-203, 1981
• M.J. Hunt, et al., "Issues in High Quality LPC Analysis and Synthesis", Proceedings of Eurospeech '89 , vol. 2, p. 348-351, Paris, 1989
• B. LeGoff and C. Benoit, "A Text-to-Audiovisual-speech Synthesizer for French", Proceedings of the International Conference on Spoken Language Processing, Philadelphia, 1996
• J. Olive, et al., "Acoustics of American English Speech: A Dynamic Approach". Springer-Verlag, p. 20; New York, 1993
• K.C. Scott, et al., "Synthesis of Speaker Facial Movement to Match Selected Speech Sequences", Proceedings of the Fifth Australian Conference on Speech Science and Technology, 2:620-625, 1994
• S.H. Watson, et al., "An Advanced Morphing Algorithm for interpolating Phoneme Images to Simulate Speech", Unpublished Technical Report, Jet Propulsion Laboratory, California Institute of Technology
• S.E. Chen and L. Williams, "View Interpolation for Image Synthesis", Siggraph '93 Proceedings, p. 279-288, Anaheim, California, 1993
• Y. Lee, at al., "Realistic Modeling for Facial Animation", Siggraph '95 Proceedings, p. 5-62, Los Angelos, 1995
• A. Montgomery and P. Jackson, "Physical Characteristics of the Lips Underlying Vowel Lipreading Performance", Journ. Acoustical Society of America, 73(6):2134-2144, 1983
• E. Owens and B.Blazek, "Visemens Observed by Hearing-Impaired and Normal-Hearing Adult Viewers", Journ. Speech and Hearing Research, 28:381-383, 1985
• D. Terzopoulos and K. Waters, "Analysis and Synthesis of Facial Image Sequences using Physical and Anatomical Models", IEEE Transactions on Pattern Analysis and Machine Intelligence, 15(5):569-579, 1993
• C.G. Fisher, Confusions Among Visually Perceived Consonants, Journ. Speech and Hearing Research, 11:796-804, 1968
• F.I. Parke, "A Parametric Model of Human Faces", Ph.D theisis, University of Utah, 197