Automated synchronization of video image sequences to new soundtracks

Patent 5880788 Issued on March 9, 1999. Estimated Expiration Date:

March 25, 2016. Estimated Expiration Date is calculated based on simple USPTO term provisions. It does not account for terminal disclaimers, term adjustments, failure to pay maintenance fees, or other factors which might affect the term of a patent.

Abstract Claims Description Full Text

Patent References

Creating visual images of lip movements
Patent #: 4260229
Issued on: 04/07/1981
Inventor: Bloomstein

Cinematic works with altered facial displays
Patent #: 4827532
Issued on: 05/02/1989
Inventor: Bloomstein

Image encoding and synthesis
Patent #: 4841575
Issued on: 06/20/1989
Inventor: Welsh , et al.

Apparatus and method for lip-synching animation
Patent #: 4913539
Issued on: 04/03/1990
Inventor: Lewis

Method and apparatus for generating animated images
Patent #: 5025394
Issued on: 06/18/1991
Inventor: Parke

Audiovisual synchronization signal generator using audio signature comparison
Patent #: 5040081
Issued on: 08/13/1991
Inventor: McCutchen

Method and means for computer sychronization of actions and sounds
Patent #: 5430835
Issued on: 07/04/1995
Inventor: Williams, et al.

Automatic audio to video timing measurement device and method
Patent #: 5572261
Issued on: 11/05/1996
Inventor: Cooper

Sound-synchronized video system Patent #: 5608839
Issued on: 03/04/1997
Inventor: Chen

Inventor

Bregler, Christoph

Application

No. 620949 filed on 03/25/1996

US Classes:

348/515, Audio to video345/473, Animation348/576, Selective image modification (e.g., touch up)352/87, Animation386/52, Editing704/235, Speech to image715/500.1Synchronization of diverse media

Examiners

Primary: Lee, Michael G.

Attorney, Agent or Firm

Burns, Doane, Swecker & Mathis, L.L.P.

Foreign Patent References

0179701A1 EP. 04/13/1986
0673107A2 EP. 09/13/1995
0674315A1 EP. 09/13/1995
0689362A2 EP. 12/13/1995
06162166 JP. 06/13/1994
2 231 246 GB. 11/13/1990
WO97/15926 WO. 05/13/1997

International Class

H04N 009/475

Abstract

The synchronization of an existing video to a new soundtrack is carried out through the phonetic analysis of the original soundtrack and the new soundtrack. Individual speech sounds, such as phones, are identified in the soundtrack for the original video recording, and the images corresponding thereto are stored. The new soundtrack is similarly analyzed to identify individual speech sounds, which are used to select the stored images and create a new video sequence. The sequence of images are then smoothly fitted to one another, to provide a video stream that is synchronized to the new soundtrack. This approach permits a given video sequence to be synchronized to any arbitrary utterance. Furthermore, the matching of the video images to the new speech sounds can be carried out in a highly automated manner, thereby reducing required manual effort.

Other References

Chen, Tsuhan et al, "Lip synchronization in talking head video utilizing speech information", SPIE vol. 2501, Jul. 1995, pp. 1690-1701
Rabiner, Lawrence R., A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition, Proceedings of the IEEE, vol. 77, No. 2, Feb. 1989. pp. 257-285
Covell, M., et al, "Spanning the Gap Between Motion Estimation and Morphing", pp. V-213-V-216, 1994
Henton, Caroline, et al, "Saying And Seeing It With Feeling: Techniques For Synthesizing Visible, Emotional Speech"
Scott, K.C., et al, "Synthesis of Speaker Facial Movement to Match Selected Speech Sequences", Speech Science and Technology '94, Dec. 5-8, Perth, Australi