Microsoft has introduced a new AI model called VASA-1, which is capable of generating “lifelike audio-driven talking faces in real time.” VASA stands for “visual affective skill,” and the model only requires a single portrait photo and a speech audio track to create its output.
According to Microsoft, the output is “hyper-realistic” and can capture a wide range of expressive facial nuances, with precise lip sync and natural head motions. It can also handle arbitrary-length audio and stably output seamless talking face videos.
VASA-1 is capable of handling types of photos and audio inputs that were not in the training dataset, such as singing audio, artistic photos, and non-English speech. As an example, Microsoft provided a clip of Da Vinci’s Mona Lisa portrait singing a rap song.
Potential use cases for VASA-1 include gaming, social media, film making, customer support, education, and therapy. In the offline processing mode, the model can generate video frames of 512×512 size at 45 frames per second. In the online streaming mode, it can go up to 40 frames per second with a preceding latency of 170 milliseconds.
Similar lip sync and head movement technology is available from other AI companies, but experts say that VASA-1 appears to be of a much higher quality and realism. It can make the face move in three dimensions and the eye gaze move in different directions, which makes it much more realistic.
However, as with any video-generating AI model, observers have flagged that VASA-1 makes it easier to create deepfakes and that there is potential for misuse. Microsoft has said that it has no plans to release an online demo, API, product, additional implementation details, or any related offerings until it is certain that the technology will be used responsibly and in accordance with proper regulations.
