Then the system grafts and blends those mouth shapes onto an existing target video and adjusts the timing to create a new realistic, lip-synced video. So you have to render the mouth region perfectly to get beyond the uncanny valley.”Ī neural network first converts the sounds from an audio file into basic mouth shapes. “If you don’t render teeth right or the chin moves at the wrong time, people can spot it right away and it’s going to look fake. “People are particularly sensitive to any areas of your mouth that don’t look realistic,” said lead author Supasorn Suwajanakorn, a recent doctoral graduate in the Allen School. ![]() When synthesized human likenesses appear to be almost real - but still manage to somehow miss the mark - people find them creepy or off-putting. ![]() The new machine learning tool makes significant progress in overcoming what’s known as the “ uncanny valley” problem, which has dogged efforts to create realistic video from audio. ![]() “So if you could use the audio to produce much higher-quality video, that would be terrific.”īy reversing the process - feeding video into the network instead of just audio - the team could also potentially develop algorithms that could detect whether a video is real or manufactured. “When you watch Skype or Google Hangouts, often the connection is stuttery and low-resolution and really unpleasant, but often the audio is pretty good,” said co-author and Allen School professor Steve Seitz. “In the future video, chat tools like Skype or Messenger will enable anyone to collect videos that could be used to train computer models,” Kemelmacher-Shlizerman said.īecause streaming audio over the internet takes up far less bandwidth than video, the new system has the potential to end video chats that are constantly timing out from poor connections. The team chose Obama because the machine learning technique needs available video of the person to learn from, and there were hours of presidential videos in the public domain. In a visual form of lip-syncing, the system converts audio files of an individual’s speech into realistic mouth shapes, which are then grafted onto and blended with the head of that person from another existing video. This is the kind of breakthrough that will help enable those next steps.” ![]() “Realistic audio-to-video conversion has practical applications like improving video conferencing for meetings, as well as futuristic ones such as being able to hold a conversation with a historical figure in virtual reality by creating visuals just from audio. Allen School of Computer Science & Engineering. “These type of results have never been shown before,” said Ira Kemelmacher-Shlizerman, an assistant professor at the UW’s Paul G. 2 at SIGGRAPH 2017, the team successfully generated highly-realistic video of former president Barack Obama talking about terrorism, fatherhood, job creation and other topics using audio clips of those speeches and existing weekly video addresses that were originally on a different topic. University of Washington researchers have developed new algorithms that solve a thorny challenge in the field of computer vision: turning audio clips into a realistic, lip-synced video of the person speaking those words.Īs detailed in a paper to be presented Aug.
0 Comments
Leave a Reply. |