Towards AI Technology for Enabling the Metaverse

On Tuesday, December 20, the last lecture in 2022 of ASEF Speaker Series, a series of lectures where ASEF professors present their research areas and findings, took place. This time the speaker was Dr. Sanja Fidler, Associate Professor at the University of Toronto, affiliated faculty at the Vector Institute (and one of the co-founding members), and VP of AI Research at NVIDIA, leading a research lab in Toronto. Her work is in the area of Computer Vision and Machine Learning. Her main research interests are in the intersection of computer vision and graphics, 3D vision, 3D reconstruction and synthesis; and interactive methods for image annotation.

At the beginning of the lecture, Dr. Fidler described her career path. She completed her Ph.D. in Computer Science at the University of Ljubljana and then went on a research visit to UC Berkeley. The insights from courses and her experience abroad made so positive impact on her that she decided to pursue her academic career abroad, which led to her postdoctoral period at the University of Toronto and later becoming a VP of AI Research at NVIDIA. 

With metaverse, it is important to understand that it must have two components, without which it does not exist – neurographics with AI and avatar. During the lecture, Dr. Fidler presented various examples of how her research team is using AI to create images and 3D objects based on the training of artificial intelligence. Tools for 3D reconstruction based on photos existed for decades, but AI-based tools are enabling us to take object reconstruction to the next level. Based on the available data we can teach AI to change the object, construct the missing part, add a new texture, or create a 3D model based on a description. According to our lecturer, this technology still needs development, but it is going to change the world.  

Afterward, Dr. Fidler presented how they train AI to animate humans. For that Dr. Fidler’s team filmed people performing different movements, so then AI can train and predict how the character is going to move or behave. This technology is also quite fast, as what needed months before, now requires only a couple of days of training. At the end of the lecture, she touched upon the topic of autonomous vehicles and the metaverse for simulations. Here, it is a combination of all concepts presented above, which are needed to generate real-life simulations to train the car.  

The event was held in a hybrid format, both in the Prešeren Hall at SAZU and via Zoom, and was moderated by 2022 Junior Fellow Sara Oblak.