Meta has announced Movie Gen, a new generative AI research for media, including image, video, and audio. This technology represents Meta’s third wave of generative AI work and offers four main capabilities: video generation from text inputs, personalized video creation, precise video editing, and audio generation.
The core of Movie Gen is a 30B parameter transformer model for video generation, capable of creating 16-second videos at 16 frames per second. For audio, a separate 13B parameter model can generate up to 45 seconds of synchronized audio for videos. These models can create videos from text prompts, combine a user’s image with a text prompt for personalized videos, allow precise edits based on text inputs, and generate synchronized audio including ambient sound, sound effects, and instrumental background music.
According to Meta, Movie Gen outperforms similar models in human evaluations and achieves state-of-the-art results in personalized video creation and audio generation. The development of these models required innovations in architecture, training objectives, data recipes, evaluation protocols, and inference optimizations. They were trained on a combination of licensed and publicly available datasets.
View this post on Instagram
Meta acknowledges that the current models have limitations. The company plans to work on decreasing inference time and improving quality in future iterations. They also intend to collaborate with filmmakers and creators to gather feedback and refine the technology.
Potential applications for Movie Gen include animating “day in the life” videos for Reels and creating customized animated greetings. However, Meta emphasizes that this technology is not meant to replace artists and animators. Instead, it’s designed to provide new creative tools and opportunities for a wide range of users, from aspiring filmmakers to casual content creators.
Meta views Movie Gen as a step towards making advanced video and audio creation tools more accessible, potentially enabling users to bring their artistic visions to life in new ways. However, they also recognize the need for continued development and refinement of the technology.
