Text-to-Video

MoCha can generate talking character animations from speech and text, allowing for multi-character conversations with turn-based dialogue.

RealisDance-DiT can generate high-quality character animations from images and pose sequences. It effectively handles challenges like character-object interactions and complex gestures while using minimal changes to the Wan-2.1 video model and is part of the Uni3C method.

PA-VDM can generate high-quality videos up to 1 minute long at 24 frames per second.

HunyuanCustom can generate customized videos with specific subjects while keeping their identity consistent across frames. It supports various inputs like images, audio, video, and text, and it excels in realism and matching text to video.

Phantom can generate videos that keep the subject’s identity from images while matching them with text prompts.

SkyReels-V2 can generate infinite-length videos by combining a Diffusion Forcing framework with Multi-modal Large Language Models and Reinforcement Learning.

FramePack aims to make video generation feel like image gen. It can generate single video frames in 1.5 seconds with 13B models on a RTX 4090. Also supports full fps-30 with 13B models using a 6GB laptop GPU, but obviously slower.

TTT-Video can create coherent one-minute videos from text storyboards. As the title of this paper says, this uses test-time training instead of self-attention layers to be able to produce consistent multi-context scenes, which is quite the achievement. The paper is worth a read.

AccVideo can speed up video diffusion models by reducing the number of steps needed for video creation. It achieves an 8.5x faster generation speed compared to HunyuanVideo, producing high-quality videos at 720x1280 resolution and 24fps, which makes text-to-video generation way more efficient.

MotionMatcher can customize text-to-video diffusion models using a reference video to transfer motion and camera framing to different scenes.

PP-VCtrl can turn text-to-video models into customizable video generators. It uses control signals like Canny edges and segmentation masks to improve video quality and control without retraining the models, making it great for character animation and video editing.

Mobius can generate seamlessly looping videos from text descriptions.

MovieAgent can generate long-form videos with multiple scenes and shots from a script and character bank. It ensures character consistency and synchronized subtitles while reducing the need for human input in movie production.

VideoMaker can generate personalized videos from a single subject reference image.

Step-Video-T2V can generate high-quality videos up to 204 frames long using a 30B parameter text-to-video model.

Magic 1-For-1 can generate one-minute video clips in just one minute.

Diffusion as Shader can generate high-quality videos from 3D tracking inputs.

Lumina-Video can generate high-quality videos with synchronized sound from text prompts.

FlashVideo can generate videos from text prompts and upscale them to 1080p.

VideoGuide can improve the quality of videos made by text-to-video models without needing extra training. It enhances the smoothness of motion and clarity of images, making the videos more coherent and visually appealing.