UnityVideo is a multimodal text-to-video generation model designed to handle a range of video and modality-based tasks. The model supports five distinct input modalities: depth, DensePose, optical flow (using RAFT), segmentation, and skeleton. Its architecture incorporates joint RGB and modality self-attention, split text cross-attention, modality identity embeddings, and separate output heads for RGB and modality data.
UnityVideo is capable of performing three primary tasks: generating RGB video from text prompts combined with modality information (text-to-RGB+modality), converting video to modality representations (video-to-modality), and generating video from modality inputs (modality-to-video). This flexibility allows it to address a variety of use cases involving both video synthesis and analysis across multiple data types.
The model is distributed as a checkpoint within a Hugging Face repository, and source code along with usage instructions are available at the referenced JIA-Lab-research/UnityVideo repository. 0 license, allowing for open use and modification. Its design and feature set position it within the class of text-to-video and multimodal video generation tools.
In the Video generation space, UnityVideo takes a focused approach. It enables developers and researchers to generate videos from text and multimodal inputs using open-source models. UnityVideo is an open-source project aimed at ai researchers. The project is open source (MIT). The product ships for the command line.
Behind UnityVideo is Kling Team, and the product first shipped in 2025. The project is developed in the open on GitHub with 218 stars and 1 commits in the last 90 days. Among its 5 catalogued features are text-to-video, multimodal input, and depth estimation.
Latest indexed changes and source events
KlingTeam/UnityVideo verified by the PulseGate indexer
Other apps tracked under the same category.
