Silent video to speech reconstruction

StreamLip Audio Reconstruction

StreamLip restores speech audio from silent face video with a self-trained StreamLip V5 visual-text branch and timbre-conditioned deterministic waveform reconstruction.

View Code Run the Demo Report Poster
Generated Trump silent-video reconstruction demo
Trump silent/reference demo: restored speech audio is muxed back to the original face video. Checked-in example
V5 Self-trained visual-to-text branch used as the default raw-video path.
3.04s Optional reference audio window for same-layout timbre conditioning.
1 cmd Silent mp4 plus optional reference audio to generated voiced mp4.

Why StreamLip

The system is not a pure vision-to-text-to-audio cascade. Text is useful as a semantic condition, but audio quality is driven primarily by lip timing, visual speech features, Mimi audio latents, and speaker/timbre conditions. This makes the reconstruction less dependent on perfect transcript accuracy.

Self-Trained V5 Branch

StreamLip V5 consumes visual speech features and decodes text through an LM-based decoder with visual cross-attention.

Deterministic Recon Head

The audio branch predicts Mimi latent residuals directly instead of relying on a slow sampling or flow-matching trajectory.

Timbre Conditioning

A short reference audio segment provides speaker and prompt statistics, improving perceived voice consistency.

System Architecture

Silent video is normalized, lip features are extracted, StreamLip V5 provides semantic text features, Mimi supplies the audio latent space, and the recon head generates speech conditioned on visual, text, and timbre signals.

StreamLip system architecture
StreamLip V5 and the reconstruction model form the default end-to-end inference path.
Reconstruction method compared with sampling
Direct reconstruction keeps inference simple and stable for the current release.
Silent reference audio pipeline
Reference mode places timbre audio into the same first-window prompt layout used by training.

Generated Examples

The Trump example is the primary checked-in silent/reference demo. Five additional short generated videos show the same reconstruction pipeline on non-Trump clips.

Trump Silent/Reference Demo

Silent input with a separate reference segment for timbre conditioning.

Generated Example A

Longer post-prompt generated output from the raw-video pipeline.

Generated Example B

Additional non-Trump reconstruction example.

Generated Example C

Short post-prompt generated output from the release pipeline.

Generated Example D

Additional generated face-video output with restored audio.

Generated Example E

HRX reprocessed example using the current visual preprocessing path.

Run the Pipeline

The release uses local checkpoints under ckpt/. Our trained weights are distributed through a project checkpoint repository, while public Mimi and SmolLM2 dependencies can be restored from Hugging Face mirror.

Default one-command silent/reference demo

After installing requirements and restoring checkpoints, this command reproduces the checked-in Trump-style reference workflow.

.venv/bin/python \ scripts/run_raw_video_avsr_recon_pipeline.py \ --input data/assets/trump_silent_ref_demo/trump_silent_input_no_tail3s.mp4 \ --ref_audio data/assets/trump_silent_ref_demo/trump_ref_tail3s.mp4 \ --silent_input \ --exp trump_silent_ref_demo_full \ --force
Repository Code, README, scripts, and tests. Checkpoints Project-specific model bundle. Report Technical summary and experiments. Poster Presentation poster PDF.