Runtime MetaHuman Lip Sync (AI for NPCs) (+ CC4, Genesis, ARKit, and More) – Unreal Engine
Note: The images with plugin examples and the demo project were created using the Runtime Audio Importer and/or Runtime Text To Speech / Runtime AI Chatbot Integrator plugins. So, to follow these examples, you will need to install these plugins as well. However, you can also implement your own audio input solution without using them.
Three Advanced Models for Every Project Need!
Bring your MetaHuman and custom characters to life with zero-latency, real-time lip sync! Now featuring three quality models to suit your project requirements:
Mood-Enabled Realistic Model – Emotion-aware facial animation for MetaHuman characters with 12 different moods (Happy, Sad, Confident, Excited, etc.), configurable intensity, and smart lookahead timing
Realistic Model – Enhanced visual fidelity specifically for MetaHuman characters with more natural mouth movements (81 facial controls)
Standard Model – Broad compatibility with MetaHumans and custom characters (14 visemes)
Transform your digital characters with seamless, real-time lip synchronization that works completely offline and cross-platform! Watch as your characters respond naturally to speech input, creating immersive and believable conversations with minimal setup.
Key features:
Real-time Lip Sync from microphone input and any other audio sources
Emotional Expression Control with 12 different moods and configurable intensity
Dynamic laughter animations from detected audio cues
Pixel Streaming microphone support – enable live lip sync from browser-based input!
Offline Processing – no internet connection required
Cross-platform Compatibility: Windows, Mac, iOS, Linux and Meta Quest
Optimized for real-time performance on all platforms
Works with both MetaHuman and custom characters:
Popular commercial characters (Daz Genesis 8/9, Reallusion CC3/CC4, Mixamo, ReadyPlayerMe)
FACS-based character models
ARKit blendshape standard
Any character with custom morph targets for facial expressions
Multiple Audio Sources:
Live microphone input (via Runtime Audio Importer’s capturable sound wave)
Captured audio playback (via Runtime Audio Importer’s capturable sound wave)
Synthesized speech (via Runtime Text To Speech or Runtime AI Chatbot Integrator)
From audio file/buffer
Custom audio source (including streaming)
Works great with:
Runtime Audio Importer – For microphone capture and audio processing
Runtime Text To Speech – For local (offline) text-to-speech
Runtime AI Chatbot Integrator – For ElevenLabs and OpenAI text-to-speech
Technical details:
This plugin provides real-time lip synchronization for MetaHuman (as well as custom) characters by processing audio input to generate visemes.
The plugin relies on the onnxruntime library (MIT licensed) as the cross-platform native machine learning accelerator for lip sync inference. It leverages the NNERuntimeORT plugin that comes out-of-the-box in UE 5.6+ to provide onnxruntime functionality.
The plugin internally generates visemes (visual representations of phonemes) or facial control data based on the audio input, and offers three models: Standard Model (14 visemes, works with all characters), Realistic Model (81 facial controls, MetaHuman-exclusive), and Mood-Enabled Realistic Model (emotional expressions with 12 moods, MetaHuman-exclusive).
Features:
Simple, intuitive setup
Real-time and offline viseme generation
Emotional expression control with 12 moods
Universal language support – works with any spoken language through audio analysis
Multiple audio input sources (microphone, playback, synthesized speech, such as ElevenLabs or OpenAI, custom PCM, Pixel Streaming)
Direct integration with MetaHuman’s face animation system
Support for character systems (Daz Genesis 8/9, Reallusion CC3/CC4, Mixamo, ReadyPlayerMe)
Compatible with FACS-based blendshapes, ARKit standard, and custom morph targets
Configurable interpolation settings
Blueprint-friendly implementation
No external dependencies or internet required
Supported Unreal Engine Versions: 5.0 – 5.6
Recent Comments