HOW-TO: Work with lipsync
Learn how to preview, import, edit and export lipsync.
1. Preview lipsync
There are several ways to view an existing lipsync in the Editor.
1.1 In the Scene Editor
If you open up a scene (for eg. in: quests\part_1\quest_files\q104_mine\scenes\
) then the Editor tries to read the current lipsync data based on the locale, from the cooked depot data (from eg.: r4data\enpc.w3speech
).
If the data loading was successful, then you’ll see the Entities' lipsync animations with the corresponding voices. The locale can be changed in the Section tab.
Scene Editor with loaded lipsync from cooked data should look like this:
1.2 In the Animationset
You can also view an existing mimic animation inside a mimic animation set resource file (for eg. in: animations\mimics\
). These animations can be exported/imported in a similar way as regular animations (explained in the Blender/REDkit tutorials).
To preview these animations, first you need an entity placed inside the editor’s viewport. Right-click on the viewport and select an entity or choose one of the presets.
You can only call the context menu by right-clicking on the viewport when the editor is not in gameplay mode. If you are currently in Gameplay mode, make sure that the gameplay is stopped by pressing the F10 key.
One set of Geral’s mimic animation:
2. Exporting and editing lipsync from scene
In the Scene Editor you can export the lipsync animations all at once with the exporting tool.
It is found under Tools -> Lipsync -> Export all lipsyncs from scene...
. You can select where do you want to export all lipsync animations inside the folder selection dialog. In that folder all lipsyncs will be exported in separate .re
files with their respective voiceFileName
.
All lipsync animations which have the same name inside the selected folder will be replaced.
2.1 Editing a lipsync .re file manually
With the HDFView Tool you can inspect and edit a HDF file.
To make the content editable, select a parent from the treeview and mark it as “Read/Write” like this:
Inside the .re_curve_node -> trackdata
dataset, you can see and edit the tracks of the mimic animation.
3. Importing lipsync animation into a scene
In the Scene Editor you can select a specific Timeline SceneLine element on the bottom panel.
If you selected one, on the right side you’ll see two properties: overriddenLipsyncFilePath
and overriddenAudioFilePath
. In these property fields you can browse for a specific .re
lipsync animation file, and .ogg
voice file which will replace the ones currently in the Scene.
overriddenLipsyncFilePath
and overriddenAudioFilePath
properties can override the current lipsync data in the Scene:
4. lipsync in Blender
Here’s how you can preview, start editing, automatically generating and exporting lipsync in Blender.
4.1. Previewing lipsync in Blender
You can preview an exported .re
animation lipsync in Blender.
With the Blender plugin loaded, you have to open the prepared Blender project located in the editor files: blender_project_files\geralt_head.blend
.
You can load in an .re
lipsync animation file, by right clicking on the h_01_mg__geralt_lod0
object from the Scene Collection at the top right side.
From the context menu, you need to select the Import .re mimic animation
item, and then you can browse to an .re
file which contains a lipsync animation.
4.2. Editing lipsync in Blender
In the prepared Blender project (blender_project_files\geralt_head.blend
), you can edit the phonemes manually via the Shape Key Editor
panel visible in the middle.
If the h_01_mg__geralt_lod0
object is selected, then on the left side you have to make sure to “unmute” all the channels. This way when you are editing a phoneme manually on the timeline, then all the mimic weights will be set automatically.
If you want to edit the mimic weights (they are marked as purple in the Shape Key Editor
panel) separately then you’ll need to “mute” the channels again.
Editing phonemes manually, channels are “unmuted“
Editing mimic weights manually, channels are “muted“
4.3. Generating lipsync automatically in Blender
In your Blender project (for this tutorial: blender_project_files\geralt_head.blend
), you can import a .wav
file for automatic lipsync generation. After right clicking on the h_01_mg__geralt_lod0
object from the Scene Collection at the top right side, you need to select the Generate mimic animation from .wav file
item, and then you can browse to an .wav
file for automatic lipsync generation.
The .wav
audio file should be mono with 16kHz
sample rate preferably without any background noise. The detected phonemes for each timestamps will be taken from an English dictionary, but it works surprisingly well with foreign languages too.
For fixing the detection issues, the generated lipsync can be edited manually.
4.4. Exporting lipsync from Blender
In your Blender project (blender_project_files\geralt_head.blend
), you can export an .re
lipsync animation file.
After right clicking on the h_01_mg__geralt_lod0
object from the Scene Collection at the top right side, you need to select the Export .re mimic animation
item, and then you can select the file save location from the popup menu.
In the popup menu, from the settings, it is important to unselect the Rotate by 180 deg around Z axis
option, otherwise the characters head will be spun around its axis incorrectly.
While exporting the lipsync animations, the base bones of the head are also exported. These bones can also be viewed and edited inside Blender (it’s the head_anim
object inside the Scene Collection panel).
______________________________________
The Witcher 3: Wild Hunt Complete Edition © 2024 CD PROJEKT S.A. Published by CD PROJEKT S.A. Developed by CD PROJEKT RED. CD PROJEKT®, The Witcher®, REDengine® are registered trademarks of CD PROJEKT Capital Group. All rights reserved. The Witcher game is set in the universe created by Andrzej Sapkowski in his series of books. All rights reserved.