HOW-TO: Work with lipsync

Open

Learn how to preview, import, edit and export lipsync.

1. Preview lipsync

There are several ways to view an existing lipsync in the Editor.

1.1 In the Scene Editor

If you open up a scene (for eg. in: quests\part_1\quest_files\q104_mine\scenes\) then the Editor tries to read the current lipsync data based on the locale, from the cooked depot data (from eg.: r4data\enpc.w3speech).

If the data loading was successful, then you’ll see the Entities' lipsync animations with the corresponding voices. The locale can be changed in the Section tab.

Scene Editor with loaded lipsync from cooked data should look like this:

Open

1.2 In the Animationset

You can also view an existing mimic animation inside a mimic animation set resource file (for eg. in: animations\mimics\). These animations can be exported/imported in a similar way as regular animations (explained in the Blender/REDkit tutorials).

To preview these animations, first you need an entity placed inside the editor’s viewport. Right-click on the viewport and select an entity or choose one of the presets.

One set of Geral’s mimic animation:

2. Exporting and editing lipsync from scene

In the Scene Editor you can export the lipsync animations all at once with the exporting tool.

It is found under Tools -> Lipsync -> Export all lipsyncs from scene.... You can select where do you want to export all lipsync animations inside the folder selection dialog. In that folder all lipsyncs will be exported in separate .re files with their respective voiceFileName.

2.1 Editing a lipsync .re file manually

With the HDFView Tool you can inspect and edit a HDF file.
To make the content editable, select a parent from the treeview and mark it as “Read/Write” like this:

Inside the .re_curve_node -> trackdata dataset, you can see and edit the tracks of the mimic animation.

3. Importing lipsync animation into a scene

In the Scene Editor you can select a specific Timeline SceneLine element on the bottom panel.

If you selected one, on the right side you’ll see two properties: overriddenLipsyncFilePath and overriddenAudioFilePath. In these property fields you can browse for a specific .re lipsync animation file, and .ogg voice file which will replace the ones currently in the Scene.

overriddenLipsyncFilePath and overriddenAudioFilePath properties can override the current lipsync data in the Scene:

4. lipsync in Blender 

Here’s how you can preview, start editing, automatically generating and exporting lipsync in Blender.

4.1. Previewing lipsync in Blender

You can preview an exported .re animation lipsync in Blender.

With the Blender plugin loaded, you have to open the prepared Blender project located in the editor files: blender_project_files\geralt_head.blend.

You can load in an .re lipsync animation file, by right clicking on the h_01_mg__geralt_lod0 object from the Scene Collection at the top right side.

From the context menu, you need to select the Import .re mimic animation item, and then you can browse to an .re file which contains a lipsync animation.

4.2. Editing lipsync in Blender

In the prepared Blender project (blender_project_files\geralt_head.blend), you can edit the phonemes manually via the Shape Key Editor panel visible in the middle.

If the h_01_mg__geralt_lod0 object is selected, then on the left side you have to make sure to “unmute” all the channels. This way when you are editing a phoneme manually on the timeline, then all the mimic weights will be set automatically.

If you want to edit the mimic weights (they are marked as purple in the Shape Key Editor panel) separately then you’ll need to “mute” the channels again.

Table of contents:

4.3. Generating lipsync automatically in Blender

In your Blender project (for this tutorial: blender_project_files\geralt_head.blend), you can import a .wav file for automatic lipsync generation. After right clicking on the h_01_mg__geralt_lod0 object from the Scene Collection at the top right side, you need to select the Generate mimic animation from .wav file item, and then you can browse to an .wav file for automatic lipsync generation.

The .wav audio file should be mono with 16kHz sample rate preferably without any background noise. The detected phonemes for each timestamps will be taken from an English dictionary, but it works surprisingly well with foreign languages too.

For fixing the detection issues, the generated lipsync can be edited manually.

4.4. Exporting lipsync from Blender

In your Blender project (blender_project_files\geralt_head.blend), you can export an .re lipsync animation file.

After right clicking on the h_01_mg__geralt_lod0 object from the Scene Collection at the top right side, you need to select the Export .re mimic animation item, and then you can select the file save location from the popup menu.

In the popup menu, from the settings, it is important to unselect the Rotate by 180 deg around Z axis option, otherwise the characters head will be spun around its axis incorrectly.

While exporting the lipsync animations, the base bones of the head are also exported. These bones can also be viewed and edited inside Blender (it’s the head_anim object inside the Scene Collection panel).