Difference between revisions of "Lip sync"

Revision as of 17:51, 16 October 2018

Name	By
How to generate lip sync files with Rhubarb Lip Sync & use them in Visionaire Studio (animation, tsv files)	AFRLme

This tutorial will show you how to setup Rhubarb Lip Sync, how to generate lip sync files from an audio recording containing speech, & how to make your talk animations lip sync ready.

Tutorial

Installing Rhubarb Lip Sync

First things first, you will need to download the latest version of Rhubarb Lip Sync. You can find it here.

Once you have downloaded the latest version of Rhubarb, you should navigate to c:/program files & create a new folder. Rename the new folder rhubarb. Now open up the zip file containing the latest version of Rhubarb & drag the contents into the rhubarb folder you just created.

Now minimize or close all open windows & double click on my pc/my computer; or whatever it is called on your pc. Wait for the window to open up & then right click somewhere where there is empty space, then click on properties. On the next window that opens up, click on advanced system settings, then click on environment variables on the next window that pops up.

Now you need to locate & select path found under system variables. Once you have done that, click on the edit button. Another window will pop up. Click on the new button & type C:/Program Files/rhubarb/ into the new entry that you just created, then click on the ok button. That's all there is to setting up Rhubarb Lip Sync on Windows. Close all the windows & you are done.

Using Rhubarb Lip Sync

Assuming you have installed Rhubarb Lip Sync, it's now time to generate your first tsv lip sync file - but first, what is a tsv file? tsv stands for tab separated values. It consists of a timestamp (in seconds) followed by a mouth shape letter or an animation frame number. It will look a little something like this...

0.00	X
0.67	B
etc.	etc.

Quick note: the tsv lip sync files can contain either the mouth shape letters or animation frame numbers, so feel free to edit the tsv files & use numbers instead if you want to go beyond basic lip sync animations - maybe your characters contain animations for a wide range of emotions, or you want the head to face a different direction to the body, etc.

Resources

Name	Description
mask_example.zip	A working .ved file, complete with resources. Check out the readme.txt file for instructions.

@@ Line 7: / Line 7: @@
 This tutorial will show you how to setup [https://github.com/DanielSWolf/rhubarb-lip-sync Rhubarb Lip Sync], how to generate lip sync files from an audio recording containing speech, & how to make your talk animations lip sync ready.
 == Tutorial  ==
@@ Line 16: / Line 17: @@
 [[File:Lip sync 1.png|frameless|800x800px]]
-Now minimize or close all open windows & double click on '''my pc''', '''my computer''' or whatever it is called on your pc. Wait for the window to open up & then right click somewhere where there is empty space, then click on '''properties'''. On the next window that opens up, click on '''advanced system settings''', then click on '''environment variables''' on the next window that pops up.
+Now minimize or close all open windows & double click on '''my pc'''/'''my computer'''; or whatever it is called on your pc. Wait for the window to open up & then right click somewhere where there is empty space, then click on '''properties'''. On the next window that opens up, click on '''advanced system settings''', then click on '''environment variables''' on the next window that pops up.
 Now you need to locate & select '''path''' found under '''system variables'''. Once you have done that, click on the '''edit''' button. Another window will pop up. Click on the '''new''' button & type '''C:/Program Files/rhubarb/''' into the new entry that you just created, then click on the '''ok''' button. That's all there is to setting up Rhubarb Lip Sync on Windows. Close all the windows & you are done.
@@ Line 22: / Line 23: @@
 <html><iframe width="800" height="450" src="https://www.youtube.com/embed/iUPP2feLyN8" frameborder="0" allowfullscreen></iframe></html>
-{| class="ts" style="width:100%"
-|-
-| ''Quick note: alpha transparency needs to be present in both the image/animation that you want to mask & the image that you will be using as the mask. If you are using a simple shape for the image, such as a square or rectangle, make sure you extend the canvas on all sides by a couple of pixels or so & that the canvas background is transparent. This seems to prevent any strange drawing behaviors when the engine calculates which parts of the masked image to draw in.''
-|}
-[[File:Mask plugin (02v2).png|frameless|800x800px]]
+=== Using Rhubarb Lip Sync ===
-Now that you have an image you want to mask, you will need to create an image to use as a mask. Your mask image should comprise of white & alpha transparency. Only the parts of the masked image that overlap the white parts of the mask image will be drawn in.
+Assuming you have installed Rhubarb Lip Sync, it's now time to generate your first tsv lip sync file - but first, what is a tsv file? tsv stands for tab separated values. It consists of a timestamp (in seconds) followed by a mouth shape letter or an animation frame number. It will look a little something like this...
-Now create a new scene condition & name it something along the lines of '''mask''' - it doesn't actually matter what you call it - then link the condition to the properties section of the mask object. The condition will be used to hide the mask object, as we do not actually need to see the mask image in-game to be able to mask another image.
+{| class="ts"
+|-
+| 0.00 || X
+|-
+| 0.67 || B
+|-
+| etc. || etc.
+|}
-[[File:Mask_plugin_(03v2).png|800px]]
-The final step is to use the start mask on object action part that will allow you to specify the target (object, scene, or character) & the scene object that contains the mask. To make things simple, let's just apply the mask as soon as the scene loads.
+{| class="ts" style="width:100%"
+|-
-Click on the scene actions tab then click on the [[File:add_25.png|25px|link=]] button to create a new action. Set the action type as '''at beginning of scene''' & then click on the [[File:add_25.png|25px|link=]] button. Type "mask" into the search bar at the bottom of the action part selection modal. Select the '''mask object with object''' action part. Now link '''object''' to the scene object containing the image/animation you want to mask, then link '''mask''' to the scene object containing the image you want to use as the mask. Voila.
+| ''Quick note: the tsv lip sync files can contain either the mouth shape letters or animation frame numbers, so feel free to edit the tsv files & use numbers instead if you want to go beyond basic lip sync animations - maybe your characters contain animations for a wide range of emotions, or you want the head to face a different direction to the body, etc.''
+|}
-[[File:Start_mask_plugin_(v2).gif|800px]]
-== Reference Video ==
-<html><iframe width="800" height="450" src="https://www.youtube.com/embed/EVsndUQ0VY0" frameborder="0" allowfullscreen></iframe></html>
 == Resources ==