Demersus Devlog #18 - Mocap Animations, Huge Assets Progress, Milestones And More

Trigger warning - fictional character suicide attempt!

Mocap Research

If you didn’t know yet, I am a huge VR nerd and own a full, custom SteamVR based VR + FBT + FT setup:

So, naturally (because of both experience and no meaningful additional cost) I gravitated to virtual reality mocap via Mocap Fusion a piece of software I’ve had the chance to lightly test out while working on my animation and rendering final project a bit ago:

Even though this is the core of the whole mocap pipeline, it’s not everything, so I’ll delve in to how everything was done below!

Scouring Through the Docs

The thing I started out with was reading the entirety of the Mocap Fusion docs (which admittedly were a bit hard to find at first). However, when I finally got to them… I have to say, they are a sight to behold, incredibly in-depth and extremely detailed!

It has great notes on the actual hardware setup, technical details / how everything works, feature of the software itself and a bunch of tutorials on how to go about adding maps, avatars, simulating vehicles, adding props, physics, doing Live-Link , multi-computer / multi-GPU setups and more (I highly recommend anyone interested in mocap, or just VR in general to give it a read):

‹ ›

Initial Finger Track Tests

Something that I really put a lot of emphasis on here (especially because of the starting cutscene) was a more life-like and realistic finger animation. That’s why these pages in the docs really caught my attention:

‹ ›

These confused and worried me a little, as they both seemed to reference finger tracking but at the same time offered quite different solutions (the dating of them was also a bit confusing). The thing is, my setup consists of:

• Meta Quest 3 (Wired = Meta Link)
• EyeTrackVR + Project Babble (Wired = USB)
• 3x Base Station 2.0
• 7x Vive 3.0
• Valve Index Knuckles

• Space Calibrator (for continuous calibration and merging of the two play spaces)
• Project Baballonia (for more optimized and lighter ETVR + Babble)
• VRCFaceTracking (for connecting Baballonia to VRChat)

I use this setup daily and it’s very important to me as I use it to talk to people that are very close and important to me on VRChat, therefore I didn’t really want to mess with anything too much in case I either break things or have to do a bunch of setup, every time I need to do mocap again. Thing is, this setup (because of the wire Meta Link) doesn’t allow for sending finger track data from the built in optical finger tracking system (meaning I’d be forced to use my Knuckles).

Luckily I heard that using apps such as Steam Link or Virtual Desktop does allow you to do just that, so even though it’s not my preferred way to do things (I go with wired for a variety of reasons) I temporarily installed and connected Steam Link to my PC and pretty much without changing pretty much anything in my setup (only had to switch the HMD ID in Space Calibrator), I was able to launch Mocap Fusion and easily switch from my controllers to Quest’s built in optical finger tracking, whilst using all of my Steam VR based gear, through the Steam VR runtime!

Here is the first test (Quest 3 + optical finger track + 6x Vive 3.0s using Space Calibrator - no offsets):

This test was done with some initial offsets for the hands:

Here are the same offsets as above, shown from a different perspective (using Mocap Fusion’s built in camera):

Initial Animation Test

After figuring out the initial setup to get optical finger track working I switched to one of the built in Mocap Fusion human models and tried doing an actual animation test.

Tracker Assignment Issues

One issue I’ve run in to was tracker to body part assignment (mostly with just elbows). I tried messing around with positioning of them on my body but that didn’t seem to change much. After ‘hard coding’ them to my elbows I kept getting weird artifacts, so I decided to do clavicles instead (which seemed to fix it):

Index Knuckle vs Optical Finger Track

This is just meant to be a little quick comparison of the tracking quality you can get from both systems, obviously both have their pros and cons (that’s why I daily the Knuckles) but for our use case, there is a clear winner (keep in mind that both of these are just screen caps of the SteamVR headset view not actual mocap + no clean up!):

• Valve Index Knuckles:

• Meta Quest 3 Optical Finger Track:

Mocap Fusion Export

After actually recording the first animation test, I had to tackle the exporting. First I had to setup the path to the Blender executable (which might seem easy enough). However, because of the ‘weird’ input / keyboard situation in Mocap Fusion I had to close everything down, start it up in desktop mode and just paste it in that way:

At first I wasn’t sure if it’ll work correctly with Blender 5.0, but luckily, after exporting to BVH, opening the scene up in Blender and running the import script:

Here is what I got:

I decided to quickly parent and setup a rough FPP camera by straight parenting it to the head bone and rendered it in the solid viewport view (this is the raw result with NO clean up):

Looking pretty great if I say so myself ;).

Smoothing Adding a Prop and Initial Clean Up

Although, the results above are impressive, they do leave a bit to be desired (I mean, it was just the very first test). So, I did some more testing to ‘smooth things out’ and try working with props and here is how that went!

Adding a Prop

Something that I really wanted to utilize was the ability to import your own props in to Mocap Fusion, this would allow me to easily and accurately act out the movements that I want to do (for example with the gun) because I have the model, right in my hands!

So, I started out by downloading yet another (third) version of Unity on to my PC and proceeded to install the Mocap Fusion SDK (which for some reason took a surprisingly long time to download…).

Next I read through the import docs of Mocap Fusion:

After that, I jumped in to Unity, created a new URP Universal 3D project, moved over the assets for the revolver from the game project (I made sure to leave the cylinder open as I didn’t bother making it actually interactive, but did want to have the reference available for when I’m putting in the bullet):

Something important here is that if you’re having magenta missing material issues here, you should try doing APP_SDK > Render Pipeline Fixer > Set Build-in Render Pipeline!

After that I took care of setting it up as a Mocap Fusion prop:

‹ ›

I set up the prop config and the initial ‘grab point’ (which is there pretty much only for spawning the prop and you’re able to grab it however you want later on). When that was all done I exported it to Mocap Fusion:

Something that worried me quite a bit here was the fact that I had to have my hand squeezed in to pretty much a full fist to be able to hold the gun which would make the clean up process much more complex for no reason, but luckily it turns out that there is a prop menu and you’re able to setup auto hold:

After figuring that out I recorded the second animation test.

Initial Smoothing Tests

After noticing the bitterness and small tracking mishaps in the first animation test, I really wanted to try smoothing out (at least the camera) in the second test I did. Smoothing out just the camera turned out to be a bit problematic (as it would require to pretty much smooth out the whole neck, spine and so on), so instead I just smoothed out everything, which yielded quite good looking results if I say so myself!

This is basically the process I did:

1. Select the armature with the animation you want to smooth.
2. In the main layout (Pose Mode) Pose > Animation > Bake Action (check Visual Keying and Clear Constraints) = This converts live, computed animation data in to keyframes on the object:

3. In the graph editor, select the bones you want to smooth.
4. In the graph editor Key > Clean Keyframes = This removes redundant keyframes without changing the curve shape, which helps before smoothing.
5. Select the keys you want smoothed and press Alt + O a few times (2x or 3x have worked pretty well for me).

Important things to remember are:

• perform this AFTER you’re done with clean up as that will make your work much easier
• if you have a prop that your character is holding, you need to smooth out the motions of everything together for best results
• there still needs to be a bit more testing done regarding Clean Keyframes as I’ve noticed that it can cause a ‘floating’ effect on props being held by the character

Here is how the first animation test looks after smoothing:

I did get some input saying that the shakiness adds to the realism as the person is about to take their own life, however the point where shakiness becomes tracking jitter is very thin, therefore I think we’ll most likely go with 2x or 3x smoothing for everything just to make everything a bit easier on the eyes although there is an argument to be had about a creative decision to go either way.

Initial Clean Up

Now, I’m not an animator, so as of now, the only ‘clean up’ I knew how to do was, removing problematic keyframes and fixing up the movements / positions manually for the problematic duration. I’ve already done some research, and I’m pretty sure I should be able to do it better. This person has a really great, informative video about working with mocap in Blender !

So the approach I described above is basically what I did here, just removed or replaced some of the problematic movements (mostly to fix clipping with things such as finger and the grip and so on).

Oh, and of course I also manually animated the switch and cylinder of the revolver!

‹ ›

Final Result

Here is how all of these tests turned out:

Initial Scene Import

asdf

Initial Joining of Sequences + Blending

asdf

Export Testing

asdf

Looping Walk Cycle Test

asdf

Initial Main Pipeline Notes

asdf

Initial Test of Working in Timeline

asdf

The Pain of Making Our Own Human Character

asdf

Master Rig, Avatars and Playing Animations in Timeline

asdf

Better Exports, Handling Cameras and Gameplay Blending

asdf

Fixing Final Custom Character Issues

asdf

Mocap Pipeline

Because of time constraints, tight deadlines and not a lot of manpower, this pipeline is a cut down summary of a lot of research and trial and error that went in to figuring this all out. There was a lot more content, but sadly I simply didn’t have time to go through all of it in detail, but nonetheless, this should hopefully be sufficient.

So, without further ado, this is the whole pipeline for the process of recording, cleaning up, and importing VR based, motion capture animation in to the engine.

summary, notes and overall cleaned up info, some tips, warnings, notes and such

Overall Process

asdf

Exports & Imports

asdf

Handling Props and Pickups

asdf

Handling Cameras

asdf

Blending With Gameplay

asdf

Prepping Scene and Props

• scenes are the environments the cutscene will take place in and can help you get more immersed and physically align with different objects in the space, you can quite easily just select and export a bunch of models from your files in to a single FBX file and import them like so, use the APS_SDK, then just make sure it’s alligned to 0,0,0 world position and add a collider for the floor so you can stand and walk around and such
• props can be useful to hold them and better align movements and such however they can be quite problematic, therefore, in my experience, it’s just better to use them as reference and remove them later on
• if you want to animate things moving in some way (for example doors opening), even though you might be tempted to just use transforms, that won’t allow you to have the world origin offset, therefore it is recommended to use a simple armatuer (tests show that you don’t actually need a root bone that sits on world origin it seems)
• instead of going with props, I’ve found using the Child Of modifier on a root bone of a properly setup model to work better (check out more info on this below)
• you can also setup a custom avatar rig, this is quite problematic though, we went with makehuman and had these problems: X that we solved like this: Y.
  • https://unity.com/blog/engine-platform/mecanim-humanoids
  • https://mocap-fusion-vr.fandom.com/wiki/SDK_-_Importing_Custom_Avatars
  • https://www.youtube.com/watch?v=oEwkhIr4ffw
  • https://www.youtube.com/watch?v=mlAgW8KlUtk
  • https://www.youtube.com/watch?v=ykJ7O0Bs8oQ
  • https://www.youtube.com/watch?v=ZfpbS3cZXDk

Add missing origin / stationary root bone

1. Select your existing armature
2. Go in to Edit mode
3. Press Shift + A
4. Edit the newly created bone’s values
5. (Still in edit mode)
6. Select the existing top bone of the armature
7. With Ctrl being held, select the new root
8. Ctrl + P
9. Select Keep Offset

Working in Mocap Fusion

• don’t redo the tpose if you’re still recording the same sequence as that will put the ‘master rig’ out of whack, not to mention your tracker offsets and such will be ruined
• also beware of tpose issues if you mess with a playspace mover like OVR before you do try to do one (it’s pretty finicky overall)

• to be able to use finger track you need to use steam link / air link / virtual desktop and so on (it won’t work over wired)
• load your avatar
• load your preset
• do the tpose (if mocap fusion is bugging out just restart it and try again)
• perform tracker offset calibration (sometimes you just have to move them around on your body)
• load in to your reference environment
• when everything is setup, place yourself correctly (you can also use a play space mover like OVR) in the scene (make sure you have enough space for the action you want to perform)
• depending on the situation and what you’re recording it may be beneficial to switch from the built in optical hand tracking to the Valve Index Knuckles as they won’t cause tracking glitches and jumps when recording walking animations for example (where finger movements don’t matter)
• record a sequence and after you’re done (before pressing end) export to BHV and make sure everything is good, then you can press end and go on to record another one

Working in Blender

• tweaking the animation would be better done using IK, influence and such but we didn’t have time to figure that out
• if you’re working on something such as a walk cycle, you can clear out movement recorded on the pelvis, make the animation loop perfectly and then use a follow path bone modifier on the pelvis to animated that repeated walk however you’d like (make sure you zero out the transforms on the pelvis before applying it though)
• with first person camera, doing just a linear transition of the movement along a bezier cufve is perfectly fine and will feel natural as long as you roughly match it to the walking speed
• after loading sequences using the script, make sure to put them in to the NLA editor so you can export them correctly (remember to set an identifiable name for the action and also don’t trim out anything)
• before exporting, check the 0 frame t-pose and make sure you export with the playhead positioned on it (otherwise you’ll have glitches bad retargetin glitches, tremors and so on)!!!
• remember to select things properly (children too) and use the proper export preset / setup (but mesh + armature but no anim for master and just armature and anim for rest)

• make sure to set a proper framerate (FPS) for the ‘joining’ blender file (also, remember that this can be influenced by imports, new cameras and so on so remember to change it back if that happens)
• you should get the master rig from the first recording of the sequence because of small scale differences (this will cause stuff to be shifted in space sometimes)
• all next clips in the same sequence (if the tpose wasn’t update in mocap fusion) should be exported as just armatures (these will be put on the master rig later for blending and such)
• in the ‘joining’ blender file, load up the master rig, after that, load in all of the animations (make sure to rename the armatures ; meshes and such for a clean hierarchy) one by one
• after that, select the master and in the Action editor (subtype), pick the animation you want to push down on to the NLA, pick the rig of the master rig, make sure everything is good and after that, rename it and push the action down
• after getting all sequences set up, you can now hid the source armatures, trim the clips, blend between them and so on
• now with everything working, import all of the scene models (such as objects, environment and so on)
• via tweaking Tab and such, you can influence and / or fix up the imported animations (remember to have the proper armature selected and proper editing mode ; beware that tweaking removes the clip start / end changes you did!)
• in this step, the Action, NLA and Graph editor will be your friends (it’s important to note, in the graph editor you can use the pointer button toggle to show all / only selected bones, try this if you’re not seeing the bones you’d like to work with)
• now you can also animate all of the props / objects in the scene (also using modifiers such as Child Of )
• when trying to attach props to a character’s hand, use the child of modifier on the sub root bone of the object, make sure you set the object in the correct ‘rest’ pose, then when the hands is near do set inverse, keyframe the original position and rotation, keyframe the influence and adjust and after that keyframe the proper position in hand (make sure to add move and rotaiton keyframes and you should be okay)
• DON’T add cameras here (unless it’s only for reference of course) as that doesn’t really work out in timeline and it’ll be impossible to control or do smooth transitions between cameras properly

• make sure to bake the animation of every object that is animated (you don’t have to smooth everything though, just the mocap to reduce jitter)
• when baking, make sure you’re in pose mode and have the entire armature selected of the thing you want to back, then go to Pose > Animation > Bake Action and make sure Only Selected Bones ; Visual Keying and Clear Constraints are selected (depending on what it is you’ll want to do either Pose or Object mode as well)
• after baking, if you see everything as ‘flat’ in the graph editor, make sure to turn off normaliziation
• on the baked clip you can smooth everything out by selecting every bone + track and then pressing Alt + O, X amount of times (I’ve found 2x works best for everything and 4x for the head bone as the camera gets attached to it)
• after everything is baked (should be one clip per object), you can proceed to export everything one by one in to separate FBX files (both mesh and animations in to FBX files ; also make sure you applied all transforms on everything before export)
• when exporting make sure you have the proper clip and proper elements in the hierarchy selected (remember that having the parent selected doesn’t mean it’s children are too)!!!
• for exporting use the Mocap preset and make sure Simplification is set to 0!!! (otherwise you’ll get things that are floaty and slightly out of position!!!)

Exporting to Unity

• this should be done by having ‘master rigs’ and then simply exporting armatures (just animation) as FBX but I had big issues with missing bones, not to mention retargetting problems as well and just decided to so to speak ‘bite the bullet’ and simply export it with the mesh included
• if you get parts of the models disappearing, make sure to check Update When Offscreen on their Skinned Mesh Renderers
• if you get a ‘wrong’ origin point, make sure you’re in pivot and not center mode
• animating emissions maps is tricky, so if you’re trying to do that, do it by turning OFF Use Emission Intensity and instead animating it using the Emissive Map HDR color (and also to not have issues, if the object has multiple material slots, make sure to add the map to every one of the materials it uses!)
• you can create a black / white transition using an ‘image’ of a color that covers the whole screen using a ui canvas and just animating it’s opacity
• don’t try to blend mocap animations that have movement stored in them as the root will cause very weird, unnatural jumpy transitions between clips (do all this in Blender)
• depending on the activation clip settings and such, if you have a hold cnlip that is present at the last frame of the timeline, you can have they object stay on after the timeline ends!

• import the baked animations and meshes as FBX files of each humanoid, prop and so on and place them in the same point for alignment (you can also utilize empty game objects for easily moving things)
• on each FBX with an animation, make sure to set Anim. Compression to Off (otherwise you’ll get things that are floaty and slightly out of position!!!)
• import the static FBXs from the reference scene (as they have the right origin offsets)
• create an empty game object which will serve as the container for the cutscene
• set up a new timeline on the new object
• grab all of the FBXs you imported earlier and drag them on to the empty object
• add a camera (you can attach it securely to the head bone of your humanoid rig, or have it be on a dolly or whatever)
• add and animate additional things such as lighting and so on
• after you’re done setting everything up, you can proceed to adding audio, SFX, voice lines and so on (this is not the best or the most proper way to do things, but because of the manpower and time, this is how we’re doing it)
  • recording the timeline precisely using the Unity Recorder package
  • adding and mastering the SFX and such in Premiere
  • exporting them as a single or few tracks
  • importing them in to unity
  • using a singular 2D audio source on the player (doesn’t need to be on the cutscene protag as it’s 2D anyways)
  • setting them up and playing via the timeline editor
• do a gameplay transition (to be able to can animate the player’s position, smoothly blend between cutscene and FPP cameras and give back control to the player, you need to follow this checklist)
  • in order to do that, you need to add a signal track
  • add an emitter where you want the hand off to happen
  • add a CutscenePlayerManager with the script + setup the script
  • drag the CutscenePlayerManager in to the Signal track slot
  • setup a Signal receiver component on the CutscenePlayerManager (like so)
  • make sure to name (using the inspector) the player and camera pivot tracks properly so they can be muted by the cutscene handoff controller (this is kind off ass backwards but I’m not sure of a better way to go about doing this right now)

Huge Progress on Assets

My bud has made huge progress with the assets, both in terms of IRL and clay ones, not to mention the environmental level design as well. I also performed some asset finalization and baking. Here is how all of that turned out:

Initial IRL Assets

These are the initial IRL assets (mainly for the office scene) before they were finalized:

‹ ›

Office Layout and Scaling

Because of how Mocap Fusion handles calibration and exports, getting a proper scale reference was a bit tricky, but in the end we finally managed to get it working:

‹ ›

Attic Clay Assets

These assets are clay models mostly for the attic:

‹ ›

IRL Asset Finalization

Before being able to start working on mocap for the start cutscene, to make the process easier (as models get exported from the Blender mocap scene, so the baked in world origins allow to simply set all of their world transforms to the same point and have everything line up automatically).

I wanted to have the models finalized as otherwise the UV’s could change later on, which means the baked textures wouldn’t map correctly and it would be a whole lot of additional work (it’s just less hassle this way). So, I devised and followed this checklist (which was retrofitted to Devlog #17 ) as it was directly related to the IRL pipeline) on all the assets needed for this scene:

‹ ›

‹ ›

Triangulation Issue

Something worth mentioning however is an issue I’ve encountered with triangulation done by Unity’s FBX importer. Because of these specific quads in the door geometry, the triangulation process added new edges there, causing a slight UV mismatch, making the corners black (because the UV changed). I Initially tried fixing it by performing triangulation in Blender, but that resolved one issue and created new ones.

Finally I ended up fixing it, by simply manually adding additional edges in the problematic areas, re-doing UV unwraps and the whole export process:

‹ ›

Shade Auto Smooth & Solidify Glitches

While finalizing some assets, I’ve found that two things can cause some weird behaviour, to be specific visual glitches. I’ve found that Shade Smooth Auto, in some cases, if it’s not applied on the final model, right before export from Blender it can create very weird shadow artifacts:

Aside from that, while working on the office environment, where my bud used a Solidify modifier, I’ve found that the Fill option, also causes similar issues in certain cases:

‹ ›

More Clay Assets

This is a more general set of clay models (mostly furniture):

‹ ›

Environment Layouts

Aside from the previously mentioned models, my bud also worked a bunch on clay environments:

‹ ›

Dialing in the Baking Guidelines

Keep in mind, that these are just general, and depending on the case will have to be adjusted (use common sense)!

Because my bud has started finalizing and baking the assets (and environments) for in-engine use, we had to finally sit down and figure out the texture scaling among a few other parameters for consistent, good looking results. Here is the guidelines we came to:

• Furniture
  • scale = 20
• Environments
  • scale = ~20
  • cracks = ~5
  • bumps = by Global Bump and Voronoi Strength
  • fingerprints = ~5 with a lower color usage of a dark value
  • resolutions = normals 8K ; others 4K (this is because of the larger physical size)

For both types, the displace is done pretty much how it was described here basically, setting the Midlevel the same as in the source material and the Strength as 1/3 of the source material’s Global Displacement.

Milestone Updates

When mentioning milestones on here, I usually show a bullet point list, but because of how dynamically things have been changing lately (with the deadlines inching closer and closer), I moved over from Figma to an Excel spreadsheet to have a clearer view of things against dates:

One More Thing

Before I close this devlog off, I’m very excited to show off something that has been in the works for quite a while:

What’s Next?

I’ll be moving on to actually recording, cleaning up and setting up the animations that we need for both gameplay and cutscenes, when I receive enough assets from my bud to be able to work on the next stage. As said above, my bud is toiling away at getting all of the assets baked and finalized!

In other news, quite a bit ago now, I finally managed to add EyeTrackVR and Project Babble based face tracking to my Quest 3 VR setup (which, fun fact I am able to use with Mocap Fusion), it’s been a lot of fun in VRChat though ;)