Blender Game Engine + Shape-Key animation


I am using the “ton” example Python script in Blender with a different mesh and shape keys for the facial animation (Windows 10, Kinect v.2) and all is working well–but I want to add a “live feed” plane (of the user) to the scene–this requires the game engine available in the Ni-Mate plug-in, but when I try both simultaneously the shape keys are static; no animation of the above. Is it possible to do both together? Is there another way to do this? the only other scripts I have seen for live feed of webcam (as ‘video texture’) to Blender also require the game engine.
Any thoughts, suggestions, much appreciated!

Live motion capture and object interaction with Blender + Ni-Mate

Sorry about the delay in responding.

It seems the Blender Game Engine does not do shape key updates at all. You would have to do something with armatures instead, binding the expressions to some bones and drive them instead of shape keys. Let’s do this in steps to make sure every part of the process works.

  1. Create a cube in the scene. This is the object that will contain the main game engine update loop. You can hide it this cube from the 3D view with the “H” key later.
  2. With this cube selected in object mode, press the “Create game logic” button in the NI mate plugin
  3. In NI mate, enable skeleton tracking
  4. To make things easier to debug, in NI mate change the “Source” on the main sensor page to the example clip and start playback. This way you can sit at your computer and have some valid skeleton data constantly outputting to Blender.
  5. Set the “Create” mode to “cubes” in the NI mate plugin
  6. Start receiving normally, cubes should appear and move with the playback from NI mate
  7. Stop receiving
  8. In 3D view, press “P” on the keyboard to start the game engine
  9. The cubes should now move in the game engine with the NI mate playback

The cube object containing the main game logic is now calling the main NI mate plugin’s “updateGE” function every game engine tick, which results in the cubes moving in the game engine.

Blender Game Engine does not update armatures in its update loop by default. To get them to update you need to add an Always sensor with “Activate TRUE level triggering” enabled (the ''' icon), an armature actuator, then connect the always sensor to the actuator. Do this for every armature (in Object mode) and the rigs should start moving in BGE.

To use the shape keys, well, you’ll have to do something a bit strange, I think. You could change the Python expressions you’re currently using to driving some object that controls a constraint for some bone. You could, for example, change the python expression to be:"['mouth_open_driver']".location.x= {V}. This would be assuming you have an object called “mouth_open_driver” in the scene that controls some constraint that eventually leads to controlling the mouth pose.