Live motion capture and object interaction with Blender + Ni-Mate


#1

Hi all

I’ve been experimenting with the Blender Game Engine, and need some advice.

I want to control my character with a Kinect and the Ni-Mate add-on, and allow it to interact with other objects in game mode - stand-alone player if possible.

I have rigged a character and hooked it up to the Kinect using the Ni-Mate python script. So far so good, it works well.

Then I wanted to have my character interact with other objects, so I started doing simple game tutorials to find out how the BGE worked. That went well too.

Then I started trying to bring my character into the game environment, and hit various problems and crashes which I won’t go into. I am prepared to continue struggling through this, but I need some advice as to whether this is advisable or possible. Has anyone been doing work like this, and had success?

I would appreciate some feedback!


#2

The blenderartists.org forum “Python support” might be a better place to ask about BGE things:

https://blenderartists.org/forum/forumdisplay.php?11-Python-Support

You may want to include some minimal examples when you ask a question there, with a .blend file that displays what kind of problems you’re having.


#3

Thanks Jesse. I did also post a question on the forum under game engine - I just checked and they have responded with some suggestions.
As a matter of interest, have you guys done any experimenting with live capture and object interaction using your plugin?


#4

I think all the work we’ve done has largely been in capturing the skeleton / face data into Blender objects and then using them to drive objects in the scene with constraints, or to create new objects in the scene such as in the “hands” demo: Blender Conference 2015 RealSense demos and slides available


#5

Thanks Jesse, I’ll take a look at that demo. I don’t have a RealSense camera to test with, so I guess I won’t be able to run the demo blends.


#6

Could anyone help me figure out what I’m missing?
I can receive the kinect data and move my rigged character around in object mode, but as soon as I hit Play it freezes. The whole aim is to have all the functionality of the game environment, with the character being controlled by the kinect.

I have a feeling it’s to do with the NI-Mate game logic, but I’m really stuck. I’ve spent weeks climbing the Blender mountain which is very steep indeed, and it seems I’ve been doing it all for nothing if I can’t get over this hurdle.

(I’ve added the actuator suggested in this post)


#7

If you post your .blend file I can take a quick look at it to see if there’s something going on with the NI mate BGE logic, but unfortunately it may take a while until I have the time to do this.


#8

Thanks Jesse, I appreciate it. I’m going to try a little more experimenting first - I have some intermittent movement now - it may be that I had the logic applied to the capture armature and not the result armature.
My aim is to have a standalone game that automatically receives from Ni-mate - but the only way I can find is to press “start” from within Blender and then hit the game simulator. Do you think what I want to do is feasible?


#9

There’s the option to create a standalone executable: https://docs.blender.org/manual/en/dev/game_engine/blender_player.html

I think you may need to add a few lines of Python that automatically start the BGE receiver things.


#10

Yes, I’m aware of the player and executables, I just imagined there would be a need to be some extra code to start the receiver automatically as you say. Not being a programmer, I wouldn’t know where to start :anguished:


#11

Just a quick note to say I’ve gone back to your simple bunny blend to test. I found that there’s a tiny glitch when playing the game (Blender 2.79 on MacBook Pro OS 10.12.6).
If I start the Ni-mate plugin and then hit P, nothing happens. I have to escape and then hit P again, and it works. It works both in the embedded and stand-alone player.


#12

Hi Jesse

I’m not getting much further with my problem. I still cannot attach anything to an armature and get it to play in game mode. I take your mocap_updated.blend file and add a simple cube and parent it to the armature. The cube moves around in the 3D view, but when I hit play, it stops. I’ve added a spinning cube just to make sure that the game isn’t frozen. I feel like I’m missing something really simple - it means I can’t apply your mocap file to any other character!

I’ve also been trying to test with the simple bunny blend, but I have some weird issues. I’ve adjusted the skeleton names in my profile. The bunny gets completely distorted in play mode, but looks OK in the 3D viewport. I haven’t modified your file at all.

Here is my profile file for the bunny: https://d.pr/dajL70

This is the simple test file trying to move the cube: https://d.pr/eFGeo
Here is what happens when I hit play: https://d.pr/llEk4

On a side note, how do I stop the bunny’s arms going through it’s body?

I am really stuck - I got a bit of help on blenderartists.org, but no-one is able to solve this for me, as most people contributing don’t work with Ni-Mate. I would appreciate your thoughts when you have a chance.


#13

I tested your .blend file and it seemed to work by doing the following:

  • I created an empty cube and added the NI mate game logic on it (using the upper button - the lower one is for live feeds, and I think this button should be renamed to make it more clear)
  • I deleted the empties from the scene and received a few frames in the “cubes” mode with the plugin, to make it easier to see the objects in the game engine.
  • Pressed “P” in the game engine render mode
  • The cubes moved with NI mate data
  • Selected some of the bones in the rig and constrained them with the “copy rotation” constraint to some of the cubes
  • Added the actuator stuff for the rigs
  • Added a couple of cubes that followed the armature
  • Tested the armature + cubes move in normal receiving mode outside BGE
  • Started BGE, the cubes moved with the armature in the game engine

#14

Thanks Jesse, I appreciate you taking the time to test this.
I followed your steps, and found that the cubes move in the embedded player, but not in the standalone player. Is this the same for you?

Could you clarify: “added the actuator stuff for the rigs”? Do you mean the game logic needs to be added for each element in the rig?

If I apply the same process to an existing rig (as shown in your tutorial), what do I need to do differently to make it work in the BGE?


#15

Yes, the actuator stuff refers to adding the BGE game engine logic to all armatures (armatures, not bones). The game engine does not update armatures by default.

I didn’t try the standalone player, but we’ve done some projects with the plugin and standalone executables. As long as there’s an object in the scene that contains the NI mate plugin game engine logic (created with the button) the standalone executable should start receiving data the same way the Blender’s internal game engine does. If the internal game engine works you should be set up. It’s possible something has changed in how the standalone executables work these days. We have also tested this on Windows - it’s possible something on Macs works differently.


#16

Thanks Jesse. I’ll try that out.
Weirdly, the bunny demo file works in both standalone and embedded mode, but my simple test doesn’t. So I don’t think it’s a Mac thing. It must be something odd in my setup. I’m trying to simplify everything to diagnose my problem.

Going back to an earlier issue in this thread - when I run your simple bunny it is totally distorted, though I haven’t changed anything in the .blend file. I’ve changed the Ni-Mate profile to reflect the OSC joint names, but that hasn’t solved it. Do you have an updated profile that works with this blend? The original link is missing, but is for an older version anyway.


#17

We haven’t worked with the Bunny since NI mate 1.xx days, so there isn’t an updated version. Renaming the OSC joint names should be all you need to do in NI mate, so most likely something in the rig itself is not working with the latest Blender version. Another possibility is that some orientations from NI mate are now flipped compared to what they were earlier. Looking at the screenshot, it could be that you need to flip the axis or change the axis entirely for some of the constraints, most likely along the Blender Z axis (up).


#18

Thanks Jesse. I also found that setting the sensor origin to “sensor” helps with positioning the character.
The weirdest thing is happening though - in the embedded player the mesh is totally deformed as in the screenshot, but in standalone player it is perfect! I thought it might be the orientations, but can’t explain why it behaves differently in the two players. In any case, it’s the standalone one I need to work.
Thanks again for responding to this rather obscure and outdated issue :grin:


#19

A follow up and possible answer to my problem:

My mesh only worked (i.e. follows the Ni-Mate data) when in the 3D viewer, but wouldn’t work when in game play mode. I discovered that if I renamed the vertex groups to match the same names as the armature/empties, it works. So even if I just added a cube, the vertex group would have to match one of those names to move in game play mode. Does this make sense?


#20

Jesse, I could really use your help. I’ve just hired a Blender expert to help me and together we cannot solve this. We can get the model to move in the 3D viewport, but not in game play mode. I have shared your instructions with him (he can also get the cubes to move in game play, but nothing else).
We take a model and follow Janne’s tutorial on retargeting. It works fine but not when we hit P.
We are trying to figure out why the bunny works and not the retargeted model and we cannot.
I’m sharing the model we have retargeted and I’m pleading with you to help me to get it moving in standalone game play mode!
Thanks…