V1.X Documentation

Installation

Configuration

OSC

Advanced usage

Add-ons/Plugins

General top

Delicode NI mate is a program that uses OpenNI standard compliant devices such as the Microsoft Kinect, the ASUS Xtion and the PrimeSense Carmine to track body movement, converting this motion data into standard OSC or MIDI messages that can be received in any client program.

Installation top

On Windows top

If you have previously installed SensorKinect, OpenNI or NITE, it’s strongly recommended that you either uninstall all of them or update them to the latest versions before installing Delicode NI mate.

If you have any other device drivers (like the Kinect for Windows SDK) installed other than SensorKinect you must uninstall these before installing Delicode NI mate. Since December 2012, an OpenNI v2.0 compliant version has been available for Windows machines, which is known to be compatible with the Kinect for Windows device and drivers.

On Windows Vista and Windows 7 it doesn’t matter if your sensor device is connected or not during the install. On Windows XP and Windows 8 you should not have the sensor device connected during the install.

In order to use the non-OpenNI2 version of NI mate on Windows 8 please follow the instructions for Windows 8 below.

Installation options:

  • Fresh install - this option should only be necessary in case something went wrong during a previous installation as it cleans up the NI mate installation folder to make sure there are no corrupted files in the folder. Existing profiles and licence will be saved (these can be removed during uninstallation if wanted).

  • Kinect driver - this option is usually autodetected and will be unselected if you already have the SensorKinect driver installed. If this option is checked you should most probably not change it as your sensor device won’t be usable without the driver.

    If you don’t have an appropriate driver installed and this option is unchecked there could have been a problem with the automatic driver detection and you should select it for the installation.

Windows 8 (non-OpenNI2 version of NI mate)

By default Windows 8 requires that all installed device drivers are signed and doesn’t allow installation of non-signed drivers. The Kinect driver that NI mate uses with OpenNI 1 is not signed so some extra steps are required to enable trouble free installation.

  1. Press “Win+R” to open up the run prompt and enter “shutdown.exe /r /o”.

  2. Press “OK” to restart to the “Choose an option” screen.

  3. Select “Troubleshoot” (icon with tools).

  4. Select “Advanced options” (Icon with checkmarks).

  5. Select “Windows Startup Settings” (Icon with gearwheel).

  6. Press “Restart” to restart the computer to the “Advanced Boot Options” screen.

  7. Select “Disable Driver Signature Enforcement”.

  8. Once the computer starts install NI mate.

  9. Once the NI mate installation is complete plug in the Kinect and start NI mate.

Windows XP

  1. Install NI mate before plugging in your compliant sensor device and make sure to check the Driver, since automatic detection is unreliable under XP. If you plug in the sensor device before installing NI mate you should just click cancel on the driver installation prompts.

  2. Plug in your Microsoft Kinect or similar OpenNI compliant device. Now you should get a prompt for installing drivers for “Kinect Motor”. If you don’t get a prompt try a different usb port.

  3. Select “Not this time” on connecting to Windows Update and select next.

  4. Select “Install drivers automatically” and click continue.

  5. After the “Kinect Motor” is installed you will get similar prompts for “Kinect Audio” and “Kinect Camera” and you should repeat the steps from 2.

  6. After the drivers are installed, and you are using the Microsoft Kinect device, you should blink a green light and you can start NI mate.

On OS X top

Simply drag the NI mate application from the disk image to the Applications folder.

On Ubuntu 12.04 top

Installation

Either

  • Doubleclick on the package and install via Ubuntu Software Center.
    (the installation will complete with the package still showing the “Install” option untill the package database is updated)

or

  • Install via terminal
  1. sudo dpkg -i ~/Downloads/Delicode-NI-mate_1.11-ubuntu_amd64.deb

  2. if you don’t have qt libraries installed this will complain about missing dependencies, the dependencies can be fixed with:
    sudo apt-get -f install

If you have plugged the sensor to the computer before installing NI mate please restart the computer after installation.

Please note that in order to use the connected sensor NI mate needs to stop the default sensor driver in the Ubuntu kernel from being used. This will stop the sensor from working as a webcam.

Uninstallation

Either

  • Ubuntu Software Center -> Installed -> Show nnnn technical items -> Search for “delicode-ni-mate” -> Uninstall (this requires the package database to be updated after installation)

or

  • Uninstall via terminal:
    sudo dpkg -r delicode-ni-mate

NI mate Configuration top

Profiles menu top

All combinations of settings for NI mate are called “profiles”. From the profile menu you can save and load different profiles. You can also export and import the profiles to and from external text files to share them with others.

The currently loaded profile’s name will always be shown in the application title. When you start NI mate the “Default” profile is always loaded. You can use the “Default” profile to save the most common settings that you wish to have active on startup.

  • New - load default values to all settings. This is useful if you want to start building a new profile from a clean set of settings. This also loads the “Default” profile to avoid overwriting a saved profile.

  • Load - Load a previously saved profile quickly.

  • Save - save the current set of settings as a profile that can be accessed quickly later. You can save the profile either with a new name or over an existing profile.

  • Import - you can easily import profiles created by others by importing them to NI mate.

  • Export - you can easily share the profiles you create by exporting them to external files.

Preferences top

The NI mate preferences are global settings that are not part of a profile, so they stay the same even if you load or import a different profile. On Windows you can find the preferences under the Edit menu and on OS X in the application menu.

  • Startup - The configuration window can be shown or hidden on startup. If it’s hidden it can be accessed through the tray icon. NI mate can also check if there is a newer version available at each startup.

    The “Kiosk mode” is intended for long installation use and should be used if NI mate is running as a background program without supervision. It will start a watchdog module together with NI mate, disable the splash screen and start NI mate minimized to the system tray. The watchdog module will keep an eye on NI mate and restart it with the same settings in case of a crash. All crash reporting windows will also be blocked while the watchdog is running.

    **Note:**The kiosk mode is only available with a full licence. On OS X the watchdog will disable crash reporting for all programs as this is a global setting in OS X.

  • General - “Tray on close”: clicking the X button usually closes the program, but with this option you can set it to be minimized to the tray icon instead. On multi-monitor setups there is also a “Screen” selector, which allows to set the screen the ghost image is drawn on.

  • Sensor - “Undistorted depth”: normally the depth image that’s visible in the live view (and sent through syphon and the FreeFrameGL plugin) is distorted in order to match it with the rgb image. However this leaves black borders around the data, which may not be desirable if you only want to use the depth data. This option leaves the depth data undistorted.

    “IR instead of RGB”: this will replace the RGB data from the sensor with the raw infrared data.

    “High speed”: this option will lower the sensor resolution to 320*240 but increase the maximum framerate to 60 fps to increase tracking responsiveness. This option is only available for the Asus Xtion or Primesense Carmine devices.

    Note: changing the above sensor options requires a restart.

    “Read QR codes”: The sensor’s rgb camera is used to read QR codes and the read data is sent to the OSC address “/qr_code”. This OSC message is sent to any enabled OSC output (Full Skeleton or OSC Controller).

    Note: the rgb resolution is only 640*480 and the sensors don’t have any focusing capabilities so the QR code will have to relatively large in order to be readable, for example a 10 cm by 10 cm (4 inch by 4 inch) QR code will work quite well from a 1 m (3 feet) distance.

  • Ghost rendering - The “ghost” is a smoothed outline of the active/all users which can be used either as a click- and see-through overlay image on the desktop or as a great composition input to video mixing software. In the ghost rendering settings you can select the degree of smoothing and which users should be visible in the ghost image.

  • Display ghost - Enables a click- and see-through window that shows the ghost image on top of all other windows. You can set the location, transparency and color of this window with these controls and also set the ghost to fade away on mouse over. The “Draw points and skeleton” will draw the skeleton and other tracked points on top of the ghost window.

  • OSC input - NI mate can be controlled with OSC commands, which are sent to the port specified here and are directed to the OSC path “/NI mate”. See the advanced usage topic for more information about the supported OSC messages.

    Note: this functionality is only available with a full licence.

  • Audio input - in addition to motion data from the sensor device, NI mate can also monitor audio input from any audio input device connected to the system. For example together with a headset microphone this allows for basic mouth motion for virtual characters in real time performances and can help create synchronization times for motion capture. The sent data will be between 0 and 1 and will correspond to the overall volume of the audio input’s left/right channel. The actual OSC settings for this functionality are located in the configuration window’s Full Skelton tab.

  • License - The currently connected sensor device’s ID is shown together with the options to remove the current license or activate a new license.

User tab top

NI mate will designate one users in the sensor device’s field of view as the “active user”. This tab controls the way this user is chosen.

Activation mode

  • Closest - the closest user to the sensor device is always the active user. In this mode the user doesn’t have to perform any poses or special motions in order to become the active user. This is generally desirable, but in this also means that the active user can change very easily if a different person gets closer to the sensor device.Note: The distance measurement slightly prioritizes users closer to the center of the sensor’s field of view, so a user directly in front of the device will be the active user even if a different user is slightly closer but at the edge of the sensor’s field of view.

  • Psi pose - any user wishing to become the active user must perform the “psi pose” with elbows bent at 90 degrees, arms directly to the sides and forearms directly upwards. If an other user wishes to become the active user he/she must perform the psi pose closer to the sensor than the current active user. The benefit from this mode is that other users can’t interfere with the current active user even if they are closer to the sensor device.

  • Wave, raise hand - the user wishing to become the active user can also perform one of these gestures. Wave means simply to wave one hand a couple of times like saying hi to the sensor. Raise hand means simply raising a hand.

  • Stationary - the user wishing to become the active user must simply remain still in front of the sensor. The threshold value adjusts the maximum movement the user can have while still remaining active.

De-activation

The Psi pose, wave and raise hand gestures are available as de-activation gestures that an active user can perform in order to become non-active. Note that when a user becomes active there’s a 5 second delay before the de-activation gesture can be performed. This delay also applies vice-versa (after de-activation there’s a 5 second delay before the same user can become active again) and is there to make it possible to use the same gesture for both activation and de-activation.

Activation area

Activation inside area - If enabled a user can only be active within a certain area in front of the sensor. This works well if you want the user to be able to interact within a specific area (for example a small area marked to the floor).

Activation at corners - If enabled a user can only be active within a certain radius from the used corner locations. This works well if you want the user to be able to interact at specific spots in front of the sensor.

The area or corner points can be defined by manually entering the four locations of the corner points, but a much easier way is to use the interactive calibration by clicking the “Calibrate” button. The interactive calibration allows you to calibrate the area/corners by simply walking to the locations of the corners of the area while guided by on-screen instructions.

During the calibration a top view of the sensor’s field of view is drawn in place of the depth view. This view also contains a circle representing the user doing the calibration.

Once the four corners are defined you can test the area by walking near the area. The user circle will turn green whenever the user is inside the activation area and red whenever the user is not inside the activation area. You can return to this test mode at any time by clicking the “Test Calibration” button.

Note: the activation area/corners takes priority over the activation mode, so if a user wishes to become the active user he/she must first be inside the activation area (or near the corner points in “Activation at corners” mode) before performing the psi pose (in “psi pose” activation mode). In the case that multiple users are inside the activation area the closest one is chosen (in “closest” activation mode).

Activation area coordinates

These coordinates are sent for all users to both Full Skeleton OSC and Controller OSC. The OSC message will be sent to the specified OSC path and the message will be five floats.

In “Activation inside area” mode the first four numbers describe how close the user is to each corner of the activation area. These kinds of coordinates are called barycentric coordinates. For example if a user stands exactly at the location of the first activation area corner the coordinates will be (1,0,0,0).

In “Activation at corners” mode the first four numbers give the users distance to the corner locations.

The last number is the user’s “active id” as a float, which is a persistent index number for all active users.

Note: if the activation area is disabled the activation area coordinates can still be sent (in the format for "Activation inside area mode). In this case the last calibrated corners are used for the activation area coordinate calculation.

Full Skeleton top

This tab contains all of the controls related to full skeleton tracking and OSC messages generated from the tracked skeleton.

  • Enable Skeleton OSC / Skeleton OSC Enabled - no skeleton OSC data is sent if this is disabled.

  • IP - the IP address the OSC data is sent to. This is separate from the IP that’s used for OSC Controller data. If the client program is running on the same computer as NI mate this should be 127.0.0.1.

  • Port - the UDP port the OSC data is sent to. This is separate from the port that’s used for OSC Controller data. This has to match the client programs receiver port. For example for Animata this has to be 7110.

  • Format

    • Basic - The individual joint OSC paths define the whole OSC path and the joint coordinates are sent to this path as 3 floating point numbers.

    • Animata joint - The actual OSC path is “/joint” and the message will consist of the joint OSC path as a string and two floating point numbers describing the x and y-coordinates of the joint.

    • OSCeleton - The actual OSC path is “/joint” and the message will consist of the joint OSC path (string), number 1 (integer) and the xyz-coordinates of the joint (3 floats).

    • Omni - The OSC path can be defined in the OSC paths subtab and the message will be 15 times 3 floats, the xyz coordinates of all the joints in the order: head, neck, torso, left shoulder, left elbow, left hand, right shoulder, right elbow, right hand, left hip, left knee, left foot, right hip, right knee, right foot. The OSC paths for the individual joints are not used in this mode.

    • Orientation - The OSC path is defined by the individual joint OSC paths and the message will be 4 floats describing the orientation quaternion (in the order w,x,y,z) of the joint.

    • Basic + Orientation - The OSC path is defined by the individual joint OSC paths and the message will be 3 floats describing the xyz-coordinates of the joint followed by 4 floats describing the orientation quaternion (in the order w,x,y,z) of the joint.

OSC Paths

The individual OSC paths in this tab are used to describe the destination for each of the join OSC messages. The exact way in which these paths are used depends on the OSC format.

The left and right audio input OSC paths are used for the overall volume of the left and right channels for the audio input device chosen in the preferences. These will always be sent to the exact OSC path specified in the text box and the message will be three floats regardless of the skeleton format. This data will be between 0 and 1 on the x-axis and zero on the y- and z-axes.

Note: You can disable sending any specific value by setting it’s “OSC path” to blank.

Note: When skeleton data is enabled the tracked skeleton will be drawn over the ghost and in the depth view.

Skeleton Settings

  • Mirror left and right - send the data belonging to the right hand as data from the left hand and vice versa. This can be useful for example for capturing motion while at the same time monitoring the motion from the screen, since people area accustomed to watching their own motions in the mirror.

  • Allow multiple users - track multiple users simultaneously. The user number assignment is permanent and a new user is always assigned the lowest available number. What this means in practice is that the first active user will be given the number 1, the second the number 2 and so on. If user 1 deactivates then the second user will still be number 2 and the next activated user will become number 1.

    In Basic, Orientation, Basic + Orientation and Animata formats the OSC name is appended with “_[user number]”. In OSCeleton format the integer number is the user number. The Omni mode doesn’t currently support multiple users.

  • Send only torso location - sends the joint location only for the “torso”, and orientation data for all joints. This is usefull in the case that the target character has different body proportions than the actual user as you can move the character through the torso location and animate the character by applying the rotations character skeleton bones. Note: This option is only available with the OSC Format “Basic & Orientation” and requires a full licence.

  • Smoothing - The raw motion data can be quite noisy at times, so it’s advisable to apply some smoothing to the data before it’s sent to the client program.

  • Confidence - The tracking algorithm gives confidence estimates for all of the recognized joints. This value allows for filtering out the joints that are not very well tracked. When the skeleton is drawn over the live view image joints that have a confidence below this value will be drawn in red.

  • Coordinate mode - For motion capture purposes the actual real world coordinates of the joints are required, but if you want to match the sent data to a video image it becomes difficult to match the sent data to the image due to perspective differences. With the “Screen” coordinate mode the joint’s x- and y-coordinates are sent in untransformed screen coordinates starting from the top left corner in the range 0-1.

  • Keep fixed - Especially for real-time performances it can be useful to be able to fix the skeleton data to a certain point (for example to a speaker stand in a virtual environment). The chosen point or height will always be moved to zero and all other sent values will be modified accordingly.

  • Autocorrect orientation - Try to straighten the tracked coordinates automatically to be level with the ground even if the sensor device is tilted (this feature does not work with the Xtion PRO as it doesn’t have an accelerometer).

  • Pre/Post Offset - These values add offsets in meters (pre) or scaled units (post) to the data before it’s sent. This is useful if the client program doesn’t have means to move the origin of the received data internally.

  • Scale - These values scale the data before it’s sent. This is useful if the client program doesn’t have means to scale the received data internally.

    Note: Animata uses a pixel based coordinate system, that’s also inverse from the one used in NI mate. Suitable scaling values for Animata are therefore around x=-[x resolution of Animata window], y=-[y resolution of Animata window], z doesn’t matter as only x and y are sent for Animata.

MIDI Controller top

You can use NI mate as a MIDI controller for any software or hardware that supports MIDI. Different coordinates and features of the active user’s hands, and body are mapped to MIDI control change messages, which can be sent to any open MIDI input port.

The “controller” tracking uses an algorithm that differs from the one used for skeleton tracking. This algorithm doesn’t require any calibration and can be more robust for tracking the upper body in cases where the skeleton can’t be tracked well.

  • Enable MIDI Controller / MIDI Controller Enabled - no MIDI messages are sent if this is disabled.

  • Default port and channel - These settings can be used to quickly change the output of many tracked values to a certain port and channel (the tracked value has to have “Default” as the selected port and “D” as the selected channel to use the default values).

  • Smoothing - The raw motion data can be quite noisy at times, so it’s advisable to apply some smoothing to the data before it’s sent to the client program. This smoothing is applied to both MIDI and OSC Controller data.

  • Motion range - These values define the range of motion which will be mapped to the OSC message value range. The motion ranges of x, y and z coordinates are measured in millimeters, for angles (hand angle, body twist, bow and lean) in degrees and for velocities (left/right hand and body) in millimeters per second.

  • Message range - The specified motion range is mapped to a message range from zero to the specified message range value (up to 127). For example a hand distance range from 200 mm to 400 mm and a message range of 1 means that when the hands are closer than 30 cm the sent value will be 0 and when the hands are further than 30 cm the sent value will be 1.

  • Message destination - This defines the control change, channel and port the MIDI message will be sent to.

  • Solo buttons - These can be used to teach the midi messages of different joint coordinates to other software easily. When the Solo button is active only the midi messages corresponding to that joint coordinate are sent. This allows you to activate a “learn” function in the client software and automatically map the NI mate message to a control in the client software.

Note: when the OSC Controller is enabled the tracked points will be drawn over the ghost and in the live view: “H” for head, “B” for body, “L” for left hand and “R” for right hand.

OSC controller top

The OSC Controller mostly functions in the exact same way as the MIDI Controller except that it sends single valued OSC messages to the specified message paths instead of MIDI messages. All of the sent messages are single values except for the Exact values for which the message is three floats.

The “controller” tracking uses an algorithm that differs from the one used for skeleton tracking. This algorithm doesn’t require any calibration and can be more robust for tracking the upper body in cases where the skeleton can’t be tracked well.

  • Enable OSC Controller / OSC Controller Enabled - no OSC Controller data is sent if this is disabled.

  • IP - the IP address the Controller OSC data is sent to. This is separate from the IP that’s used for Full Skeleton data. If the client program is running on the same computer as NI mate this should be 127.0.0.1.

  • Port - the UDP port the OSC data is sent to. This is separate from the port that’s used for Full Skeleton data. This has to match the client programs receiver port.

  • Smoothing - The raw motion data can be quite noisy at times, so it’s advisable to apply some smoothing to the data before it’s sent to the client program. This smoothing is applied to both MIDI and OSC Controller data.

  • Motion range - These values define the range of motion which will be mapped to the OSC message value range. The motion ranges of x, y and z coordinates are measured in millimeters, for angles (hand angle, body twist, bow and lean) in degrees and for velocities (left/right hand and body) in millimeters per second.

  • Message range - The specified motion range is mapped to a message range from zero to the specified message range value (up to 127). For example a hand distance range from 200 mm to 400 mm and a message range of 1 means that when the hands are closer than 30 cm the sent value will be 0 and when the hands are further than 30 cm the sent value will be 1.

  • Message path - This defines the OSC path where the OSC messages will be sent to.

  • Exact data - This sends the 3d coordinates of the users head, body and hands to the client program. In many cases skeleton data could be used in stead of exact data, but the different tracking algorithm used for exact tracking makes it more robust in some cases where not enough of the body can be seen for proper skeleton tracking.

    • Mirror left and right - send the data belonging to the right hand as data from the left hand and vice versa.

    • Send screen coordinates - For motion capture purposes the actual real world coordinates of the joints are required, but if you want to match the sent data to a video image it becomes difficult to match the sent data to the image due to perspective differences. With this option exact x- and y-coordinates are sent in untransformed screen coordinates starting from the top left corner in the range 0-1.

Note: when the OSC Controller is enabled the tracked points will be drawn over the ghost and in the live view: “H” for head, “B” for body, “L” for left hand and “R” for right hand.

MIDI Triggers top

The MIDI triggers are very responsive areas around the closest active user, which can send out MIDI messages when the user’s hand or any other body part moves through the trigger area. These trigger areas are great for starting a sample or toggling an effect on or off in a client program.

There are three independent layers of triggers with each layer containing different triggers with the following attributes:

  • Type - The over all shape of the trigger layer. The difference between “Circle” and “Fixed Circle” is that “Circle” follows the user’s center while “Fixed Circle” is always centered directly in front of the sensor.

  • Trigger count - How many triggers this layer has.

  • Start & end angle - The angles where the trigger areas start and end around the user.

  • Radius - How far away the triggers are from the user’s center.

  • Height - Add a height offset to the triggers.

  • Mode - “Trigger” sends the value 127 when the trigger area is activated and the value 0 when the trigger area is de-activated. “Toggle” sends the value 127 when the trigger area is activated for the first time and the value 0 when the trigger area is activated a second time. “Velocity Trigger” sends the velocity of the triggering motion as the value when the trigger area is activated and the value 0 when the trigger area is deactivated.

  • MIDI for 1st trigger - Set the MIDI message for the first trigger area. The following areas will be sent to the MIDI messages following the first. For example selecting control change 11 as the first trigger’s MIDI message will make the second trigger send control change 12, the third to send control change 13 and so on. In addition to control change messages you can also send note on/off and program change messages.

  • “Include in” - makes it possible to draw the triggers over the live feeds which are selected in the “Live view” tab.

Live view top

This tab shows the current live depth or rgb data feed from the sensor. The different feeds can be selected from the buttons on the left edge of the view:

  • Depth & extras - All detected but not active users are drawn in blue and the active user is drawn in green in the depth view.

  • Depth - Pure depth feed without any user identifications.

  • RGB - Pure RGB feed.

  • Ghost - A smoothed version of the active users silhouette.

  • Encoded depth - A view where the depth values have been encoded to the different rgb channels in order to keep all of the depth information from the sensor. The depth in millimeters can be calculated with
    depth = red << 5 | green or
    depth = red * 32 + green,
    where “red” and “green” are the first and second rgb color values of the pixel [0-255]. The blue color contains the id of a detected user starting from 1 (0 means not a user pixel).

  • Color ID - Each detected user is given a different color for easy identification based on the color chart below. You can use this mode to extract a single person easily in a client program.

The selected feed is sent through Syphon on OS X and to the NI mate FreeFrameGL plugin on Windows. You can also select a secondary output feed by shift clicking a feed button. The Syphon server names for the feeds are:

application name: “Delicode_NI_mate”
primary feed server name: “Delicode Server”
secondary feed server name: “Delicode Server 2”

The “A” button sets the ghost view as the alpha channel for every output feed. This makes it super easy to extract the user/users from the background.

The full screen button makes the selected live view to fill the whole screen. To return from full screen mode press the ESC-key.

The “D” (for detached) button sets the live view to a separate window which can be viewed while changing the NI mate parameters.

The Tilt angle controls the sensor device’s tilt motor. Some care must be taken when operating this: The sensor tilt angle is always relative to the horizon (black line on the slider), not relative to the sensor’s base. Tilting the sensor past 30 degrees relative to the base can cause harm to the tilt motor since there are no automatic safe-guards. Disclaimer: Delicode Ltd is in no way responsible for any direct or indirect harm done to your equipment done by the use of this software. Use this functionality at your own risk.

When the MIDI or OSC Controller is enabled the live view shows the tracked points as abbreviated letters: “H” for head, “B” for body, “L” for left hand and “R” for right hand. When Full Skeleton is enabled the live view shows the tracked skeleton as a wireframe over the active user. The skeleton joints that have a confidence below the threshold set in the Skeleton Settings will be drawn in red.

The live view is useful for verifying that the sensor can see the users properly and for noticing problems when things don’t seem to work normally. For example large objects in the sensor’s field of view can some times be detected as users. In these cases it’s best to move the sensor slightly so that these objects are not in the sensor’s field of view anymore.

Processing top

Processing is a Java based open source programming language for creating images, animations and interactive graphics. This documentation mostly covers only the NI mate integration side of things, so if you want to learn more about Processing itself there are some great tutorials at the Processing website.

Installing Processing

The processing integration has been developed for Processing 2.0 beta 7 or higher. If you haven’t yet installed Processing on your computer or you have an earlier version you can download Processing for your platform from here. You can install Processing in to any directory by simply extracting the installation archive to your computer, but we suggest the following locations:

Windows: c:\Program files\ (or c:\Program files (x86)\ in the case of a 32 bit machine)

OS X: Applications

Linux: home directory

Configuring NI mate for Processing

  1. After installing Processing start NI mate and go to the Processing tab.
  • On Windows and Linux NI mate will now ask you for the location of the Processing installation. Please select the directory where the processing executable (and processing-java executable) are by clicking the “Select Processing Folder” button.
  • On OS X you should first launch the Processing.app and then from the Tools menu select “Install processing-java”. After doing this you can verify the processing-java tool was installed correctly by clicking the “Verify Processing Installation” button. Please note that if you move the Processing.app or install a new version of Processing it’s possible that you have to install the processing-java tool again from the Processing.app Tools menu.
  1. NI mate uses the oscP5 library to communicate between NI mate and the Processing sketch, so you have to install it from the oscP5 website. You should extract the library to the “libraries” folder under the processing sketchbook folder. To find out the sketchbook folder location you can start the Processing application and look the path up from “File -> Preferences -> Sketchbook location”.

    After downloading and installing the library, please select the processing sketchbook folder with the “Select Processing Sketch Folder” button.

    Note: NI mate will automatically create an “NI_mate_processing” folder in the sketchbook folder when a Processing sketch is run from within NI mate. Any modifications to this folder or the files inside the folder will be overwritten each time the sketch is run.

The Processing development interface

After a succesfull configuration you are presented with the Processing development interface.

  1. The code editor is where you write the actual Processing code. You can add more tabs by pressing the “+” at the top of the code editor and rename/remove all but the “Main” tab from the small arrow next to the tab name. The code in each tab will be included in the sketch with the only restriction that the “Main” tab should contain all library imports and the “setup()” and “draw()” functions. All of the code in the tabs is also saved into the NI mate profile when the profile is saved.

    The code is syntax highlighted and you can get help on any Processing function/variable by pressing “F1” when the cursor is over a Processing function/variable or by right clicking on a Processing function/variable. This will open up the Processing reference in a browser. On OS X you need to have a browser window open before trying to open the help.

    You can temporarily mark up a certain line in the code by clicking on it’s line number in the left edge of the code editor. This makes it easy to remember an important place in the code while you’re coding. The marks are not saved in NI mate profiles.

    Note: for long coding periods it’s useful to activate the “Pause” button in NI mate. In addition to stopping all data sending NI mate will also lower the sensor update frequency so that the CPU doesn’t have to work that hard to analyze the sensor data. Just remember to un-pause NI mate when you run your sketch or otherwise there won’t be any data sent to the running sketch.

  2. The Automatic variables list contains all of the data that NI mate can send to Processing as ready made Processing variables. You can double click on any variable name to insert it to the current cursor location. The variable type is shown to the right of the variable name. When the sketch is run or exported NI mate will add the necessary variable definitions and update code behind the scenes before compiling the sketch. Each NI mate processing sketch should contain the “setup()” and “draw()” functions if any automatic variables are used.

    The last part of the automatic variable initialization is done in the beginning of the “setup()” function after the “size()” function (which by Processing guidelines should be the very first line in the setup function if the “size()” function is used at all). You shouldn’t use any of the automatic variables in your code before this.

    The “skeleton” array contains all of the tracked Full Skeleton joint positions as PVectors. To access individual joints you can use the individual joint PVectors under the “skeleton” array.

    The different “control” arrays contain the data from each of the OSC Controller subtabs.

    The “feed1” and “feed2” PImages are updated in real time from the feeds selected in the Live view tab.

    The “area_co” array contains the area coordinates from the User tab.

    Note: the automatic variables are generated from the “Full Skeleton” and OSC Controller tabs for all of the values that have a non-empty OSC path.

  3. The Files list contains all of the asset files (images, movies etc.) that will be directly available to the Processing code. You can add asset files by clicking the “+” button above the files list or remove asset files by selecting them from the list and clicking the “-” button above the files list.

    Note: all of the asset files will be copied to the “sketchbook/NI_mate_processing/data” folder when the sketch is run.

  4. The Run button will compile the current sketch and run it. Any compile errors and messages will be printed to the NI mate Log tab with the last line showing on the left side of the Run button. You can stop a running sketch by clicking the usual “x” on the sketch window or by pressing “ESC” if you’re running a “Full screen” sketch.

    Note: the Processing “Full screen” option creates a borderless window. You still have to set the sketch size to match the screen resolution to achieve actual full screen behavior.

  5. The “Find…” tool allows you to find or replace text in the Processing code.

  6. The “Export…” tool allows you to export the processing sketch as a self-contained application (Please note that NI mate still needs to be installed on the machine where you want to run the exported application) for any supported platform. On export a folder with the set export name will be created in the Processing sketchbook folder with subfolders for each exported platform.

    The exported application will launch NI mate automatically in “silent mode” (only the tray icon will be visible) with the settings that were active in NI mate at the time of exporting. You can access the NI mate settings from the tray icon, but modifying them is not recommended as NI mate will be closed automatically when the sketch application closes.

    For more info and tips on the general Processing export functionality and restrictions please see the notes on the Processing wiki.

    Note: on Windows and Linux you will need to install Java to run the exported application or you can copy the “java” folder from Processing to the same folder as your exported application.

    Note: you should close NI mate before running the exported application as two NI mate instances can’t run at the same time.

    Note: the export functionality is only available with a full NI mate licence.

Log tab top

The log tab shows all the general events and errors that have happened since the program started. You can enable which messages you want to catch in the log:

  • Program - General program messages, like startup and enabling OSC or MIDI.

  • User - Detection of new active users and deactivation/losing active users.

  • Sent OSC - All sent OSC messages and their destinations. Notice that the amount of messages generated by NI mate can be so high that logging them can slow NI mate down. This should not be enabled under normal usage.

  • Sent MIDI - All sent MIDI messages and their destinations. Notice that the amount of messages generated by NI mate can be so high that logging them can slow NI mate down. This should not be enabled under normal usage.

Tray icon top

The NI mate tray icon can give hints of the tracking activity without taking screen space. On Windows, the Microsoft Kinect LED also mirrors this behavior and is marked within [square brackets] below (we are working on bringing the same functionality to OS X also). The tray icon will be:

  • Gray [off], when no users are detected.

  • Blue [orange], when there are users, but no active user.

  • Steady green [steady green], when there is an active user.

  • Blinking green [blinking green], when the active user’s skeleton is being tracked.

By clicking on the tray icon you can either quit the program or get to the configuration quickly if it’s hidden by other windows.

If NI mate exits normally the Microsoft Kinect’s LED returns to the default blinking green state.

Note: on Windows the tray icon could end up hidden automatically if there are a lot of icons in the taskbar. In order to make the icon always visible when NI mate is running click the small up arrow at the beginning of the taskbar and select “Customize…”. Find Delicode NI mate from the list, choose “Show icon and notifications” from the dropdown next to it and click “OK”.

OSC top

Open Sound Control is a content format for messaging among computers, sound synthesizers, and other multimedia devices that are optimized for modern networking technology.

In order to use OSC the client program needs to support receiving OSC data. For example Animata supports OSC natively, and Blender can support OSC through the NI mate add-on.

If the client is running on the same machine as NI mate you should set the OSC IP address in NI mate to it’s default value of 127.0.0.1. The OSC port number has to be set to the same number the client program is listening to (for Animata this is 7110, for Blender this can be chosen freely but must match the port in the Delicode NI mate add-on).

Advanced usage top

Startup parameters top

NI mate can be launched using certain command line parameters to load a profile or preferences:

  • -profile [file path] - Import a specific profile from a file.

  • -load [profile] - Load a specific profile on startup (valid names are listed in the “Profile -> Load” menu.)

  • -preferences [file path] - Import specific preferences from a file.

  • -clean - Load the default preferences and profile.

  • -clean_preferences - Load the default preferences.

  • -clean_profile - Load the default profile.

  • -help or -h - Display a popup describing the startup parameters.

Note: parameter paths with spaces should be enclosed in “” quotes.

Note: this functionality is only available with a full licence.

In addition to these parameters you can also load any exported profile or preferences by renaming the exported file to “.nimate” and simply doubleclicking the file.

When a startup parameter which loads a profile, preferences or an .nimate file is used NI mate goes into silent mode and doesn’t display the normal splash screen or the configuration window. To quit NI mate or access the configuration in this case use the NI mate tray icon.

On Windows this can be achieved by:

  • starting NI mate from the command line (for the default installation path):

    “C:\Program files (x86)\Delicode\NI mate\Delicode_NI_mate.exe” [parameters]

  • editing a shortcut to NI mate and adding the parameter(s) at the end of the shortcut’s target.

On OS X this can be achieved by:

  • starting NI mate from the terminal (NI mate should be in Applications):

    open -a “Delicode_NI_mate” --args [parameters]

  • creating a launcher application using the Automator:

    1. Start Automator and select to create an “Application”.

    2. Double-click “Run Shell Script” from the Library/Utilities folder

    3. Replace the default text “cat” with:

      open -a “Delicode_NI_mate” --args [parameters]

    4. Save the Automator application to anywhere you like.

    5. Start NI mate by double-clicking the saved automator application.

OSC commands top

NI mate can receive OSC commands from an external program. Every OSC control message should be sent to the address “/NI mate” and should contain one string value:

  • quit - Quit NI mate.

  • load profile_name - Load a previously saved profile (the accepted names can be found in the NI mate profile->load menu).

  • import profile/path.config - Import an exported profile file.

  • restart [profile] - Restart NI mate with an optional profile name or a profile file path argument.

  • ping - Show a popup as confirmation of received OSC command.

To enable this functionality enable the “OSC input” from NI mate preferences and select a suitable OSC port that NI mate will listen to.

Note: this functionality is only available with a full licence.

Add-ons/Plugins top

Blender top

Addon installation

1. If you haven’t already, download the Delicode NI mate addon to somewhere on your computer.

2. Launch blender and select “User Preferences…” from the “File” menu and select the “Addons” tab

3. Click “Install Addon…” from the bottom of the preferences window and browse to the location where you downloaded the Delicode NI mate addon. Select the file “animation_delicode_ni_mate.py”, and click “Install Addon…” from the top left corner of the file selection window.

4. Now the addon is installed and you can always find it in the “Addons” tab of the user preferences

Basic addon usage in the 3d view

1. Activate the Delicode NI mate addon from the user preferences window’s “Addons” tab by checking the small box on the right side of the addon. The Delicode NI mate addon is in the “Animation” addon category.

2. Close the user preferences window and show the 3d view toolbar if it’s hidden (t-key) to show the newly added “Delicode NI mate” section. You might need to refresh the toolbar by clicking on it to make the new section appear.

3. Start up NI mate and make sure the port shown in the addon matches the one in NI mate’s Full Skeleton tab. If the ports don’t match it’s ok to change either, but make sure the new port isn’t being used for anything.

4. If NI mate is running on the same computer as Blender make sure that the OSC IP address is set to 127.0.0.1.

5. Enable Skeleton OSC in NI mate and set the exact name of a Blender object to one of the OSC paths under the Full Skeleton tab. Set for example “Cube” (assuming you have an object named “Cube” in the Blender skene) to the OSC path for the right hand. Note that for Blender you shouldn’t use the slashes “/” for the osc path, but instead use the exact blender object name.

6. Start listening to the data from NI mate in blender by clicking on “Start” in the NI mate addon and move somewhere the sensor device can see you. You should now be able to move the object you named in NI mate with the specified joint. In the case of the example you should be able to move the “Cube” object with your right hand.

7. In order to record the motion activate “Automatic keyframe insertion for Objects and Bones” from the timeline (the small record button). Set a suitable frame range for the timeline and click “Play” from the timeline. The object’s motion is now automatically recorded.

8. To stop recording press the pause button that’s in place of the play button. In order to stop receiving OSC data from NI mate press “Stop” in the NI mate addon.

Basic addon usage in the game engine

1. Open the Delicode NI mate addon as a text file to Blender’s text editor.

2. Go to the logic editor view and add an “Always” sensor for a game object that isn’t destroyed at any point, for example the main camera. Activate “TRUE level triggering (pulse mode)” (the ‘’’-button) for the sensor.

3. Add a “Python” controller and select “animation_delicode_ni_mate.py” (the name might be truncated to “animation_delicode_ni” in the script dropdown.

4. Connect the always sensor to the python controller by dragging a line between the sensor and controller dots.

5. Add an integer “Game Property” with the name “NImatePort” to the object with the sensor and the controller and set it’s value to the OSC port you want to use. (If you don’t set this property the script will listen to OSC on the default port 7000.)

6. Start up NI mate and make sure the port set as the game property “NImatePort” matches the one in NI mate’s Full Skeleton tab. If the ports don’t match it’s ok to change either, but make sure the new port isn’t being used for anything.

7. If NI mate is running on the same computer as Blender make sure that the OSC IP address is set to 127.0.0.1.

8. Enable Skeleton OSC in NI mate and set a name of a Blender object to one of the OSC paths under the Full Skeleton tab. Set example “Cube” (assuming you have an object named “Cube” in the Blender skene) to the OSC path for the right hand.

9. Click “Apply Changes” in NI mate to apply the new settings

10. Press “p” in the 3d viewport to start the game engine and move somewhere the sensor device can see you. You should now be able to move the object you named in NI mate with the specified joint. In the case of the example you should be able to move the “Cube” object with your right hand.

11. Press “ESC” to exit the game engine.

Animata top

The incoming OSC port for Animata is 7110, so set this as the port in the Full Skeleton tab. You should also enable the “Animata joint” skeleton format in the same tab.

Animata uses a pixel based coordinate system, that’s also inverse from the one used in NI mate, so the skeleton data has to be scaled accordingly in the Tracking tab. Suitable scaling values for Animata are therefore around x=-[x resolution of Animata window], y=-[y resolution of Animata window], z doesn’t matter as only x and y are sent for Animata. After this you can adjust the positioning of the sent data using the pre- and post offsets.

Maya top

The NI mate Receiver plugin for Maya lives in the Maya toolbar. To install the plugin for the first time follow these steps:

1. Extract the plugin zip archive to “/Library/Preferences/Autodesk/maya” on OS X or “/My Documents/maya”. After extracting the maya/scripts folder should contain “NImateReceiver.mel” and “NImateReceiverForMaya.py” files and the maya/prefs/icons folder should contain NImateReceiverForMaya.ico.

2. Open Maya and in the MEL command line write “NImateReceiver” without the quotes and press enter.

3. A popup should open. This allows you to select the shelf on which you want to create the NI mate receiver tool. After selecting the shelf from the menu click OK. You should now have the NI mate receiver tool in the shelf you selected.

Doubleclicking the tool button opens up the receiver preferences and single clicking starts receiving data from NI mate (the tool icon should turn green when it’s receiving).