Documentation: User interface


#1

Setup

The setup page lists the current sensors and assorted network related settings.

  • Default IP and port: The IP address and port specify where all out-going OSC traffic is sent to. In most cases leaving 127.0.0.1 (localhost, the computer you’re currently using) with the default port of 7000 are the desired settings. If you’re using a different computer (either in LAN or somewhere on the Internet) you can insert their IP address and port here. Make sure that the receiving computer, if not localhost, has the correct UDP ports forwarded.

  • Sensors: All currently connected sensors are listed here. The status of each sensor as well as some controllers are also listed.

    • Sensor: The sensor’s ID is listed here. If a dummy sensor is used, the sensor does not have an ID and the sensor’s name is displayed instead.

    • Status: This describes the sensor’s current status. There are 5 modes:

      • Not used: The sensor has been detected in an USB port, but the user has not yet clicked the “Use” button to start using the sensor. These sensors aren’t doing anything yet, but NI mate knows they exist.
      • Running: The sensor is functioning normally and outputs data as specified in the various component pages for that sensor.
      • Stopped: The sensor has been initialized and has been running earlier, but it has been stopped for some reason.
      • Dummy: It is possible to simulate a sensor device without actually having one connected to your computer. A dummy sensor will playback earlier recordings and sends this recorded data to NI mate just as a real sensor would. Dummy sensors are useful for testing NI mate or for using some recorded data in those cases where live data is not suitable. Dummy sensors don’t have any of the feeds and currently only contain skeleton tracking data.
      • Error: Some kind of error happened when starting or running the sensor. More information is displayed in the log page.
  • Add sensor: Clicking this button for some sensor family will add a new dummy device of the specified type.

  • Rescan: This button will check if any new sensor devices have been connected or if some have been removed. If a sensor device is detected and a dummy device of the same type exists, then the dummy device will be converted into a real sensor device. This is desirable in the cases where the sensor’s settings need to be specified before the real sensor starts up.

Motion recorder

The Motion Recorder page gives an overview of existing sensor recordings and allows choosing if a sensor is currently using its live data as input or if it should use an earlier clip.

  • Sensor list: All current sensors are listed here. Their playback and record settings can be changed using the controls in this list.

    • Record arm: Clicking this button while a sensor is in live mode marks the sensor to be recorded once the main timeline’s record button is pressed. While recording, NI mate collects tracking data from the sensor and stores this in a .nirec file once the recording completes.

    • Mute: When playbacking, make NI mate ignore all data from this sensor.

    • Solo: When playbacking, only allow this sensor’s data to be received by NI mate.

    • Source: Choose what kind of input data the sensor uses. Live mode will use the sensor in its normal operation mode, getting the data from the sensor normally. Choosing a clip will turn the sensor into playback mode.

  • Clip listing: This container displays all existing .nirec files for the current project. The files are stored in the project folder and are imported on NI mate start up or project load. The clips are listed under the family of sensor devices that are capable of playbacking the clip in question. For example, Asus XTion can playback data recorded by the PrimeSense Carmine as they both are OpenNI 2 sensors.

  • Export BVH: Clicking this button will open BVH export settings.

    • Smoothing: Specifies the amount of smoothing applied to the data. Higher smoothing levels result in less noise in the skeleton data and smoother movement.

    • Framerate: Specifies the framerate at which the data is exported. It’s possible to use a much higher framerate that the sensor natively uses. NI mate uses an algorithm to extrapolate the skeleton data and aligns it so that all frames are evenly spaced with equal delta times between frames.

    • OK: Exports all currently selected clips as BioVision (.bvh) files into your current project folder. The .bvh format can be used in most animation software.

    • Cancel: Closes the BVH export menu.

  • Project folder: Opens your current project folder.

Sensor pages

Once a sensor is added a new series of pages will be created for it. Clicking the sensor’s name in the page tree on the left side of the UI will open the sensor’s main page. This page controls all the most important settings related to the sensor’s operations

.

Top controls

  • Record arm: The red circle toggles the sensor’s record state. This can not be done if the sensor is in playback mode.

  • Source: Selects the sensor’s current input data. Using live will use the sensor normally while selecting a recording will turn the sensor into playback mode.

  • Start / stop sensor: Tells the sensor process to start or stop. In case of errors during start-up, please refer to the log window.

  • Preset: NI mate comes with some pre-defined settings that are customized to automatically work well in certain situations. Choosing a preset via this menu will enable those settings.

Live feed

Large part of the page is covered by the live feed output. There are two live feeds which can be displayed here. The live feed dropdown menus are used to select which feed is displayed. The feed modes vary between sensors and not all sensors support all modes.

  • Depth & Extras: Displays the depth feed with all users marked on it.

  • Depth: Displays the live feed without users drawn on it.

  • RGB: Displays the color image feed.

  • Ghost: If a user is detected, NI mate forms a mask, or a “ghost” out of the user. This is displayed as a white blob in the live feed with the background removed.

  • Encoded depth: Displays the depth feed encoded into the red and green color channels. The depth value is stored with the function:
      depth = red << 5 | green   or   depth = red * 32 + green
    where red and green are the first and second rgb color values of the pixel [0-255]. The blue color contains the id of a detected user starting from 1 (0 means there are no users at that specific pixel).

  • Color ID: Each detected user is given a different color. This mode can be useful if you do some kind of user segmentation or detection in an external program.

  • 3D: Turns the feed display into a simple 3D scene where the user’s skeleton data is displayed.

  • Left camera: Displays the live feed from the left camera.

  • Right camera: Displays the live feed from the right camera.

  • Both cameras Displays the live feed from the left and right cameras together.

Sensor parameters

These values are used to change how the sensor performs its tracking. Many of the values require the sensor to be restarted if they are modified.

  • Alignment: This value specifies a relation between the depth and color live feed pixels.

    • Off: The sensor’s depth and color stream are sent out individually without any relation to each other.
    • Depth to color: Allows for matching the sensor’s depth feed so that its
      pixels align with the color feed. This produces a depth image that’s sized the same as the color image. In addition, the ghost feed is upscaled. This is available only on Windows.
    • Fast depth to color: Same as above, but the depth feed is upscaled to a quarter of the color image’s resolution. This is available only on Windows.
    • Color to depth: Maps the color feed’s pixels so that they are aligned with the depth feed. You will see some blackness in the image due to noise in the depth feed. Sensors that don’t have hardware sync between the feeds will also display black “trails” near moving objects. This is available on Windows and Mac.
  • Noise filter: Keeps a short history of the depth data and tries to use old data for
    the pixels where no data is found in the current frame. This makes the
    depth feed much more stable looking, but can create artifacts in the
    feed for large movements.

  • Custom OSC path: If specified, any outgoing OSC messages will be prepended by this OSC address. ie. If multiple Kinect sensors are connected and they are all outputting skeleton tracker data, it can get confusing in the receiving program (such as Blender) to figure out which skeleton joint comes from which sensor. Choosing a OSC path for each sensor allows easily separating the data.

  • High speed: Enables the sensor’s high speed mode. For some sensors this may result in lower resolutions for the live feeds.

  • High resolution color: Changes the sensor’s color feed to be outputted in higher resolution. May result in lower framerates.

  • USB 3 mode: If enabled, the sensor is started with the assumption that it’s connected to a USB 3 port. Most Kinect sensors don’t work well in USB 3 while some have firmware upgrades available for USB 3 support.

Ghost rendering

These settings are used to customize how the ghost is drawn on the feeds.

  • Smoothing: Specifies the amount of smoothing applied on the ghost. Value of 0 will result in a direct copy of the user’s shape from the depth feed with sharp, jagged edges. Maximum value is produce a highly circular shape for all edges.

  • Sharpness: When smoothing is non-zero, this value specifies how much blurring is done on the ghost image. Low values result in a large blur kernel size while high values produce sharper edges.

  • Shift: Makes the ghost thicker or thinner.

  • User inclusion: This box is used for specifying which users are included in the ghost feed.

  • Use as output alpha: If enabled, the ghost image is used as an alpha mask on the live feeds.

  • Color: Chooses the color used for drawing the ghost

  • Background: Chooses the color used for the background in the ghost feed.

Display ghost

Enabling this component will display a ghost on the foreground of your desktop environment.

  • Opacity: Specifies the opacity of the ghost. Lower values will display more of the underlying windows behind the ghost.

  • Fade on mouseover: Hides the ghost if the mouse cursor is positioned over the ghost.

  • Draw extras: If enabled, triggers and other extra graphics are drawn in the display ghost image.

  • Set location: Opens a box for selecting where the display ghost is placed

  • Reset location: Moves the display ghost back to its original location at the bottom middle of the screen.

Detect QR codes

If enabled, the sensor’s color live feed can be used for parsing QR codes.

  • OSC Path: The address of the OSC message that is sent when a QR code is detected.

  • IP address and port override: The IP address and port boxes allow changing where QR codes are sent. If left locked the setup page address is used.

Sensor components

The sensor’s page will have a number of subtrees, referred to as components. These are documented separately.

Log

The log window displays any messages related to NI mate’s operations as well as errors and status updates from sensor devices.

  • Filter: These options can be used to remove or include specific types of messages.

    • Program: Filters any messages that are related to NI mate’s normal operations.
    • User: Filters messages related to user detection and tracking.
    • OSC: Filters outgoing OSC message. If disabled and you’re using skeleton tracking this can result in extremely large quantities of messages and may affect performance.
    • MIDI: Filters outgoing MIDI messages.
  • Clear log: Empties the current log. This can be useful if the log contains impractical quantities of useless logging.

Playback controls

This container appears if some sensor is currently in playback or record mode.

  • Record: This button will start advancing the timeline and begins recording data for all the sensors that are currently armed for recording. The sensors can be armed from either the Motion Recorder page or the sensor’s main page.

  • Play/stop: This button will start playbacking data for those sensors that have their input in “playback” mode.

  • Timeline: This slider allows controlling the current time when playbacking. The timeline is always sized so that the sensor with the longest playback clip defines the length of the timeline.

  • Loop: Once the timeline reaches the end, if this button is enabled the timeline will continue from the beginning.

  • Framerate: Chooses the framerate which sensors will use for their playbacked data. Choosing a high value may result in memory problems as the playback clips are re-computed for the given framerate. It’s possible to record a sensor that has the framerate of 30 (ie. the Kinect for XBox 360) and use a much higher framerate (ie. 90) when playbacking. NI mate uses an algorithm that extrapolates the recorded data so that the playbacked frames are aligned smoothly and evenly for the current framerate.


Documentation: Table of contents
Delicode NI mate Tutorials: Blender Motion Capture Basics
Why don't i see any skeleton tracker page?
Does NI Mate really works?
Depthmap to Spout?