Kinect for Windows V2 SDK: Hello (Skeletal) World for the 3D JavaScript Windows 8.1 App Developer

Posted on by

Following on from my previous posts;

and, again, highlighting the official videos and samples for the Kinect for Windows V2 SDK bits;

Programming-Kinect-for-Windows-v2

I thought I’d go out on a bit of a limb and combine my lack of skill in JavaScript with my lack of skill in 3D and attempt to move my Windows 8.1 Store app such that it was able to draw in 3D using JavaScript.

In order to do that, I wanted a high level library to help out on the 3D aspects (much like I did when working in WPF) and so I went and looked at http://threejs.org/ to do that for me. It took a little bit of reading but not more than about 10 minutes to get a basic scene up and running and then to figure out how I might be able to use it to draw what I wanted in terms of a connected skeleton.

There were more parallels between 2D/3D JavaScript and 2D/3D .NET in that I ended up trying to come up with a common class that served as a base class for a “2D body drawer” and a “3D body drawer” although I found it slightly more of a challenge than I did in the WPF world.

Here’s a little video of the code running showing both 2D and 3D drawing;

In terms of having both 2D and 3D drawing in the same app, I ended up having 2 canvas instances that I switch on and off as (AFAIK) you can either draw 2D to a canvas or 3D to a canvas but you can’t switch between them (based on: http://msdn.microsoft.com/en-us/library/ie/ff975238(v=vs.85).aspx).

So, my UI became as below which is pretty much the same as the previous post except that I have a 2nd Canvas and I have a basic toggle button on the AppBar to switch between 2D/3D mode which shows/hides the canvases. The code I have though is not capable of doing this halfway through running – it requires stopping the frame reader and releasing the sensor.

<!DOCTYPE html>
<html>
<head>
    <meta charset="utf-8" />
    <title>App200</title>

    <script src="//Microsoft.WinJS.2.0/js/base.js"></script>

    <link href="/css/default.css" rel="stylesheet" />
    <script src="js/three.min.js"></script>
    <script src="js/Iterable.js"></script>
    <script src="js/JointConnection.js"></script>
    <script src="js/BodyDrawerBase.js"></script>
    <script src="js/CanvasBodyDrawer.js"></script>
    <script src="js/3jsBodyDrawer.js"></script>
    <script src="js/KinectControl.js"></script>
    <script src="js/UIHandler.js"></script>
    <script src="js/default.js"></script>
    <script src="//Microsoft.WinJS.2.0/js/ui.js" type="text/javascript"></script>
    <link href="//Microsoft.WinJS.2.0/css/ui-dark.css" rel="stylesheet" type="text/css">
</head>
<body>
    <!-- NB: setting this to 1920x1080 but CSS then scales it to the available space -->
    <!-- took some direction from http://stackoverflow.com/questions/2588181/canvas-is-stretched-when-using-css-but-normal-with-width-height-properties -->
    <canvas id="canvas3d" width="1920" height="1080"></canvas>
    <canvas id="canvas2d" width="1920" height="1080"></canvas>
    <div id="appBar" data-win-control="WinJS.UI.AppBar" data-win-options="{ sticky:true }">
        <button id="chkThreeD" data-win-control="WinJS.UI.AppBarCommand" data-win-options="{label:'3D', selected:true, type:'toggle'}"></button>
        <button data-win-control="WinJS.UI.AppBarCommand"
                onclick="Sample.UIHandler.onGetSensor()"
                data-win-options="{icon:'camera', label:'get sensor', section:'global', type:'button'}"></button>
        <button data-win-control="WinJS.UI.AppBarCommand"
                onclick="Sample.UIHandler.onOpenReader()"
                data-win-options="{icon:'play', label:'open reader', section:'global', type:'button'}"></button>
        <button data-win-control="WinJS.UI.AppBarCommand"
                onclick="Sample.UIHandler.onCloseReader()"
                data-win-options="{icon:'stop', label:'close reader', section:'global', type:'button'}"></button>
        <button data-win-control="WinJS.UI.AppBarCommand"
                onclick="Sample.UIHandler.onReleaseSensor()"
                data-win-options="{icon:'closepane', label:'release sensor', section:'global', type:'button'}"></button>
    </div>
</body>
</html>

and you can no doubt spot the include for three.min.js to help out on the 3D drawing. I then have a bit of code behind this UI which is much like it was in the previous post except that it does some basic work to toggle the visibility of the canvases depending on the toggle switch;

(function ()
{
  "use strict";

  var UIHandler = WinJS.Class.define(
    function ()
    {
    },
    {
      onGetSensor: function (canvas)
      {
        var chkThreeD = document.getElementById('chkThreeD').winControl;
        var threeD = chkThreeD.selected;
        var canvas2d = document.getElementById('canvas2d');
        var canvas3d = document.getElementById('canvas3d');
        var canvas = threeD ? canvas3d : canvas2d;
        var hideCanvas = threeD ? canvas2d : canvas3d;

        canvas.style.visibility = 'visible';
        hideCanvas.style.visibility = 'hidden';

        this._controller = new Sample.KinectControl(
          function ()
          {
            return (
              threeD ? new Sample.ThreeJsBodyDrawer(canvas) : new Sample.CanvasBodyDrawer(canvas));
          }
        );

        this._controller.getSensor();
      },
      onOpenReader: function ()
      {
        this._controller.openReader();
      },
      onCloseReader: function ()
      {
        this._controller.closeReader();
      },
      onReleaseSensor: function ()
      {
        this._controller.releaseSensor();
      },
      _controller: null
    }
  );

  WinJS.Namespace.define(
    'Sample',
    {
      UIHandler: new UIHandler()
    });

})();

and it instantiates my KinectControl ‘class’,  passing it a factory function that either returns a CanvasBodyDrawer or a ThreeJsBodyDrawer depending on whether we’re in 2d/3d mode.

At this point I started to try and define a bit of commonality between drawing to a Canvas in 2D and using three.js to draw to a Canvas in 3D and I bumped up against a few hurdles;

  1. The 2D drawing model largely is simpler than the 3D one in that a context is used to draw circles/lines at various co-ordinates on screen.
  2. The 3D drawing model is more about building up a scene of elements and those elements can later be retrieved and manipulated.

The other thing that I hit was more performance related in that;

  1. I found that in the 2D model I could get away with clearing the entire Canvas and then re-drawing all circles and the connecting lines between them for each frame of data off the sensor.
  2. I found that the 3D model was much less forgiving and I needed to be more reasonable and draw the 3D spheres once and then move them around. I also got much, much better performance by using the capability that three.js has to draw lots of line segments in one batch rather than treating my connections between skeletal points as separate lines.

Because of this, I changed my KinectControl class somewhat from the previous post such that the OnFrameArrived function essentially iterates around its set of 6 body drawing instances and executes a DrawFrame/ClearFrame method call depending on whether that particular body is being tracked or not by the Kinect sensor. This is different from the previous post where the whole Canvas was always cleared before the tracked bodies were drawn and it leans towards the 3D model where each object responsible for drawing a single body has a ‘memory’ of the elements that it has drawn and can re-draw them by moving them or can clear them entirely from the scene without having an impact on other drawn bodies;

(function ()
{
  "use strict";

  var nsKinect = WindowsPreview.Kinect;

  var constants = {
    bodyCount : 6
  };

  var kinectControl = WinJS.Class.define(
    function (bodyDrawerFactory)
    {
      this._bodyDrawerFactory = bodyDrawerFactory;
    },
    {
      getSensor : function()
      {
        var bodyCount = 0;

        this._sensor = nsKinect.KinectSensor.getDefault();
        this._sensor.open();

        this._bodies = new Array(constants.bodyCount);
        this._bodyDrawers = new Array(constants.bodyCount);

        for (bodyCount = 0; bodyCount < constants.bodyCount; bodyCount++)
        {
          this._bodyDrawers[bodyCount] = this._bodyDrawerFactory();
          this._bodyDrawers[bodyCount].init(bodyCount, this._sensor);
        }
      },
      openReader : function()
      {
        this._boundHandler = this._onFrameArrived.bind(this);
        this._reader = this._sensor.bodyFrameSource.openReader();
        this._reader.addEventListener('framearrived', this._boundHandler);
      },
      closeReader : function()
      {
        this._reader.removeEventListener('framearrived', this._boundHandler);
        this._boundHandler = null;
        this._reader.close();
        this._reader = null;
      },
      releaseSensor : function()
      {
        this._bodyDrawers = null;
        this._bodies = null;
        this._sensor.close();
        this._sensor = null;
      },
      _onFrameArrived : function(e)
      {
        var frame = e.frameReference.acquireFrame();
        var i = 0;

        if (frame)
        {
          frame.getAndRefreshBodyData(this._bodies);

          for (i = 0; i < constants.bodyCount; i++)
          {
            if (this._bodies[i].isTracked)
            {
              this._bodyDrawers[i].drawFrame(this._bodies[i]);
            }
            else
            {
              this._bodyDrawers[i].clearFrame();
            }
          }
          frame.close();
        }
      },
      _boundHandler:null,
      _bodyDrawerFactory : null,
      _sensor: null,
      _reader: null,
      _bodyDrawers: null,
      _bodies : null
    }
  );

  WinJS.Namespace.define('Sample',
    {
      KinectControl : kinectControl
    }
  );

})();

Because of the 2D/3D drawing differences, I found that the ‘base class’ abstraction I built to try and represent the commonality of both of these approaches is a bit clunky but I ended up with this ‘class’ which wasn’t part of the previous post;

(function ()
{
  "use strict";

  var nsKinect = WindowsPreview.Kinect;

  var bodyDrawerBase = WinJS.Class.define(
    function ()
    {
    },
    {
      init: function (index, sensor)
      {
        this._index = index;
        this._sensor = sensor;
      },
      drawFrame: function (body)
      {
        // could almost certainly get this all done in one pass.
        var jointPositions = this._drawJoints(body);
        this._drawLines(jointPositions);
        this._drawFrameDone();
      },
      _mapPoint: function (point)
      {
        return (point);
      },
      clearFrame : function()
      {
        throw new Error('Abstract base class method call');
      },
      _drawJoint : function(mappedPoint, isLeaf, color)
      {
        throw new Error('Abstract base class method call');
      },
      _drawConnectionBetweenJoints:function(jointPosition1, jointPosition2, lineColor)
      {
        throw new Error('Abstract base class method call');
      },
      _drawJoints: function(body)
      {
        var that = this;
        var jointPositions = {};

        Iterable.forEach(body.joints,
          function (keyValuePair)
          {
            var jointType = keyValuePair.key;
            var joint = keyValuePair.value;
            var isTracked = joint.trackingState === nsKinect.TrackingState.tracked;    
            var mappedPoint = that._mapPoint(joint.position);

            if (that._isJointForDrawing(joint, mappedPoint))
            {
              that._drawJoint(
                jointType,
                mappedPoint,
                that._isLeaf(jointType),
                isTracked ? bodyDrawerBase._colors[that._index] : bodyDrawerBase._inferredColor);

              jointPositions[jointType] = mappedPoint;
            }
            else
            {
              that._ensureJointNotDrawn(jointType);
            }
          }
        );
        return (jointPositions);
      },
      _ensureJointNotDrawn : function(jointType)
      {

      },
      _drawFrameDone: function ()
      {
      },
      _drawLines: function(jointPositions)
      {
        var that = this;

        bodyDrawerBase._jointConnections.forEach(
          function (jointConnection)
          {
            jointConnection.forEachPair(
              function (j1, j2)
              {
                // do we have this pair recorded in our positions? 
                // i.e. have we drawn them?
                if (jointPositions[j1] && jointPositions[j2])
                {
                  that._drawConnectionBetweenJoints(
                    jointPositions[j1], jointPositions[j2], bodyDrawerBase._lineStyle);
                }
              }
            );
          }
        );
      },
      _isLeaf: function(jointType)
      {
        var leafs = [nsKinect.JointType.head, nsKinect.JointType.footLeft, nsKinect.JointType.footRight];
        return (leafs.indexOf(jointType) !== -1);
      },
      _isJointForDrawing: function(joint, point)
      {
        return (
          (joint.trackingState !== nsKinect.TrackingState.notTracked) &&
          (point.x !== Number.NEGATIVE_INFINITY) &&
          (point.y !== Number.POSITIVE_INFINITY));
      },
      _index : -1,
      _sensor : null
    },
    {
      _colors: ['red', 'green', 'blue', 'yellow', 'purple', 'orange'],
      _lineColor: 'black',
      _inferredColor: 'grey',
      _lineStyle : 'black',
      _jointConnections:
        [
          Sample.JointConnection.createFromStartingJoint(nsKinect.JointType.spineBase, 2),
          Sample.JointConnection.createFromStartingJoint(nsKinect.JointType.shoulderLeft, 4),
          Sample.JointConnection.createFromStartingJoint(nsKinect.JointType.shoulderRight, 4),
          Sample.JointConnection.createFromStartingJoint(nsKinect.JointType.hipLeft, 4),
          Sample.JointConnection.createFromStartingJoint(nsKinect.JointType.hipRight, 4),
          Sample.JointConnection.createFromStartingJoint(nsKinect.JointType.neck, 2),
          Sample.JointConnection.createFromJointList(nsKinect.JointType.spineMid, nsKinect.JointType.spineShoulder, nsKinect.JointType.neck),
          Sample.JointConnection.createFromJointList(nsKinect.JointType.shoulderLeft, nsKinect.JointType.spineShoulder, nsKinect.JointType.shoulderRight),
          Sample.JointConnection.createFromJointList(nsKinect.JointType.hipLeft, nsKinect.JointType.spineBase, nsKinect.JointType.hipRight),
          Sample.JointConnection.createFromJointList(nsKinect.JointType.handTipLeft, nsKinect.JointType.handLeft),
          Sample.JointConnection.createFromJointList(nsKinect.JointType.handTipRight, nsKinect.JointType.handRight),
          Sample.JointConnection.createFromJointList(nsKinect.JointType.wristLeft, nsKinect.JointType.thumbLeft),
          Sample.JointConnection.createFromJointList(nsKinect.JointType.wristRight, nsKinect.JointType.thumbRight)
        ]
    }
  );

  WinJS.Namespace.define('Sample',
    {
      BodyDrawerBase : bodyDrawerBase
    });

})();

I can then use WinJS.Class.derive to built out a version of my previous post’s CanvasBodyDrawer class which derives from this base class;

(function ()
{
  "use strict";

  var nsKinect = WindowsPreview.Kinect;

  var constants =
  {
    circleLeafRadius: 30,
    circleNonLeafRadius: 10,
    lineWidth: 3,
    bodyCount: 6
  };

  var canvasBodyDrawer = WinJS.Class.derive(Sample.BodyDrawerBase,
    function (canvas)
    {
      this._canvas = canvas;
    },
    {
      init: function (index, sensor)
      {
        Sample.BodyDrawerBase.prototype.init.call(this, index, sensor);

        this._sensorColourFrameDimensions = {};

        this._sensorColourFrameDimensions.width =
          this._sensor.colorFrameSource.frameDescription.width;

        this._sensorColourFrameDimensions.height =
          this._sensor.colorFrameSource.frameDescription.height;
      },
      clearFrame: function ()
      {
        // we don't clear because we can't clear on a per-frame basis. we
        // flag the fact we've been asked to clear.
        canvasBodyDrawer._drawCount++;
      },
      _drawFrameDone: function ()
      {
        // at the end of the frame we also flag that we've drawn.
        canvasBodyDrawer._drawCount++;
      },
      _clearBeforeFirstFrame: function ()
      {
        var context;

        // if we've done 6 draw/clears then it must be time to really clear
        // the lot before the next 6 draw/clears.
        if (canvasBodyDrawer._drawCount >= constants.bodyCount)
        {
          context = this._getContext();
          context.clearRect(0, 0, this._canvas.width, this._canvas.height);
          canvasBodyDrawer._drawCount = 0;
        }
      },
      _getContext: function ()
      {
        return (this._canvas.getContext('2d'));
      },
      _mapPoint: function (point)
      {
        var colourPoint = this._sensor.coordinateMapper.mapCameraPointToColorSpace(
          point);

        colourPoint.x *= this._canvas.width / this._sensorColourFrameDimensions.width;
        colourPoint.y *= this._canvas.height / this._sensorColourFrameDimensions.height;

        return (colourPoint);
      },
      _drawJoint: function (jointType, mappedPoint, isLeaf, color)
      {
        var context = this._getContext();

        this._clearBeforeFirstFrame();

        context.fillStyle = color;

        context.beginPath();

        context.arc(
          mappedPoint.x,
          mappedPoint.y,
          isLeaf ? constants.circleLeafRadius : constants.circleNonLeafRadius,
          2 * Math.PI,
          false);

        context.fill();
        context.stroke();
        context.closePath();

        canvasBodyDrawer._drawn = true;
      },
      _drawConnectionBetweenJoints: function (jointPosition1, jointPosition2, lineColor)
      {
        var context = this._getContext();
        context.strokeStyle = lineColor;
        context.lineWidth = constants.lineWidth;

        context.beginPath();
        context.moveTo(jointPosition1.x, jointPosition1.y);
        context.lineTo(jointPosition2.x, jointPosition2.y);
        context.stroke();
        context.closePath();
      },
      _canvas: null,
      _sensorColourFrameDimensions: null
    },
    {
      _drawCount : 0
    }
  );

  WinJS.Namespace.define('Sample',
    {
      CanvasBodyDrawer: canvasBodyDrawer
    });

})();

and then I wrote a new 3D drawing derivation using three.js to do the 3D work for me. This fits better with the base class than the 2D one which I felt I shoe-horned into a new shape from where it was in the previous blog post;

(function ()
{
  "use strict";

  var nsKinect = WindowsPreview.Kinect;

  var constants =
  {
    cameraFieldOfView: 45,
    nearPlaneDistance : 0.1,
    farPlaneDistance: 1000,
    circleRadius: 0.03,
    leafScale: 3.0
  };

  var threeJsBodyDrawer = WinJS.Class.derive(Sample.BodyDrawerBase,
    function (canvas)
    {
      threeJsBodyDrawer._canvas = canvas;
    },
    {
      init: function (index, sensor)
      {
        Sample.BodyDrawerBase.prototype.init.call(this, index, sensor);

        this._drawnJoints = {};

        threeJsBodyDrawer._initScene();
      },
      clearFrame: function ()
      {
        var jointType;

        this._clearLine();

        for (jointType in this._drawnJoints)
        {
          threeJsBodyDrawer._scene.remove(this._drawnJoints[jointType]);
        }
        this._drawnJoints = {};
        threeJsBodyDrawer._renderLoop();
      },
      _clearLine : function()
      {
        if (this._drawnLine)
        {
          threeJsBodyDrawer._scene.remove(this._drawnLine);
          this._drawnLine = null;
        }
      },
      _mapPoint : function(point)
      {
        point.z = 0 - point.z;
        return(point);
      },
      _drawJoint: function (jointType, mappedPoint, isLeaf, color)
      {
        var sphere = this._drawnJoints[jointType];
        var material, scale;

        this._clearLine();

        if (!sphere)
        {
          sphere = new THREE.Mesh(threeJsBodyDrawer._sphereGeometry, material);
          scale = isLeaf ? constants.leafScale : 1.0;
          sphere.scale.set(scale, scale, scale);
          threeJsBodyDrawer._scene.add(sphere);

          this._drawJoints[jointType] = sphere;
        }
        // ensure it's using the right material - can change between frames if joints
        // go from inferred/tracked.
        material = threeJsBodyDrawer._makeSphereMaterial(color);
        sphere.material = material;
        sphere.position.set(mappedPoint.x, mappedPoint.y, mappedPoint.z);
        this._drawnJoints[jointType] = sphere;
      },
      _ensureJointTypeNotDrawn : function(jointType)
      {
        var sphere = this._drawnJoints[jointType];

        if (sphere)
        {
          threeJsBodyDrawer._scene.remove(sphere);
          delete this._drawJoints[jointType];
        }
      },
      _drawConnectionBetweenJoints: function (jointPosition1, jointPosition2, lineColor)
      {
        // Rather than draw many lines (which made the perf horrible), batch them up
        // here into one line (which makes the perf nice :-)).
        if (!this._pendingLineGeometry)
        {
          this._pendingLineGeometry = new THREE.Geometry();
        }
        this._pendingLineGeometry.vertices.push(
          new THREE.Vector3(jointPosition1.x, jointPosition1.y, jointPosition1.z),
          new THREE.Vector3(jointPosition2.x, jointPosition2.y, jointPosition2.z));
      },
      _drawFrameDone : function()
      {
        // got a line waiting to draw?
        if (this._pendingLineGeometry)
        {
          this._drawnLine = new THREE.Line(this._pendingLineGeometry, threeJsBodyDrawer._lineMaterial,
            THREE.LinePieces);

          threeJsBodyDrawer._scene.add(this._drawnLine);

          this._pendingLineGeometry = null;
        }
        threeJsBodyDrawer._renderLoop();
      },
      _drawnLine:null,
      _pendingLineGeometry : null,
      _drawnJoints: null,
      _canvas: null
    },
    {
      _makeSphereMaterial : function(color)
      {
        if (!threeJsBodyDrawer._sphereMaterials[color])
        {
          threeJsBodyDrawer._sphereMaterials[color] = new THREE.MeshLambertMaterial(
            {
              color: color
            }
          );
        }
        return (threeJsBodyDrawer._sphereMaterials[color]);
      },
      _initScene: function (canvas)
      {
        var light;

        if (!threeJsBodyDrawer._scene)
        {
          threeJsBodyDrawer._scene = new THREE.Scene();

          threeJsBodyDrawer._camera =
            new THREE.PerspectiveCamera(
              constants.cameraFieldOfView,
              threeJsBodyDrawer._canvas.width / threeJsBodyDrawer._canvas.height,
              constants.nearPlaneDistance,
              constants.farPlaneDistance);

          threeJsBodyDrawer._camera.position.z = 1;
          threeJsBodyDrawer._scene.add(threeJsBodyDrawer._camera);

          light = new THREE.PointLight(0xFFFFFF);
          light.position.set(-1, 1, 1);
          threeJsBodyDrawer._scene.add(light);

          threeJsBodyDrawer._renderer = new THREE.WebGLRenderer(
            {
              antialias : true,
              canvas: threeJsBodyDrawer._canvas
            }
          );

          threeJsBodyDrawer._renderer.setClearColor('grey');
          threeJsBodyDrawer._renderer.clear();
        }
      },
      _renderLoop: function ()
      {
        threeJsBodyDrawer._renderer.render(
          threeJsBodyDrawer._scene, threeJsBodyDrawer._camera);
      },
      _lineMaterial: new THREE.LineBasicMaterial({ color: 0x000000 }),
      _sphereGeometry: new THREE.SphereGeometry(constants.circleRadius, 32, 32),
      _sphereMaterials: {},
      _canvas: null,
      _renderer: null,
      _scene: null,
      _camera: null
    }
  );

WinJS.Namespace.define('Sample',
  {
    ThreeJsBodyDrawer: threeJsBodyDrawer
  });

})();

and the rest of the code is identical to what I listed out in the previous post so I won’t repeat that here.

That code is here for download if you wanted to try it out or have a poke around in it.

In terms of getting a skeleton drawn in JavaScript I was pretty impressed by how little code it takes and how relatively high level that code is.

In terms of this series of posts, I’ve spent a bit of time experimenting with skeletal data in a few different development environments and so what I’d like to do next is to look at some of the other Kinect data sources that I haven’t experimented with.

More to come…