Mike Taulty's Blog
Bits and Bytes from Microsoft UK
Kinect for Windows V2 SDK: Hello (Skeletal) World for the Console Developer ;-)

Blogs

Mike Taulty's Blog

Elsewhere

Archives

Following on from the previous posts, I’ve been taking my first steps towards trying out my third source of data from the Kinect V2 sensor – the BodyFrameSource that delivers skeletal tracking data for up to 6 bodies.

As in my last 2 posts, this is all with reference to the video series on Channel 9;

Programming-Kinect-for-Windows-v2

and I’m not at all claiming to be doing anything ‘unique’ or ‘clever’ or anything like that here – I’m just learning and experimenting and a lot of what I’m doing is duplicating what’s already present in the Kinect for Windows V2 SDK samples which you can see in the sample browser but I find that by tinkering with my own code rather than someone else’s I learn a lot more about the whole experience of developing for something like this.

I wanted to try and take the lowest-tech possible approach to getting started with this skeletal data and so I thought for a while about what the “lowest tech” kind of interface was that I could put on the data and decided that a console application was the way to go Winking smile

Now, you might think “Hey, but what about some fancy 3D interface?”. Sure, that’s possible and there are examples out there and many samples in the SDK and I’ll work on building some of my own but I thought that a console application was a fun idea here because (to me) it illustrates the low barrier to entry that programming with this kind of data has.

That’s been something of a surprise to me – I’d assumed that some of these things were harder to get started with than they seem to be on the Kinect SDK and that’s a nice thing to be slowly finding out Smile

For instance, if I want to know whether there are 1-6 people being tracked by the sensor, I may not need a fancy 3D application. I might just need a piece of code that delivers a “yes/no” result and some environment in which to host that code.

So…here’s my skeletal “Selfie” provided by my console app. Yup, that’s me below;

image

I made this as lo-fi as possible using just single letters to indicate particular joints from the sensor and filtering down the set of joints that I displayed to try and avoid over-cluttering the screen but it works surprisingly well. Here’s a clip of it as video rather than as a screenshot of me showing a few of my best disco moves Winking smile

 

(the music is “Suburban Withdrawl” By Lee Rosevere taken from www.freemusicarchive.org).

In terms of putting that together, it’s a fairly simple and quickly hacked together console app. The main loop looks like this;

using Microsoft.Kinect;
using System;

namespace ConsoleApplication8
{
  class Program
  {
    static void Main(string[] args)
    {
      Console.SetWindowSize(Constants.ConsoleWidth, Constants.ConsoleHeight);
      Console.Title = "Kinect Skeleton Console";

      KinectControl c = new KinectControl(() => new ConsoleBodyDrawer());

      c.GetSensor();
      c.OpenReader();

      while (true)
      {
        var k = Console.ReadKey(true).Key;

        if (k == ConsoleKey.X)
        {
          c.CloseReader();
          c.ReleaseSensor();
          break;
        }
      }
    }
  }
}

and, as in my previous post, I made a little KinectControl class which deals with opening/closing/reading from the sensor. Unlike in the previous post though, the data coming back to my code in this case it not just a 2-dimensional array of values. It’s structured, strongly-typed data in that the sensor returns up to 6 instances of a Body object from its BodyFrameSource  and each Body that gets delivered into your code comes with a whole bunch of information about the tracked body including 20 Joint objects.

I wanted to try and abstract the details of how this Body object was drawn so I wrote a little interface;

using Microsoft.Kinect;
using System;

namespace ConsoleApplication8
{
  interface IBodyDrawer
  {
    ConsoleColor Color { get; set; }
    void DrawFrame(Body body, CoordinateMapper mapper, Rect depthFrameSize);
  }
}

It’s not quite as clean as I would have liked it in that I ended up having to pass a few more details down into the DrawFrame method than I might do if I spent longer on reworking it but then my KinectControl class does the main work here in that it;

  • Creates 6 instances of these IBodyDrawer interfaces.
  • Syncs up to the FrameArrived event from the FrameBodySource.
  • Pushes any Body instances that the Kinect sensor tells us are tracked through into the IBodyDrawer.DrawFrame method above.

The implementation for that looks like this;

using Microsoft.Kinect;
using System;

namespace ConsoleApplication8
{
  class Rect
  {
    public int Width;
    public int Height;
  }
  class KinectControl
  {
    public KinectControl(Func<IBodyDrawer> bodyDrawerFactory)
    {
      this.bodyDrawers = new IBodyDrawer[colours.Length];

      for (int i = 0; i < colours.Length; i++)
      {
        this.bodyDrawers[i] = bodyDrawerFactory();
        this.bodyDrawers[i].Color = colours[i];
      }
    }
    public Rect DepthFrameSize
    {
      get;
      private set;
    }
    public void GetSensor()
    {
      this.sensor = KinectSensor.GetDefault();
      this.sensor.Open();
      this.DepthFrameSize = new Rect()
      {
        Width = this.sensor.DepthFrameSource.FrameDescription.Width,
        Height = this.sensor.DepthFrameSource.FrameDescription.Height
      };      
    }
    public void OpenReader()
    {
      this.reader = this.sensor.BodyFrameSource.OpenReader();
      this.reader.FrameArrived += OnFrameArrived;
    }
    public void CloseReader()
    {
      this.reader.FrameArrived -= OnFrameArrived;
      this.reader.Dispose();
      this.reader = null;
    }
    void OnFrameArrived(object sender, BodyFrameArrivedEventArgs e)
    {
      using (var frame = e.FrameReference.AcquireFrame())
      {
        if ((frame != null) && (frame.BodyCount > 0))
        {
          if ((this.bodies == null) || (this.bodies.Length != frame.BodyCount))
          {
            this.bodies = new Body[frame.BodyCount];
          }
          frame.GetAndRefreshBodyData(this.bodies);

          Console.Clear();

          for (int i = 0; i < colours.Length; i++)
          {
            if (this.bodies[i].IsTracked)
            {
              this.bodyDrawers[i].DrawFrame(
                 this.bodies[i],
                 this.sensor.CoordinateMapper,
                 this.DepthFrameSize);
            }
          }
        }
      }
    }
    public void ReleaseSensor()
    {
      this.sensor.Close();
      this.sensor = null;
    }
    Body[] bodies;
    KinectSensor sensor;
    BodyFrameReader reader;
    IBodyDrawer[] bodyDrawers;

    static ConsoleColor[] colours = 
    {
        ConsoleColor.Red,
        ConsoleColor.White,
        ConsoleColor.Green,
        ConsoleColor.Yellow,
        ConsoleColor.Cyan,
        ConsoleColor.Gray
    };
  }
}

and the concrete implementation of the IBodyDrawer interface that I knocked up to draw to the console looks like;

using ConsoleExtensions;
using Microsoft.Kinect;
using System;
using System.Collections.Generic;
using System.Text;

namespace ConsoleApplication8
{
  class ConsoleBodyDrawer : IBodyDrawer
  {
    public ConsoleBodyDrawer()
    {
      this.Color = ConsoleColor.Green;
    }
    public ConsoleColor Color { get; set; }

    public void DrawFrame(Body body, CoordinateMapper mapper,
      Rect depthFrameSize)
    {
      this.Resize();

      foreach (var jointType in interestedJointTypes)
      {
        var joint = body.Joints[jointType];

        if (joint.TrackingState != TrackingState.NotTracked)
        {
          var cameraPosition = joint.Position;

          DepthSpacePoint depthPosition = mapper.MapCameraPointToDepthSpace(cameraPosition);

          if (!float.IsNegativeInfinity(depthPosition.X))
          {
            var consolePosition = MapDepthPointToConsoleSpace(depthPosition, depthFrameSize);

            ConsoleEx.DrawAt(
              consolePosition.Item1,
              consolePosition.Item2,
              jointType.ToString().Substring(0,1),
              joint.TrackingState == TrackingState.Inferred ? ConsoleColor.Gray : this.Color);
          }
        }
      }
    }
    void Resize()
    {
      if (!resized)
      {
        Console.SetWindowSize(Constants.ConsoleWidth, Constants.ConsoleHeight);
        resized = true;
      }
    }
    static Tuple<int, int> MapDepthPointToConsoleSpace(DepthSpacePoint depthPoint,
      Rect depthFrameSize)
    {
      var left = (int)((depthPoint.X / (double)depthFrameSize.Width) * (Constants.ConsoleWidth - 1));
      var top = (int)((depthPoint.Y / (double)depthFrameSize.Height) * (Constants.ConsoleHeight - 1));

      left = Math.Min(left, Constants.ConsoleWidth - 1);
      top = Math.Min(top, Constants.ConsoleHeight - 1);
      left = Math.Max(0, left);
      top = Math.Max(0, top);
      return (Tuple.Create(left, top));
    }
    static JointType[] interestedJointTypes = 
    {
      JointType.Head,
      JointType.Neck,
      JointType.ShoulderLeft,
      JointType.ShoulderRight,
      JointType.HandLeft,
      JointType.HandRight,
      JointType.ElbowLeft,
      JointType.ElbowRight,
      JointType.HipLeft,
      JointType.HipRight,
      JointType.KneeLeft,
      JointType.KneeRight,
      JointType.AnkleLeft,
      JointType.AnkleRight,
      JointType.FootLeft,
      JointType.FootRight
    };
    static bool resized = false;
  }
}

That’s it - a staggeringly small amount of pretty high level code required to get this sort of data out of the sensor and into any kind of application that wants to do something with it.

I’ve posted the source here for download – it was put together pretty quickly so (as always) keep that in mind if you’re using it and there’s a bit of code in there conditionally compiled out (#if ZERO) which attempts to display a bit more info about the first body that the sensor reports it is tracking.


Posted Fri, Aug 8 2014 1:03 PM by mtaulty
Filed under: , , ,

Comments

Mike Taulty's Blog wrote Kinect for Windows V2 SDK: Hello (Skeletal) World for the WPF Developer
on Fri, Aug 8 2014 5:41 PM

Following on from my previous posts; Kinect for Windows V2 SDK- Jumping In… Kinect for Windows V2 SDK

Mike Taulty's Blog wrote Kinect for Windows V2 SDK: Hello (Skeletal) World for the 3D WPF Developer
on Tue, Aug 12 2014 10:11 AM

Following on from my previous posts; Kinect for Windows V2 SDK- Jumping In… Kinect for Windows V2 SDK

Mike Taulty's Blog wrote Kinect for Windows V2 SDK: Updated WPF Sample with Common Code across 2D/3D
on Wed, Aug 13 2014 2:03 PM

Following on from my previous posts; Kinect for Windows V2 SDK- Jumping In… Kinect for Windows V2 SDK

Mike Taulty's Blog wrote Kinect for Windows V2 SDK: Hello (Skeletal) World for the .NET Windows 8.1 App Developer
on Thu, Aug 14 2014 1:04 PM

Following on from my previous posts; Kinect for Windows V2 SDK- Jumping In… Kinect for Windows V2 SDK

Mike Taulty's Blog wrote Kinect for WindowsV2 SDK: Hello (Skeletal) World for the JavaScript Windows 8.1 App Developer
on Tue, Aug 19 2014 2:41 PM

Following on from my previous posts; Kinect for Windows V2 SDK- Jumping In… Kinect for Windows V2 SDK

Mike Taulty's Blog wrote Kinect for Windows V2 SDK: Hello (Skeletal) World for the 3D JavaScript Windows 8.1 App Developer
on Tue, Aug 19 2014 2:51 PM

Following on from my previous posts; Kinect for Windows V2 SDK- Jumping In… Kinect for Windows V2 SDK

Mike Taulty's Blog wrote Kinect for Windows SDK V2: Mixing with some Reactive Extensions
on Fri, Aug 22 2014 12:09 PM

I’ve been writing these posts about my beginner’s experiments with the Kinect for Windows V2 SDK; Kinect

Mike Taulty's Blog wrote Kinect for Windows V2 SDK: Hello (BodyIndex, Infra Red) World for the .NET WPF Developer
on Tue, Aug 26 2014 6:09 PM

I’ve been writing these posts about my beginner’s experiments with the Kinect for Windows V2 SDK; Kinect