Kinect for Windows V2 SDK: Drawing with Win2D

Posted on by

Having encountered Win2D (see posts), I thought I’d attempt to update the 2D drawing code that I wrote in this post for a Windows 8.1 Store app;

Kinect for Windows V2 SDK- Hello (Skeletal) World for the .NET Windows 8.1 App Developer

so that it drew not only via the regular XAML Canvas object but also via Win2D.

I’m happy to admit that this mostly falls into the category of doing something “because it’s there” rather than with any specific intent in mind Smile

Here’s a little screen capture of code drawing first with a XAML Canvas and then next with Win2D.

I updated my local version of Win2D to be at 0.0.7 (the last version I had was 0.0.5), dropped the outputs of building it into my local package repository and then added it into the solution from that earlier post;

image

and then started to look at what sort of abstraction I’d built in that post for drawing skeletal data in a 2D manner (I only draw circles and lines, it’s not rocket science).

The code in that post had migrated across from the code that I wrote in this other WPF focused post;

where I’d built an abstraction that helped me deal with the differences across building for;

  • 2D drawing on a XAML Canvas
  • 3D drawing using the Helix 3D Toolkit

That abstraction worked by having common code for drawing skeletons which was based around representing the drawn elements as UIElements or DependencyObjects such that I could have code which dealt with both 2D and 3D lines/circles.

However, the abstraction was essentially assuming a “retained graphics” mode where it’s possible to draw some graphical element onto a surface and expect that;

  • some “drawing surface” will take responsibility for maintaining the list of drawn elements such that they can easily be repositioned or removed at a later point
  • some “drawing surface” will take responsibility for knowing when it should best redraw itself

In trying to now work across XAML’s Canvas and Win2D’s CanvasControl, that abstraction breaks down in that Win2D is an immediate mode graphics API and doesn’t remember the things that it’s been asked to draw so if you need that sort of functionality then you have to build it yourself and so that’s what I set about doing.

I ended up building a new abstraction for drawing the 6 bodies that flow from the Kinect sensor in each frame of data which I hope is fairly intuitive based on the interface definition below;

namespace App199
{
  using WindowsPreview.Kinect;

  interface IBodyDrawer
  {
    void Init(KinectSensor sensor);
    void PreDrawBodies();
    void PreDrawBody(int index);
    void DrawBody(int index, Body body);
    void ClearBody(int index);
    void PostDrawBody(int index);
    void PostDrawBodies();
  }
}

and I built a little class which is based around that abstraction and which ties it up with frames coming from the BodyFrameSource on the KinectSensor as below;

namespace App199
{
  using System;
  using WindowsPreview.Kinect;

  class KinectControl
  {
    public KinectControl(Func<IBodyDrawer> bodyDrawerFactory)
    {
      this.bodyDrawerFactory = bodyDrawerFactory;
    }
    public void GetSensor()
    {
      this.sensor = KinectSensor.GetDefault();
      this.sensor.Open();

      this.bodyDrawer = this.bodyDrawerFactory();
      this.bodyDrawer.Init(this.sensor);
      this.bodies = new Body[this.sensor.BodyFrameSource.BodyCount];
    }
    public void OpenReader()
    {
      this.reader = this.sensor.BodyFrameSource.OpenReader();
      this.reader.FrameArrived += OnFrameArrived;
    }
    public void CloseReader()
    {
      this.reader.FrameArrived -= OnFrameArrived;
      this.reader.Dispose();
      this.reader = null;
    }
    void OnFrameArrived(object sender, BodyFrameArrivedEventArgs e)
    {
      using (var frame = e.FrameReference.AcquireFrame())
      {
        if (frame != null)
        {
          frame.GetAndRefreshBodyData(this.bodies);

          this.bodyDrawer.PreDrawBodies();

          for (int i = 0; i < this.bodies.Length; i++)
          {
            this.bodyDrawer.PreDrawBody(i);

            if (this.bodies[i].IsTracked)
            {
              this.bodyDrawer.DrawBody(i, this.bodies[i]);
            }
            else
            {
              this.bodyDrawer.ClearBody(i);
            }
            this.bodyDrawer.PostDrawBody(i);
          }
          this.bodyDrawer.PostDrawBodies();
        }
      }
    }
    public void ReleaseSensor()
    {
      this.sensor.Close();
      this.sensor = null;
    }
    Body[] bodies;
    KinectSensor sensor;
    BodyFrameReader reader;
    IBodyDrawer bodyDrawer;
    Func<IBodyDrawer> bodyDrawerFactory;
  }
}

then I wired that to a little UI defined in XAML which has both a regular XAML Canvas and a Win2D CanvasControl with a toggle button to switch between the two different drawing surfaces;

<Page x:Class="App199.MainPage"
      xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
      xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
      xmlns:local="using:App199"
      xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
      xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
      mc:Ignorable="d"
      xmlns:w2d="using:Microsoft.Graphics.Canvas">
  <Page.BottomAppBar>
    <CommandBar IsOpen="True"
                IsSticky="True">
      <CommandBar.PrimaryCommands>
        <AppBarToggleButton Label="Use Win2D"
                            x:Name="toggleWin2D"
                            IsChecked="false" />
        <AppBarButton Icon="Camera"
                      Label="Get Sensor"
                      Click="OnGetSensor" />
        <AppBarButton Icon="Play"
                      Label="Open Reader"
                      Click="OnOpenReader" />
        <AppBarButton Icon="Stop"
                      Label="Close Reader"
                      Click="OnCloseReader" />
        <AppBarButton Icon="ClosePane"
                      Label="Close Sensor"
                      Click="OnReleaseSensor" />
      </CommandBar.PrimaryCommands>
    </CommandBar>
  </Page.BottomAppBar>
  <Grid Background="Silver">
    <Canvas x:Name="canvasBody"></Canvas>
    <w2d:CanvasControl x:Name="canvasBodyW2D"
                       Background="Silver"/>
  </Grid>
</Page>

and a little code-behind to wire this UI to the KinectControl class I’d built;

namespace App199
{
  using Windows.UI.Xaml;
  using Windows.UI.Xaml.Controls;

  public sealed partial class MainPage : Page
  {
    KinectControl controller;

    public MainPage()
    {
      this.InitializeComponent();
    }
    void OnGetSensor(object sender, RoutedEventArgs e)
    {
      IBodyDrawer drawer =
        (bool)this.toggleWin2D.IsChecked ?
          (IBodyDrawer)new Win2DCanvasBodyDrawer(this.canvasBodyW2D) :
          (IBodyDrawer)new XamlCanvasBodyDrawer(this.canvasBody);

      this.controller = new KinectControl(() => drawer);  

      this.controller.GetSensor();
    }
    void OnOpenReader(object sender, RoutedEventArgs e)
    {
      this.controller.OpenReader();
    }
    void OnCloseReader(object sender, RoutedEventArgs e)
    {
      this.controller.CloseReader();
    }
    void OnReleaseSensor(object sender, RoutedEventArgs e)
    {
      this.controller.ReleaseSensor();
    }
  }
}

and that code tries to feed one of two implementations of IBodyDrawer into the KinectControl constructor based on whether the “Win2D” toggle button is checked or not on-screen.

The two implementations of IBodyDrawer are a XamlCanvasBodyDrawer and a Win2DCanvasBodyDrawer but I ended up with more than that in my hierarchy in putting those together.

Drawers and Base Classes

I didn’t want to have to implement the essentials of drawing skeletons more than once and so I wrote a base class implementation of my interface IBodyDrawer which attempts to deal with the details of the skeletal connections and so on but does not know the specifics of how to draw the lines and circles that make up the on-screen representation;

namespace App199
{
  using System.Collections.Generic;
  using Windows.Foundation;
  using Windows.UI;
  using WindowsPreview.Kinect;

  abstract class BodyDrawerBase : IBodyDrawer
  {
    protected abstract void DrawJointAtPoint(int index, Joint joint, Point point, Color color);

    protected abstract void ClearJoint(int index, Joint joint);

    protected abstract void DrawLineBetweenPoints(int index, Point start, Point end,
      Color lineColor);

    public abstract void ClearBody(int index);

    public virtual void Init(KinectSensor sensor)
    {
      this.Sensor = sensor;
    }
    public virtual void PreDrawBodies()
    {
    }
    public virtual void PostDrawBodies()
    {

    }
    public virtual void PreDrawBody(int index)
    {

    }
    public virtual void PostDrawBody(int index)
    {

    }
    public virtual void DrawBody(int index, Body body)
    {
      Color color = bodyColors[index];

      var jointPositions = this.DrawJoints(index, body, color);

      this.DrawLines(index, jointPositions);
    }

    protected virtual Point MapPoint(CameraSpacePoint point)
    {
      return (new Point(point.X, point.Y));
    }
    protected virtual bool IsJointForDrawing(Joint joint, Point p)
    {
      return (
          (joint.TrackingState != TrackingState.NotTracked) &&
          (!double.IsInfinity(p.X)) &&
          (!double.IsInfinity(p.Y)));
    }
    protected virtual Dictionary<JointType, Point> DrawJoints(int index, Body body,
      Color color)
    {
      Dictionary<JointType, Point> jointPositions = new Dictionary<JointType, Point>();

      foreach (var item in body.Joints)
      {
        JointType jointType = item.Key;
        Joint joint = item.Value;

        Point point = this.MapPoint(joint.Position);

        bool draw = IsJointForDrawing(joint, point);

        bool isTracked = joint.TrackingState == TrackingState.Tracked;

        if (draw)
        {
          this.DrawJointAtPoint(index, joint, point,
            isTracked ? color : inferredColor);

          jointPositions[jointType] = point;
        }
        else
        {
          this.ClearJoint(index, joint);
        }
      }
      return (jointPositions);
    }
    void DrawLines(int index, IReadOnlyDictionary<JointType, Point> jointPositions)
    {
      foreach (var jointConnection in jointConnections)
      {
        // that little data structure either contains a list of joints to work through or
        // a start joint and an element count. it's discriminated and shouldn't contain
        // both!
        jointConnection.ForEachPair(
          (j1, j2) =>
          {
            if (jointPositions.ContainsKey(j1) && jointPositions.ContainsKey(j2))
            {
              Point p1 = jointPositions[j1];
              Point p2 = jointPositions[j2];
              this.DrawLineBetweenPoints(index, p1, p2, lineColor);
            }
          }
        );
      }
    }
    protected KinectSensor Sensor
    {
      get;
      private set;
    }
    // This is bad really because it depends on the actual enum values for the JointType
    // type in the SDK not changing.
    // It's easier though than having some massive array of all the connections but that 
    // would be the right thing to do I think.
    static JointConnection[] jointConnections = 
    {
      new JointConnection(JointType.SpineBase, 2),
      new JointConnection(JointType.ShoulderLeft, 4),
      new JointConnection(JointType.ShoulderRight, 4),
      new JointConnection(JointType.HipLeft, 4),
      new JointConnection(JointType.HipRight, 4),
      new JointConnection(JointType.Neck, 2),
      new JointConnection(JointType.SpineMid, JointType.SpineShoulder, JointType.Neck),
      new JointConnection(JointType.ShoulderLeft, JointType.SpineShoulder, JointType.ShoulderRight),
      new JointConnection(JointType.HipLeft, JointType.SpineBase, JointType.HipRight),
      new JointConnection(JointType.HandTipLeft, JointType.HandLeft),
      new JointConnection(JointType.HandTipRight, JointType.HandRight),
      new JointConnection(JointType.WristLeft, JointType.ThumbLeft),
      new JointConnection(JointType.WristRight, JointType.ThumbRight)
    };
    static readonly Color[] bodyColors = 
    {
      Colors.Red,
      Colors.Blue,
      Colors.Green,
      Colors.Yellow,
      Colors.Purple,
      Colors.Orange
    };
    static readonly Color lineColor = Colors.Black;
    static readonly Color inferredColor = Colors.LightGray;
  }
}

and so this class is delegating work down to derived classes via the functions;

  • DrawJointAtPoint
  • DrawLineBetweenPoints
  • ClearJoint
  • ClearBody

Drawing via Win2D

With that in place, it’s relatively easy to implement an IBodyDrawer that uses Win2D to draw. I implemented that as below;

namespace App199
{
  using Microsoft.Graphics.Canvas;
  using Microsoft.Graphics.Canvas.Numerics;
  using System;
  using System.Collections.Generic;
  using System.Linq;
  using Windows.Foundation;
  using Windows.UI;
  using WindowsPreview.Kinect;

  class Win2DCanvasBodyDrawer : BodyDrawerBase
  {
    public Win2DCanvasBodyDrawer(CanvasControl canvas)
    {
      this.canvas = canvas;
    }
    public override void Init(KinectSensor sensor)
    {
      base.Init(sensor);

      this.joints = new List<Tuple<Joint, Point, Color>>[
        this.Sensor.BodyFrameSource.BodyCount];

      for (int i = 0; i < this.joints.Length; i++)
      {
        this.joints[i] = new List<Tuple<Joint, Point, Color>>();
      }
      this.lines = new List<Tuple<Point, Point, Color>>[
        this.Sensor.BodyFrameSource.BodyCount];

      for (int i = 0; i < this.lines.Length; i++)
      {
        this.lines[i] = new List<Tuple<Point, Point, Color>>();
      }

      this.canvas.Draw += OnDraw;

      this.coordMapper = new CoordMapper2D(
        this.Sensor.CoordinateMapper,
        this.Sensor.ColorFrameSource.FrameDescription);
    }
    public override void PostDrawBodies()
    {
      base.PostDrawBodies();
      this.canvas.Invalidate();
    }
    void OnDraw(CanvasControl sender, CanvasDrawEventArgs args)
    {
      args.DrawingSession.Clear(sender.ClearColor);

      Vector2 centre = new Vector2();

      for (int i = 0; i < this.joints.Length; i++)
      {
        foreach (var item in this.joints[i])
        {
          centre.X = (float)item.Item2.X;
          centre.Y = (float)item.Item2.Y;

          args.DrawingSession.FillCircle(
            centre, 
            largeJoints.Contains(item.Item1.JointType) ? LARGE_JOINT_SIZE : SMALL_JOINT_SIZE, 
            item.Item3);
        }
      }
      Vector2 start = new Vector2();
      Vector2 end = new Vector2();

      for (int i = 0; i < this.lines.Length; i++)
      {
        foreach (var item in this.lines[i])
        {
          start.X = (float)item.Item1.X;
          start.Y = (float)item.Item1.Y;
          end.X = (float)item.Item2.X;
          end.Y = (float)item.Item2.Y;

          args.DrawingSession.DrawLine(start, end, item.Item3);
        }
      }
    }
    protected override void DrawJointAtPoint(
      int index, Joint joint, Point point, Color color)
    {
      this.joints[index].Add(Tuple.Create(joint, point, color));
    }
    protected override Point MapPoint(CameraSpacePoint point)
    {
      return (this.coordMapper.MapCameraSpacePoint(point,
        this.canvas.ActualWidth, this.canvas.ActualHeight));
    }
    protected override void ClearJoint(int index, Joint joint)
    {
    }
    public override void PreDrawBody(int index)
    {
      base.PreDrawBody(index);
      this.joints[index].Clear();
      this.lines[index].Clear();
    }
    public override void ClearBody(int index)
    {
      this.joints[index].Clear();
      this.lines[index].Clear();
    }
    protected override void DrawLineBetweenPoints(
      int index, Point start, Point end, Color lineColor)
    {
      this.lines[index].Add(Tuple.Create(start, end, lineColor));
    }
    static JointType[] largeJoints = 
    {
      JointType.Head,
      JointType.FootLeft,
      JointType.FootRight                                   
    };
    static readonly float SMALL_JOINT_SIZE = 5.0f;
    static readonly float LARGE_JOINT_SIZE = 10.0f;
    List<Tuple<Joint, Point, Color>>[] joints;
    List<Tuple<Point, Point, Color>>[] lines;
    CanvasControl canvas;
    CoordMapper2D coordMapper;
  }
}

The main challenge here is that I have up to 6 bodies to draw with each made up of 25 joints and the connections between them and I want to do that with the minimum of redrawing.

The CanvasControl has a Draw event which I handle to do my drawing and if the control gets invalidated for any reason then I need to redraw whatever was on it and so, consequently, I need to keep a list of all the things that I’ve drawn.

As the base class calls the Draw/Clear methods on this class it can’t do any drawing because that only happens in the Draw event which is raised from an invalidation of the control. Those methods simply build up lists of things that need to be drawn with the override of PostDrawBodies taking care to invalidate the CanvasControl once all the data for a particular skeletal frame has arrived.

Data kept on lists for an individual body is cleared out in the PreDrawBody override and should a body go from being tracked in one frame to untracked in another the ClearBody override takes care of that.

Drawing via XAML Canvas

In order to draw via a XAML Canvas, I inserted a second base class which thinks in terms of a retained graphics model where the joints and connections to be drawn are added to some kind of retained drawing surface ( most likely just a Canvas but I think this would also work as the basis for a 3D surface too in WPF ). That class ended up looking like;

namespace App199
{
  using System.Linq;
  using System.Collections.Generic;
  using Windows.Foundation;
  using Windows.UI;
  using Windows.UI.Xaml;
  using Windows.UI.Xaml.Media;
  using WindowsPreview.Kinect;

  abstract class UIElementBodyDrawerBase : BodyDrawerBase
  {
    static UIElementBodyDrawerBase()
    {
      brushes = new Dictionary<Color, Brush>();
    }
    protected abstract DependencyObject CreateXamlUIElementForJoint(
      JointType jointType, Brush brush);

    protected abstract void UpdateXamlUIElementForJoint(
      DependencyObject uiElement, Point point, Brush brush);

    protected abstract void RemoveXamlUIElement(DependencyObject element);

    protected abstract DependencyObject CreateXamlLineBetweenPoints(
      Point point1, Point point2, Brush brush);

    public override void Init(KinectSensor sensor)
    {
      base.Init(sensor);

      this.drawnLineElements = new List<DependencyObject>();

      this.drawnJointElements = new Dictionary<JointType, DependencyObject>[
        this.Sensor.BodyFrameSource.BodyCount];

      for (int i = 0; i < this.drawnJointElements.Length; i++)
      {
        this.drawnJointElements[i] = new Dictionary<JointType, DependencyObject>();
      }
      this.drawnLineElements = new List<DependencyObject>();
    }
    static Brush BrushFromColor(Color color)
    {
      if (!brushes.ContainsKey(color))
      {
        brushes.Add(color, new SolidColorBrush(color));
      }
      return (brushes[color]);
    }
    private void ClearDrawnLines()
    {
      foreach (var item in this.drawnLineElements)
      {
        this.RemoveXamlUIElement(item);
      }
      this.drawnLineElements.Clear();
    }
    public override void PreDrawBodies()
    {
      base.PreDrawBodies();
      this.ClearDrawnLines();
    }
    public override void ClearBody(int index)
    {
      foreach (var item in this.drawnJointElements[index])
      {
        this.RemoveXamlUIElement(item.Value);
      }
      this.drawnJointElements[index].Clear();
    }
    protected override void DrawJointAtPoint(int index, Joint joint, Point point, Color color)
    {
      DependencyObject element = null;

      Brush brush = BrushFromColor(color);

      if (!this.drawnJointElements[index].TryGetValue(joint.JointType, out element))
      {
        element = CreateXamlUIElementForJoint(joint.JointType, brush);
        this.drawnJointElements[index][joint.JointType] = element;
      }
      this.UpdateXamlUIElementForJoint(element, point, brush);
    }
    protected override void ClearJoint(int index, Joint joint)
    {
      DependencyObject element = null;

      if (this.drawnJointElements[index].TryGetValue(joint.JointType, out element))
      {
        this.RemoveXamlUIElement(element);
        this.drawnJointElements[index].Remove(joint.JointType);
      }
    }
    protected override void DrawLineBetweenPoints(
      int index, Point start, Point end, Color lineColor)
    {
      DependencyObject line = 
        CreateXamlLineBetweenPoints(start, end, BrushFromColor(lineColor));
      
      this.drawnLineElements.Add(line);
    }   
    Dictionary<JointType, DependencyObject>[] drawnJointElements;
    List<DependencyObject> drawnLineElements;
    static Dictionary<Color, Brush> brushes;
  }
}

Note that although the class is called UIElementBodyDrawerBase, it actually thinks of “UI Elements” as being anything that’s derived from DependencyObject. This class then delegates the work of drawing down to a derived class by asking for overrides for member functions;

  • CreateXamlUIElementForJoint
  • UpdateXamlUIElementForJoint
  • RemoveXamlUIElement

and this class takes a different approach to drawing the joints than it does to the connections between them. For the joints, they are drawn when they are first encountered and then, subsequently, they are moved to update them to their new positions as the body moves.

For the connections between the joints, they are drawn and cleared each frame. I think that assumption came from when I tried to make this work at a basic level with 3D in WPF and the cost of recreating joints seemed wasteful and so it’s persisted into this code.

With that all figured out, the derivation of this class which actually draws to a XAML Canvas looks like;

namespace App199
{
  using System.Linq;
  using Windows.Foundation;
  using Windows.UI.Xaml;
  using Windows.UI.Xaml.Controls;
  using Windows.UI.Xaml.Media;
  using Windows.UI.Xaml.Shapes;
  using WindowsPreview.Kinect;

  class XamlCanvasBodyDrawer : UIElementBodyDrawerBase
  {
    public XamlCanvasBodyDrawer(Canvas canvas)
    {
      this.canvas = canvas;
    }
    public override void Init(KinectSensor sensor)
    {
      base.Init(sensor);

      this.canvasCoordMapper = new CoordMapper2D(
        this.Sensor.CoordinateMapper,
        this.Sensor.ColorFrameSource.FrameDescription);
    }
    protected override DependencyObject CreateXamlUIElementForJoint(JointType jointType,
      Brush brush)
    {
      JointType[] leafTypes =
        {
          JointType.Head,
          JointType.FootRight,
          JointType.FootLeft
        };
      bool large = leafTypes.Contains(jointType);
      int size = large ? Constants.LargeJointEllipseDiamater : Constants.JointEllipseDiameter;

      Ellipse ellipse = new Ellipse()
      {
        Width = size,
        Height = size,
        VerticalAlignment = VerticalAlignment.Center,
        HorizontalAlignment = HorizontalAlignment.Center,
        Fill = brush
      };
      this.canvas.Children.Add(ellipse);

      return (ellipse);
    }
    protected override Point MapPoint(CameraSpacePoint point)
    {
      return (this.canvasCoordMapper.MapCameraSpacePoint(point,
        this.canvas.ActualWidth, this.canvas.ActualHeight));
    }
    protected override void UpdateXamlUIElementForJoint(
      DependencyObject uiElement, Point point, Brush brush)
    {
      Ellipse ellipse = (Ellipse)uiElement;
      ellipse.Fill = brush;

      Canvas.SetLeft(ellipse, point.X - (ellipse.Width / 2));
      Canvas.SetTop(ellipse, point.Y - (ellipse.Height / 2));
    }
    protected override DependencyObject CreateXamlLineBetweenPoints(
      Point point1, Point point2, Brush brush)
    {
      Line line = new Line()
      {
        X1 = point1.X,
        Y1 = point1.Y,
        X2 = point2.X,
        Y2 = point2.Y,
        Stroke = brush,
        StrokeThickness = Constants.StrokeThickness
      };

      this.canvas.Children.Add(line);

      return (line);
    }
    protected override void RemoveXamlUIElement(DependencyObject element)
    {
      this.canvas.Children.Remove((UIElement)element);
    }
    Canvas canvas;
    CoordMapper2D canvasCoordMapper;

    static class Constants
    {
      public static readonly int JointEllipseDiameter = 10;
      public static readonly int LargeJointEllipseDiamater = 30;
      public static readonly double StrokeThickness = 1.0;
    }
  }
}

and so the derived class becomes pretty simple and only focuses on the specifics of getting the drawing done to a Canvas.

Code

The bits from this post are here for download – feel free to try them out.