Following on from my previous posts;
- Kinect for Windows V2 SDK- Jumping In…
- Kinect for Windows V2 SDK- Hello (Color) World
- Kinect for Windows V2 SDK- Hello (Skeletal) World for the Console Developer 😉
- Kinect for Windows V2 SDK- Hello (Skeletal) World for the WPF Developer
I thought I’d take some of the pieces that I’d just put together for that last post where I was attempting to draw the skeletal data in 2D using a Canvas in WPF and that I’d attempt to move that into more of a 3D world.
Once again, at the risk of repeating myself, I’m not trying to do anything “new” here and the samples in the SDK have much more than this functionality but I’m experimenting for myself and I figure I might as well write it up here as I go along. You can get the official view from the Channel 9 videos;
Moving into 3D for me, is a little bit like building sand-on-sand because I’m at the “conscious incompetence” stage when it comes to the Kinect SDK and I’m pretty much at the same stage when it comes to 3D in general although I have done some bits and pieces with 3D in WPF (and Silverlight) but I’m a long way from being a 3D developer.
With that in mind, I wanted to find something to help out on the 3D aspects of drawing a skeleton so I looked to the Helix WPF toolkit to help me out with the basics of drawing the spheres that I wanted to draw to represent body joints and the tubes that I wanted to draw to connect joints together.
I ended up with something that looks like this – I’ve not yet tested it out with multiple bodies so that’d be interesting to try from a perf/correctness point of view but it seems to work ok for a single body;
and the general approach that I took was to keep pretty much what I’d done in the previous blog posts and then to tweak it to include a drawing component that slotted in an attempt to draw in 3D.
The UI needed to change a little though because previously I was drawing to a Canvas and for drawing to a 3D surface I switched to ‘drawing’ to a Model3D which I wrapped up in a HelixViewport3D as per the XAML below;
<Window x:Class="WpfApplication6.MainWindow"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
Title="MainWindow"
Height="350"
Width="525"
xmlns:helix="http://helixtoolkit.codeplex.com">
<Grid>
<helix:HelixViewport3D HorizontalAlignment="Stretch"
VerticalAlignment="Stretch"
Background="Silver">
<helix:HelixViewport3D.DefaultCamera>
<PerspectiveCamera Position="0,0,5"
LookDirection="0,0,-1"
UpDirection="0,1,0" />
</helix:HelixViewport3D.DefaultCamera>
<helix:SunLight />
<ModelVisual3D x:Name="modelVisual" />
</helix:HelixViewport3D>
<StackPanel VerticalAlignment="Bottom"
Orientation="Horizontal"
HorizontalAlignment="Center">
<StackPanel.Resources>
<Style TargetType="Button">
<Setter Property="Margin"
Value="5" />
</Style>
</StackPanel.Resources>
<Button Content="Get Sensor"
Click="OnGetSensor" />
<Button Content="Open Reader"
Click="OnOpenReader" />
<Button Content="Close Reader"
Click="OnCloseReader" />
<Button Content="Close Sensor"
Click="OnReleaseSensor" />
</StackPanel>
</Grid>
</Window>
The key thing here is the ModelVisual3D which is named so that my code behind can pick it up. That code behind delegates everything down to my KinectControl class as it did in previous examples but in this case I’ve modified the class so that it is constructed with a factory that knows how to make a HelixModelVisual3DBodyDrawer to do the drawing and that class is passed the ModelVisual3D at construction time as below;
namespace WpfApplication6
{
using System.Windows;
public partial class MainWindow : Window
{
KinectControl controller;
public MainWindow()
{
InitializeComponent();
this.Loaded += OnLoaded;
}
void OnLoaded(object sender, RoutedEventArgs e)
{
this.controller = new KinectControl(() => new HelixModelVisual3DBodyDrawer(this.modelVisual));
}
void OnGetSensor(object sender, RoutedEventArgs e)
{
this.controller.GetSensor();
}
void OnOpenReader(object sender, RoutedEventArgs e)
{
this.controller.OpenReader();
}
void OnCloseReader(object sender, RoutedEventArgs e)
{
this.controller.CloseReader();
}
void OnReleaseSensor(object sender, RoutedEventArgs e)
{
this.controller.ReleaseSensor();
}
}
}
The KinectControl class is (I think) mostly unchanged from previous posts, it just takes on the job of getting the sensor up and running, receiving BodyFrames and then managing a set of implementations of an interface IBodyDrawer which I tweaked again slightly but it’s fundamentally the same as it was in that it takes a colour (Brush here) and it then has DrawFrame/ClearFrame methods to draw in response to frames coming off the sensor. Because the sensor can track up to 6 bodies, the KinectControl spins up an array of 6 of these with different colours to represent these bodies;
namespace WpfApplication6
{
using Microsoft.Kinect;
using System.Windows.Media;
interface IBodyDrawer
{
Brush Brush { get; set; }
void Init();
void ClearFrame();
void DrawFrame(Body body);
}
}
So, moving from 2D to 3D didn’t change this much. Of course, what does change is the actual rendering code and the HelixModelVisual3DBodyDrawer class that I wrote is as below. I’m not 100% sure on aspects of this at the time of writing with the crucial one for me being a bit of an uncertain feeling about the co-ordinate system. I feel that I understand the co-ordinates that the Kinect uses as ColorSpacePoint and also the ones that it uses as DepthSpacePoint but I’m a bit more vague in terms of understanding what I get from the Body class as CameraSpacePoint. I’ll happily admit that I somewhat took an approach of plugging stuff together to see if it worked and it seems to but I’d like to get a better level of understanding of the numbers that I’m dealing with here.
namespace WpfApplication6
{
using HelixToolkit.Wpf;
using Microsoft.Kinect;
using System.Collections.Generic;
using System.Linq;
using System.Windows.Media;
using System.Windows.Media.Media3D;
class HelixModelVisual3DBodyDrawer : IBodyDrawer
{
static class Constants
{
public static readonly double LeafScaleSize = 4.0;
public static readonly double RegularScaleSize = 1.0;
public static readonly double SphereRadius = 0.02;
public static readonly double TubeDiameter = 0.02;
}
public HelixModelVisual3DBodyDrawer(ModelVisual3D model3dGroup)
{
this.model3dGroup = model3dGroup;
this.drawnJointElements = new Dictionary<JointType, ModelVisual3D>();
this.drawnLineElements = new List<TubeVisual3D>();
}
public Brush Brush
{
get;
set;
}
public void Init()
{
this.sphereVisual3d = MakeSphereForBrush(this.Brush);
if (inferredSphereVisual3d == null)
{
inferredSphereVisual3d = MakeSphereForBrush(inferredBrush);
}
}
static SphereVisual3D MakeSphereForBrush(Brush brush)
{
var sphere = new SphereVisual3D();
sphere.Radius = Constants.SphereRadius;
sphere.Fill = brush;
return(sphere);
}
public void DrawFrame(Body body)
{
this.RemoveLinesFromPreviousFrame();
var jointPositions = this.DrawJoints(body);
this.DrawLines(jointPositions);
}
public void ClearFrame()
{
this.RemoveJointsFromPreviousFrame();
this.RemoveLinesFromPreviousFrame();
}
void RemoveJointsFromPreviousFrame()
{
foreach (var item in this.drawnJointElements)
{
this.model3dGroup.Children.Remove(item.Value);
}
this.drawnJointElements.Clear();
}
void RemoveLinesFromPreviousFrame()
{
foreach (var item in this.drawnLineElements)
{
this.model3dGroup.Children.Remove(item);
}
this.drawnLineElements.Clear();
}
Dictionary<JointType, CameraSpacePoint> DrawJoints(Body body)
{
Dictionary<JointType, CameraSpacePoint> jointPositions =
new Dictionary<JointType, CameraSpacePoint>();
foreach (var item in body.Joints)
{
JointType jointType = item.Key;
Joint joint = item.Value;
bool draw = IsJointForDrawing(joint, joint.Position);
ModelVisual3D modelVisual3d = null;
if (draw && !this.drawnJointElements.TryGetValue(jointType, out modelVisual3d))
{
modelVisual3d = MakeModelForJointType(jointType,
joint.TrackingState != TrackingState.Inferred);
this.model3dGroup.Children.Add(modelVisual3d);
this.drawnJointElements[jointType] = modelVisual3d;
}
if (draw)
{
// have to be careful because the inferred/tracked status can change
// after we create the visual so we might need to update it here to
// make sure it's showing the right model (i.e. colour).
if (joint.TrackingState == TrackingState.Tracked)
{
modelVisual3d.Content = this.sphereVisual3d.Model;
}
else
{
modelVisual3d.Content = inferredSphereVisual3d.Model;
}
Transform3DGroup group = (Transform3DGroup)modelVisual3d.Transform;
TranslateTransform3D translate = (TranslateTransform3D)group.Children[1];
translate.OffsetX = joint.Position.X;
translate.OffsetY = joint.Position.Y;
translate.OffsetZ = 1 - joint.Position.Z;
jointPositions.Add(jointType, joint.Position);
}
else if (modelVisual3d != null)
{
this.model3dGroup.Children.Remove(modelVisual3d);
this.drawnJointElements.Remove(jointType);
}
}
return (jointPositions);
}
void DrawLines(IReadOnlyDictionary<JointType,CameraSpacePoint> jointPositions)
{
foreach (var jointConnection in jointConnections)
{
// that little data structure either contains a list of joints to work through or
// a start joint and an element count. it's discriminated and shouldn't contain
// both!
jointConnection.ForEachPair(
(j1, j2) =>
{
if (jointPositions.ContainsKey(j1) && jointPositions.ContainsKey(j2))
{
CameraSpacePoint p1 = jointPositions[j1];
CameraSpacePoint p2 = jointPositions[j2];
TubeVisual3D tube = MakeTubeForPositions(p1, p2);
this.model3dGroup.Children.Add(tube);
this.drawnLineElements.Add(tube);
}
}
);
}
}
static bool IsJointForDrawing(Joint joint, CameraSpacePoint p)
{
return (
(joint.TrackingState != TrackingState.NotTracked) &&
(!double.IsInfinity(p.X)) &&
(!double.IsInfinity(p.Y)) &&
(!double.IsInfinity(p.Z)));
}
static TubeVisual3D MakeTubeForPositions(CameraSpacePoint p1, CameraSpacePoint p2)
{
var tube = new TubeVisual3D();
tube.Diameter = Constants.TubeDiameter;
Point3DCollection points = new Point3DCollection();
points.Add(p1.ToPoint3D());
points.Add(p2.ToPoint3D());
tube.Path = points;
tube.Fill = tubeBrush;
return (tube);
}
ModelVisual3D MakeModelForJointType(JointType jointType,
bool trackedJoint)
{
JointType[] leafTypes =
{
JointType.Head
};
var isLeaf = leafTypes.Contains(jointType);
var modelVisual3d = new ModelVisual3D();
modelVisual3d.Content =
trackedJoint ? this.sphereVisual3d.Model : inferredSphereVisual3d.Model;
Transform3DGroup transform = new Transform3DGroup();
transform.Children.Add(new ScaleTransform3D()
{
ScaleX = isLeaf ? Constants.LeafScaleSize : Constants.RegularScaleSize,
ScaleY = isLeaf ? Constants.LeafScaleSize : Constants.RegularScaleSize,
ScaleZ = isLeaf ? Constants.LeafScaleSize : Constants.RegularScaleSize
});
transform.Children.Add(new TranslateTransform3D());
modelVisual3d.Transform = transform;
return (modelVisual3d);
}
SphereVisual3D sphereVisual3d;
Dictionary<JointType, ModelVisual3D> drawnJointElements;
List<TubeVisual3D> drawnLineElements;
ModelVisual3D model3dGroup;
static readonly Brush inferredBrush = Brushes.LightGray;
static SphereVisual3D inferredSphereVisual3d;
static readonly Brush tubeBrush = Brushes.White;
// This is bad really because it depends on the actual enum values for the JointType
// type in the SDK not changing.
// It's easier though than having some massive array of all the connections but that
// would be the right thing to do I think.
static JointConnection[] jointConnections =
{
new JointConnection(JointType.SpineBase, 2),
new JointConnection(JointType.ShoulderLeft, 4),
new JointConnection(JointType.ShoulderRight, 4),
new JointConnection(JointType.HipLeft, 4),
new JointConnection(JointType.HipRight, 4),
new JointConnection(JointType.Neck, 2),
new JointConnection(JointType.SpineMid, JointType.SpineShoulder, JointType.Neck),
new JointConnection(JointType.ShoulderLeft, JointType.SpineShoulder, JointType.ShoulderRight),
new JointConnection(JointType.HipLeft, JointType.SpineBase, JointType.HipRight),
new JointConnection(JointType.HandTipLeft, JointType.HandLeft),
new JointConnection(JointType.HandTipRight, JointType.HandRight),
new JointConnection(JointType.WristLeft, JointType.ThumbLeft),
new JointConnection(JointType.WristRight, JointType.ThumbRight)
};
}
}
In terms of connecting joints together, this takes the exact same approach as the previous 2D example – it relies on an array of JointConnection instances to tell the drawing code which joints need to be connected up and I won’t list that JointConnection class here in the blog post as it hasn’t changed.
That’s pretty much it – the moved from 2D to 3D took me around two hours or so but most of that was spent in trying to remember some of what I used to know about how WPF deals with 3D and also trying to figure out what the Helix toolkit could do for me in terms of making 3D drawing easier. I’m really impressed by that library – it helped me out a lot here but it’s seriously lacking when it comes to documentation (as I think the author of the library knows based on the CodePlex page for it).
In terms of source – the code for the above is here for download.
In terms of next steps – I’d like to do 2 things. One is that all of what I’ve done so far has been inside of .NET Framework 4.5.x applications – both console and WPF. I’d like to do a couple of things with this code;
- Properly sort out my IBodyDrawer interface such that I could have both 2D and 3D drawing inside the same application.
- Start to take a look at how much I’d have to change the code that I’ve built up so far in order to get it running inside of a WinRT application.
before thinking about other sources of data that the Kinect for Windows SDK can provide like the body index data and so on and other areas entirely like gestures and audio.
Mike Taulty 