Continuing the posts that I’ve written recently on the RealSense camera, I wanted to look at the more detailed aspects of hand tracking beyond just knowing whether hands are in front of the camera.
I kept with the framework that I’ve had in the previous posts where I’ve used WPF and I have a MainWindow which hosts a number of controls that implement an interface of my own in order to consume data from the camera.
In order to avoid complexities of having to map co-ordinate systems (which the SDK does help with) I decided to work with imagery coming from the depth camera and so I made some slight modifications to the ColorVideoControl that I’d written for posts like this one and changed it into a DepthVideoControl but, frankly, this is just a matter of a few tiny tweaks to the code (which is testament to the design of the APIs in the SDK) and so I won’t post the code for that control inline here as there’s nothing much new.
What was new for me was adding to my UI a new control that I wrote which I called a HandDisplay2DControl and I dropped it into my MainWindow.xaml file ‘on top’ of the DepthViewControl;
<controls:DepthVideoControl Grid.Row="1"
Grid.Column="1" />
<controls:HandDisplay2DControl Grid.Row="1"
Grid.Column="1" />
that new control has a very simple XAML file – it’s just a Canvas;
<UserControl x:Class="WpfApplication2.Controls.HandDisplay2DControl"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
mc:Ignorable="d"
d:DesignHeight="300" d:DesignWidth="300">
<Grid>
<Canvas x:Name="canvas" />
</Grid>
</UserControl>
and then the code behind has an Initialise routine which;
- Calls PXCMSenseManager.EnableHand() to switch on the hand module (PXCMHandModule)
- In a very similar way to facial tracking the hand module is asked to CreateActiveConfiguration() which gives back a PXCMHandConfiguration
- I made sure to call EnableTrackedJoints() on the PXCMHandConfiguration in order to switch on joint tracking
- I re-used my HandAlertManager class from my previous post such as to track when hands are detected/tracked/lost/calibrated and so on based on the alerts received.
Joint tracking is an ‘interesting’ thing 
It feels very similar to what happens in the Kinect v2 SDK except where with the Kinect you receive 25 different skeletal joints per tracked body, here in the RealSense SDK you receive details of all these hand joints per tracked hand;
public enum JointType
{
JOINT_WRIST = 0,
JOINT_CENTER = 1,
JOINT_THUMB_BASE = 2,
JOINT_THUMB_JT1 = 3,
JOINT_THUMB_JT2 = 4,
JOINT_THUMB_TIP = 5,
JOINT_INDEX_BASE = 6,
JOINT_INDEX_JT1 = 7,
JOINT_INDEX_JT2 = 8,
JOINT_INDEX_TIP = 9,
JOINT_MIDDLE_BASE = 10,
JOINT_MIDDLE_JT1 = 11,
JOINT_MIDDLE_JT2 = 12,
JOINT_MIDDLE_TIP = 13,
JOINT_RING_BASE = 14,
JOINT_RING_JT1 = 15,
JOINT_RING_JT2 = 16,
JOINT_RING_TIP = 17,
JOINT_PINKY_BASE = 18,
JOINT_PINKY_JT1 = 19,
JOINT_PINKY_JT2 = 20,
JOINT_PINKY_TIP = 21,
}
and so there’s a potential of 22 tracked ‘joints’ per hand and there’s a nice little diagram in the SDK which shows where they all sit;
When data arrives from the sensor, my code simply uses the same method as in the previous post to attempt to figure out which (if any) hands are currently in ‘a good state’ before interrogating them any further (i.e. a good state being detected + tracked + inside boundaries + calibrated).
It’s worth saying that this is probably overkill as you don’t need to wait for the calibrated event to make this joint detection work, it’s just something that I wrote into my code here.
Once I have a set of tracked hands, there are methods to get the co-ordinate data based on the hand identifier and the JointType as listed above and I simply build up a deferred list of shapes to be drawn at the right positions when my UI thread comes around to do the ‘rendering’. I’ve kept the rendering to the Canvas very simple in that I clear the Canvas on each frame of data and re-create it which is wasteful but simple.
namespace WpfApplication2.Controls
{
using System;
using System.Collections.Generic;
using System.Linq;
using System.Windows.Controls;
using System.Windows.Media;
using System.Windows.Shapes;
public partial class HandDisplay2DControl : UserControl, ISampleRenderer
{
enum ShapeType
{
Circle,
Box
}
public HandDisplay2DControl()
{
InitializeComponent();
}
public void Initialise(PXCMSenseManager senseManager)
{
this.senseManager = senseManager;
this.senseManager.EnableHand().ThrowOnFail();
using (var handModule = this.senseManager.QueryHand())
{
using (var handConfiguration = handModule.CreateActiveConfiguration())
{
this.alertManager = new HandAlertManager(handConfiguration);
handConfiguration.EnableTrackedJoints(true).ThrowOnFail();
handConfiguration.ApplyChanges().ThrowOnFail();
}
this.handData = handModule.CreateOutput();
}
}
public void ProcessSampleWorkerThread(PXCMCapture.Sample sample)
{
// this is me being very lazy and not inventing a class for this tuple...
this.drawShapes = new List<Tuple<ShapeType, PXCMRectI32, Brush, int>>();
if (this.handData.Update().Succeeded())
{
var handsInfoFromAlerts = this.alertManager.GetHandsInfo();
if ((handsInfoFromAlerts != null) && (handsInfoFromAlerts.Count() > 0))
{
var goodHands =
handsInfoFromAlerts
.Where(kvp => kvp.Value == HandAlertManager.HandStatus.Ok);
foreach (var entry in goodHands)
{
PXCMHandData.IHand iHand;
// gather the data to display that hand.
if (this.handData.QueryHandDataById(entry.Key, out iHand).Succeeded())
{
this.AddBox(iHand.QueryBoundingBoxImage(), Brushes.White, 2);
foreach (PXCMHandData.JointType joint in Enum.GetValues(typeof(PXCMHandData.JointType)))
{
PXCMHandData.JointData jointData;
if (iHand.QueryTrackedJoint(joint, out jointData).Succeeded())
{
this.AddCircle(
(int)jointData.positionImage.x, (int)jointData.positionImage.y,
10,
Brushes.Blue,
1);
}
}
}
}
}
}
}
public void RenderUI(PXCMCapture.Sample sample)
{
// being lazy and redrawing everything every time.
this.canvas.Children.Clear();
for (int i = 0; i < this.drawShapes.Count; i++)
{
this.MakeShape(this.drawShapes[i].Item1, this.drawShapes[i].Item2,
this.drawShapes[i].Item3, this.drawShapes[i].Item4);
}
}
private Shape MakeShape(ShapeType shapeType, PXCMRectI32 box,
Brush brush, int thickness)
{
Shape shape = shapeType == ShapeType.Box ? (Shape)new Rectangle() : (Shape)new Ellipse();
shape.Stroke = brush;
shape.StrokeThickness = thickness;
shape.Width = box.w;
shape.Height = box.h;
Canvas.SetLeft(shape, box.x);
Canvas.SetTop(shape, box.y);
this.canvas.Children.Add(shape);
return (shape);
}
void AddBox(PXCMRectI32 rectangle, Brush brush, int thickness)
{
this.drawShapes.Add(Tuple.Create(ShapeType.Box, rectangle, brush, thickness));
}
void AddCircle(int x, int y, int radius, Brush brush, int thickness)
{
this.drawShapes.Add(Tuple.Create(ShapeType.Circle,
new PXCMRectI32()
{
x = x - radius,
y = y - radius,
w = radius * 2,
h = radius * 2
},
brush,
thickness));
}
List<Tuple<ShapeType, PXCMRectI32, Brush, int>> drawShapes;
HandAlertManager alertManager;
PXCMHandData handData;
PXCMSenseManager senseManager;
}
}
and that’s all it takes to get ‘finger tracking’ switched on and working at least in a basic way. Here’s a quick screen capture of what that looks like in action where you can see the bounding box being drawn around the hand and blue circles being drawn around the joints;
and here’s the solution for download if you want to try it out on your camera at some point.
Mike Taulty 