Adding to this set of posts and particularly this last post I wanted to experiment more with hand tracking.
I went back to the WPF application ‘framework’ that I’ve used in previous posts and I wrote a new control that I called HandInformationControl and I plugged it into my MainWindow UI and ‘processing framework’ that I’ve used in previous posts;
<controls:ColorVideoControl Grid.Row="1"
Grid.Column="1" />
<controls:HandInformationControl Grid.Row="1"
Grid.Column="1" />
and so now I’ve got color video being drawn underneath some information about hand tracking.
From the previous post, I know that hands go around a bit of a lifecycle in that there are many alerts that fire around whether a hand is tracked, too near to the camera and so on and they are represented by this type in the SDK;
public enum AlertType
{
ALERT_HAND_DETECTED = 1,
ALERT_HAND_NOT_DETECTED = 2,
ALERT_HAND_TRACKED = 4,
ALERT_HAND_NOT_TRACKED = 8,
ALERT_HAND_CALIBRATED = 16,
ALERT_HAND_NOT_CALIBRATED = 32,
ALERT_HAND_OUT_OF_BORDERS = 64,
ALERT_HAND_INSIDE_BORDERS = 128,
ALERT_HAND_OUT_OF_LEFT_BORDER = 256,
ALERT_HAND_OUT_OF_RIGHT_BORDER = 512,
ALERT_HAND_OUT_OF_TOP_BORDER = 1024,
ALERT_HAND_OUT_OF_BOTTOM_BORDER = 2048,
ALERT_HAND_TOO_FAR = 4096,
ALERT_HAND_TOO_CLOSE = 8192,
ALERT_HAND_LOW_CONFIDENCE = 16384,
}
and it seemed to me that the ‘perfect’ state for a hand to be in was the state of DETECTED | TRACKED | CALIBRATED | INSIDE_BORDERS. With that in mind, I wrote this little class to try and track all these events and be able to tell me at any time whether there were any tracked hands and what sort of state they were in. That is;
namespace WpfApplication2.Controls
{
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.Text;
class HandAlertManager
{
[Flags]
public enum HandStatus
{
Ok =
PXCMHandData.AlertType.ALERT_HAND_CALIBRATED |
PXCMHandData.AlertType.ALERT_HAND_DETECTED |
PXCMHandData.AlertType.ALERT_HAND_INSIDE_BORDERS |
PXCMHandData.AlertType.ALERT_HAND_TRACKED,
NotTracked =
PXCMHandData.AlertType.ALERT_HAND_CALIBRATED |
PXCMHandData.AlertType.ALERT_HAND_DETECTED |
PXCMHandData.AlertType.ALERT_HAND_INSIDE_BORDERS,
NotCalibrated =
PXCMHandData.AlertType.ALERT_HAND_TRACKED |
PXCMHandData.AlertType.ALERT_HAND_DETECTED |
PXCMHandData.AlertType.ALERT_HAND_INSIDE_BORDERS,
NotInsideBorders =
PXCMHandData.AlertType.ALERT_HAND_CALIBRATED |
PXCMHandData.AlertType.ALERT_HAND_DETECTED |
PXCMHandData.AlertType.ALERT_HAND_TRACKED
}
public HandAlertManager(PXCMHandConfiguration handConfig)
{
this.statusValues = new Dictionary<int, HandStatus>();
handConfig.EnableAllAlerts();
handConfig.SubscribeAlert(this.OnAlert);
}
void OnAlert(PXCMHandData.AlertData alertData)
{
// NB: ignoring the low confidence state as I'm not sure what to do with
// it because there is no 'high' confidence state.
// Also, assumed that if I get an out of bottom/left/right/top then I
// will also get an OUT_OF_BORDERS so I'm ignoring the 4 explicit states.
// Also assuming that a HAND_TOO_CLOSE/FAR is followed by an INSIDE_BORDERS
// as that seems to happen.
HandStatus handStatus = HandStatus.Ok;
bool alertTypeToAdd = false;
if (this.statusValues.ContainsKey(alertData.handId))
{
handStatus = this.statusValues[alertData.handId];
}
switch (alertData.label)
{
case PXCMHandData.AlertType.ALERT_HAND_CALIBRATED:
case PXCMHandData.AlertType.ALERT_HAND_DETECTED:
case PXCMHandData.AlertType.ALERT_HAND_INSIDE_BORDERS:
case PXCMHandData.AlertType.ALERT_HAND_TRACKED:
handStatus |= (HandStatus)alertData.label;
alertTypeToAdd = true;
break;
case PXCMHandData.AlertType.ALERT_HAND_NOT_CALIBRATED:
handStatus &= (HandStatus)~PXCMHandData.AlertType.ALERT_HAND_CALIBRATED;
break;
case PXCMHandData.AlertType.ALERT_HAND_NOT_DETECTED:
handStatus &= (HandStatus)~PXCMHandData.AlertType.ALERT_HAND_DETECTED;
break;
case PXCMHandData.AlertType.ALERT_HAND_NOT_TRACKED:
handStatus &= (HandStatus)~PXCMHandData.AlertType.ALERT_HAND_TRACKED;
break;
case PXCMHandData.AlertType.ALERT_HAND_OUT_OF_BORDERS:
case PXCMHandData.AlertType.ALERT_HAND_TOO_CLOSE:
case PXCMHandData.AlertType.ALERT_HAND_TOO_FAR:
case PXCMHandData.AlertType.ALERT_HAND_OUT_OF_BOTTOM_BORDER:
case PXCMHandData.AlertType.ALERT_HAND_OUT_OF_LEFT_BORDER:
case PXCMHandData.AlertType.ALERT_HAND_OUT_OF_RIGHT_BORDER:
case PXCMHandData.AlertType.ALERT_HAND_OUT_OF_TOP_BORDER:
handStatus &= (HandStatus)~PXCMHandData.AlertType.ALERT_HAND_INSIDE_BORDERS;
break;
default:
break;
}
if ((handStatus & (HandStatus)PXCMHandData.AlertType.ALERT_HAND_DETECTED) == 0)
{
// remove a value if we've been told that it's no longer detected.
this.statusValues.Remove(alertData.handId);
}
else if (this.statusValues.ContainsKey(alertData.handId) || alertTypeToAdd)
{
// store any value into an existing slot but don't add a new slot
// unless there's a good reason (i.e. not if we are receiving an
// update to a hand that's been removed already).
this.statusValues[alertData.handId] = handStatus;
}
}
public IDictionary<int, HandStatus> GetHandsInfo()
{
return (this.statusValues);
}
Dictionary<int, HandStatus> statusValues;
}
}
and so this class is trying to handle all the alerts that come in around hands and then be able to return (via GetHandsInfo) the current state of each hand that we’re aware of in terms of whether that hand is;
- Ok
- NotTracked
- NotCalibrated
- NotInsideBorders
With that in place I wrote my HandInformationControl to display a couple of ListBoxes (which I’m being very poor about in terms of not data-binding to);
<UserControl x:Class="WpfApplication2.Controls.HandInformationControl"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
mc:Ignorable="d"
d:DesignHeight="300"
d:DesignWidth="300">
<Grid>
<Grid.RowDefinitions>
<RowDefinition Height="*" />
<RowDefinition Height="6*" />
<RowDefinition Height="*" />
</Grid.RowDefinitions>
<Grid.ColumnDefinitions>
<ColumnDefinition />
<ColumnDefinition Width="6*" />
<ColumnDefinition />
</Grid.ColumnDefinitions>
<ListBox x:Name="listGestures"
Grid.Column="1"
HorizontalAlignment="Right"
Background="#CC000000"
Width="300"
VerticalAlignment="Center"
ScrollViewer.VerticalScrollBarVisibility="Auto"
ScrollViewer.HorizontalScrollBarVisibility="Disabled"
MaxHeight="200"
Grid.Row="1">
<ListBox.ItemTemplate>
<DataTemplate>
<TextBlock Text="{Binding}"
TextWrapping="NoWrap"
FontSize="24"
Foreground="White" />
</DataTemplate>
</ListBox.ItemTemplate>
</ListBox>
<ListBox x:Name="listHandStatus"
Grid.Column="1"
HorizontalAlignment="Center"
VerticalAlignment="Bottom"
Background="#CC000000"
Grid.Row="1">
<ListBox.ItemTemplate>
<DataTemplate>
<StackPanel Orientation="Horizontal">
<StackPanel.Resources>
<Style TargetType="TextBlock">
<Setter Property="FontSize"
Value="38" />
<Setter Property="Foreground"
Value="White" />
</Style>
</StackPanel.Resources>
<TextBlock Text="hand id ">
<Run Text="{Binding Key,Mode=OneWay}" />
<Run Text="is at status" />
<Run Text="{Binding Value,Mode=OneWay}" />
</TextBlock>
</StackPanel>
</DataTemplate>
</ListBox.ItemTemplate>
</ListBox>
</Grid>
</UserControl>
and then I added some code to sit with that;
namespace WpfApplication2.Controls
{
using System.Collections.Generic;
using System.Diagnostics;
using System.Windows.Controls;
using System.Linq;
using System.Collections.ObjectModel;
public partial class HandInformationControl : UserControl, ISampleRenderer
{
public HandInformationControl()
{
InitializeComponent();
}
public void Initialise(PXCMSenseManager senseManager)
{
this.senseManager = senseManager;
this.senseManager.EnableHand().ThrowOnFail();
using (var handModule = this.senseManager.QueryHand())
{
using (var handConfiguration = handModule.CreateActiveConfiguration())
{
this.alertManager = new HandAlertManager(handConfiguration);
handConfiguration.EnableAllGestures().ThrowOnFail();
handConfiguration.ApplyChanges().ThrowOnFail();
}
this.handData = handModule.CreateOutput();
}
}
public void ProcessSampleWorkerThread(PXCMCapture.Sample sample)
{
this.handStatusMap = null;
if (this.handData.Update().Succeeded())
{
var handsInfoFromAlerts = this.alertManager.GetHandsInfo();
if ((handsInfoFromAlerts != null) && (handsInfoFromAlerts.Count() > 0))
{
this.handStatusMap = new Dictionary<int, HandAlertManager.HandStatus>(
handsInfoFromAlerts);
this.HandleGestures();
}
}
}
void HandleGestures()
{
var handsAtGoodStatus =
this.handStatusMap
.Where(kvp => kvp.Value == HandAlertManager.HandStatus.Ok);
if (handsAtGoodStatus.Count() > 0)
{
if (this.gestureTrackingHandId != handsAtGoodStatus.First().Key)
{
this.gestureTrackingHandId = handsAtGoodStatus.First().Key;
this.gesturesTracked = null;
}
var gestureCount = this.handData.QueryFiredGesturesNumber();
PXCMHandData.GestureData gestureData;
if (gestureCount > 0)
{
for (int i = 0; i < gestureCount; i++)
{
if (this.handData.QueryFiredGestureData(i, out gestureData).Succeeded())
{
if ((gestureData.handId == this.gestureTrackingHandId) &&
(gestureData.state == PXCMHandData.GestureStateType.GESTURE_STATE_END))
{
if (this.gesturesTracked == null)
{
this.gesturesTracked = new List<string>();
}
this.gesturesTracked.Insert(0, gestureData.name);
}
}
}
}
}
else
{
this.gestureTrackingHandId = -1;
this.gesturesTracked = null;
}
}
public void RenderUI(PXCMCapture.Sample sample)
{
this.listHandStatus.ItemsSource = this.handStatusMap;
// very hacky, I should really do all this properly and use binding
// and observable collections and so on but for experimentation
// purposes I get away with it...
this.listGestures.ItemsSource = null;
this.listGestures.ItemsSource = this.gesturesTracked;
}
int gestureTrackingHandId;
List<string> gesturesTracked;
Dictionary<int, HandAlertManager.HandStatus> handStatusMap;
HandAlertManager alertManager;
PXCMSenseManager senseManager;
PXCMHandData handData;
}
}
with the essence of this being;
- at Initialise time we enable hand tracking and enable all gestures.
- as data arrives we;
- attempt to determine whether we have any tracked hands and their status via the HandAlertManager helper I introduced above.
- for any hands that are in an “ok” state, we attempt to see if any gesture of any kind has been detected.
- and the UI updates are really a matter of repeatedly changing the ItemsSource properties on the 2 list boxes (which is a bit poor, but it was cheap/cheerful code to write)
It’s perhaps worth saying that I don’t think I should really be switching on all gestures because there are many gestures in the SDK’s default set that the docs explicitly tell you not to combine with others because there’s a risk of confusion but, for simplicity of coding here, I’ve switched them all on in order to see what happens.
Here’s the app running and (it seems) working reasonably well;
and here’s the code for download.
Mike Taulty 