Kinect for Windows V2 SDK: Hello ‘Custom Gesture’ World Part 2

Following up on that previous post Kinect for Windows V2 SDK- Hello ‘Custom Gesture’ World where I’d made a basic training database containing a custom continuous gesture, I wanted to take the next step and have some of my own code respond to that gesture.

With that in mind, I went and made a quick, blank WPF project and added references to the relevant assemblies;

image

and I also tweaked my build configuration such that I was building code for a specific processor (x64 in this case);

image

I built a tiny piece of UI which has some buttons to;

  • load up the database containing my custom gesture
  • open/close the Kinect sensor
  • open/close the body frame and gesture readers that I’ll need to recognise my gesture

that UI is just some buttons and a big arrow as per below;

image

and the buttons are linked to some code behind and there’s also a little TextBlock in there as well which can display a 0..1 value for how far any gesture in progress has progressed;

    <StackPanel Orientation="Horizontal"
                VerticalAlignment="Bottom">
      <StackPanel.Resources>
        <Style TargetType="Button">
          <Setter Property="Margin"
                  Value="4" />
        </Style>
      </StackPanel.Resources>
      <Button Content="Load Gesture from DB"
              Click="OnLoadGestureFromDb" />
      <Button Content="Open Sensor"
              Click="OnOpenSensor" />
      <Button Content="Open Readers"
              Click="OnOpenReaders" />
      <Button Content="Close Readers"
              Click="OnCloseReaders" />
      <Button Content="Close Sensor"
              Click="OnCloseSensor" />
    </StackPanel>
    <TextBlock x:Name="txtProgress"
               FontSize="48"
               VerticalAlignment="Top"
               HorizontalAlignment="Right" />

and then the Path that looks a little bit like an arrow just has a named GradientStop within it so that I can get to that stop from code and manipulate it;

          <Path.Fill>
            <LinearGradientBrush>
              <LinearGradientBrush.GradientStops>
                <GradientStop Color="Red"
                              Offset="0" />
                <GradientStop Color="White"
                              Offset="0"
                              x:Name="moveableStop" />
                <GradientStop Color="White"
                              Offset="1" />
              </LinearGradientBrush.GradientStops>
            </LinearGradientBrush>
          </Path.Fill>

With that little piece of UI in place, the next step is write code which loads up the .gbd file that the Visual Gesture Builder tool produced for me as part of the previous blog post and to have code which extracts the single gesture that resides in that file. Note that it’d be more usual to have a number of gestures in a database like this but I just have the one here.

I added the .gbd file to my project;

image

and made sure that Visual Studio was configured to copy that file into the output folder as part of the build process.

With that in place, I set about writing the code to open up the DB and to extract my gesture;

  public partial class MainWindow : Window
  {
    Gesture swipeForwardGesture;

    public MainWindow()
    {
      InitializeComponent();
    }
    void OnLoadGestureFromDb(object sender, RoutedEventArgs e)
    {
      // we assume that this file exists and will load
      VisualGestureBuilderDatabase db = new VisualGestureBuilderDatabase(
        @"GestureDatabase.gbd");

      // we assume that this gesture is in that database (it should be, it's the only
      // gesture in there).
      this.swipeForwardGesture =
        db.AvailableGestures.Where(g => g.Name == "swipeForwardProgress").Single();
    }
}

That’s pretty easy – open the DB and find the gesture by name. The next thing to do is to write the code which opens/closes the sensor which is code that I’ve written in a number of recent posts;

  public partial class MainWindow : Window
  {
    Gesture swipeForwardGesture;
    KinectSensor sensor;

    // Omitted the functions already covered...

    void OnOpenSensor(object sender, RoutedEventArgs e)
    {
      // assuming that the sensor is available and that it's not already open - 
      // i.e. assuming that the "user" will only press the buttons in a sensible
      // order which is not perhaps the best idea but then the user is primarily
      // me.
      this.sensor = KinectSensor.GetDefault();
      this.sensor.Open();
    }
    void OnCloseSensor(object sender, RoutedEventArgs e)
    {
      this.sensor.Close();
      this.sensor = null;
    }
}

and that’s all fairly standard although I should really be checking for sensor availability and changes in that availability but I haven’t done that here but it has featured in code in some of my previous posts.

Next up is to open up the reader for frames of body data which, again, I’ve done in previous posts;

  public partial class MainWindow : Window
  {
    Gesture swipeForwardGesture;
    KinectSensor sensor;
    Body[] bodies;
    BodyFrameReader bodyReader;
    VisualGestureBuilderFrameSource gestureSource;
    VisualGestureBuilderFrameReader gestureReader;

    // Omitted functions already covered...

    void OnOpenReaders(object sender, RoutedEventArgs e)
    {
      this.OpenBodyReader();
      this.OpenGestureReader();
    }    
    void OpenBodyReader()
    {
      if (this.bodies == null)
      {
        this.bodies = new Body[this.sensor.BodyFrameSource.BodyCount];
      }
      this.bodyReader = this.sensor.BodyFrameSource.OpenReader();
      this.bodyReader.FrameArrived += OnBodyFrameArrived;
    } 
}

and then that code is missing function OpenGestureReader which is where new things happen in terms of getting the infrastructure up and running to receive custom gesture events. Here’s that function;

    void OpenGestureReader()
    {
      this.gestureSource = new VisualGestureBuilderFrameSource(this.sensor, 0);

      this.gestureSource.AddGesture(this.swipeForwardGesture);

      this.gestureSource.TrackingIdLost += OnTrackingIdLost;

      this.gestureReader = this.gestureSource.OpenReader();
      this.gestureReader.IsPaused = true;
      this.gestureReader.FrameArrived += OnGestureFrameArrived;
    }  

and so all this is doing is;

  • Creating the VisualGestureBuilderFrameSource and passing it both the sensor and the tracking id
    • NB: this tracking id of 0 isn’t a valid tracking id as my code won’t have a valid tracking ID until tracked bodies start to arrive via the body frame reading code. For now, I pass 0.
  • Telling the VisualGestureBuilderFrameSource about the gesture that I want to track – this is the gesture I populated from the database – via the AddGesture() call.
  • Handling the TrackingIdLost event so that we know as/when the framework loses track of the id of the body that we have given it.
  • Opening up a VisualGestureBuilderFrameReader which returns frames of data as gestures are recognised.
  • Pausing that reader until a real tracking ID arrives.
  • Handling a FrameArrived event on that VisualGestureBuilderFrameReader in order to pick up the frames of data for recognised gestures as they come along.

I need some code to close off those readers at a later point and some code to handle that TrackingIdLost event which I do by pausing the gesture reader and resetting the UI – both as per below;

    void OnCloseReaders(object sender, RoutedEventArgs e)
    {
      if (this.gestureReader != null)
      {
        this.gestureReader.FrameArrived -= this.OnGestureFrameArrived;
        this.gestureReader.Dispose();
        this.gestureReader = null;
      }
      if (this.gestureSource != null)
      {
        this.gestureSource.TrackingIdLost -= this.OnTrackingIdLost;
        this.gestureSource.Dispose();
      }
      this.bodyReader.Dispose();
      this.bodyReader = null;
    }
    void OnTrackingIdLost(object sender, TrackingIdLostEventArgs e)
    {
      this.gestureReader.IsPaused = true;
      this.moveableStop.Offset = 0.0f;
      this.txtProgress.Text = string.Empty;
    }

With all of that out of the way, the main business of this code behind is to handle the body frames as they arrive and the code that I have here checks to see if there is a body being tracked or not and attempts to update the UI in the former case and to update the gesture source and reader in the latter case as below;

    void OnBodyFrameArrived(object sender, BodyFrameArrivedEventArgs e)
    {
      using (var frame = e.FrameReference.AcquireFrame())
      {
        if (frame != null)
        {
          frame.GetAndRefreshBodyData(this.bodies);

          var trackedBody = this.bodies.Where(b => b.IsTracked).FirstOrDefault();

          if (trackedBody != null)
          {
            if (this.gestureReader.IsPaused)
            {
              this.gestureSource.TrackingId = trackedBody.TrackingId;
              this.gestureReader.IsPaused = false;
            }
          }
          else
          {
            this.OnTrackingIdLost(null, null);
          }
        }
      }
    }

the key part of that code being that the TrackingId of the first body being reported as tracked by the sensor is fed through to the underlying VisualGestureBuilderFrameSource such that it is then tracking gestures on the right body.

The last piece of code is the handler for the frames of data arriving from the VisualGestureBuilderFrameReader itself. Compared with some of the other frame types on the Kinect (e.g. video, infra-red, skeletal, etc) this data source is relatively simple in that my processing here is largely just a matter of extracting a single floating point value which represents “progress of the gesture”;

    void OnGestureFrameArrived(object sender, VisualGestureBuilderFrameArrivedEventArgs e)
    {
      using (var frame = e.FrameReference.AcquireFrame())
      {
        if (frame != null)
        {
          var continuousResults = frame.ContinuousGestureResults;

          if ((continuousResults != null) &&
            (continuousResults.ContainsKey(this.swipeForwardGesture)))
          {
            var result = continuousResults[this.swipeForwardGesture];

            // change the gradient stop on the screen so that it reflects where
            // we are in terms of progress of the gesture.
            this.moveableStop.Offset = Math.Max(result.Progress, 0.0f);
            this.txtProgress.Text = Math.Max(result.Progress, 0.0f).ToString("G1");
          }
        }
      }
    }

Note that the code looks at the ContinuousGestureResults because my gesture is one that progresses from 0 to 1 rather than a discrete on/off gesture which resides in a different dictionary.

That’s it – in essence;

  • Open the gestures database & extract my gestures.
  • Open the sensor and open up body readers and gesture readers.
  • Wait for frames of gesture data to become available and check out the Progress value for any gestures that I’ve expressed interest in.

I added a KinectUserViewer control to my WPF application and ran it up with results as below;

which I think worked pretty well given that the gesture had so little training and, once again, hats off to the Kinect SDK people for making it so easy to load up one of these gestures and get back the data from it as it’s being tracked.

I should add, I only worked with a single body in this post but it’s easily expandable out to the full 6 and the SDK sample does exactly that for a “seated” gesture.

If you want the code for this – it’s here for download.

The Small Print

If you were to download that code and attempt to run it, you’d almost certainly hit the exception that I hit when I was putting it together which looks like this;

image

Essentially, what you’re getting is a DirectoryNotFoundException “This API has returned an exception from an HRESULT: 0x80070003”.

I puzzled over this for quite some time when I first hit it. I had no real clue which directory the underlying code might be looking for at this point where I had called VisualGestureBuilderFrameSource.AddGesture and I wondered if my .gdb file was corrupt.

It turned out to not be the case. Essentially, what I was missing was that I needed to go into the Kinect SDK folder which on my machine lives here;

image

and I needed to choose the right architecture x86/x64 and then go into that folder to find;

image

and then copy the vgbtechs folder ( which contains the 2 DLLs for the different algorithms for gesture detection ) into the output folder of my WPF project – i.e.

image

and that solved the problem for me. I got that solution from this forum post but, as you can probably imagine, that was after a bit of searching and so I’m reproducing it here.