Windows 10, UWP, RealSense SR300, Faces and the Surface Pro 3

The last 2 posts were really a bit of a catalogue of failures around;

    1. Not getting the RealSense SR300 preview support for the UWP working.
    2. Not getting the RealSense SR300 support for person tracking working.

But I kept persevering and I’ve made some progress at least around step 1 above and the main piece of progress that I’ve made is;

Work on the Surface Pro 3

For whatever reason, I’ve found that plugging the SR300 into my Surface Pro 3 with its i7-4650 CPUs seems to work out better than plugging it into my Surface Book with its much newer CPUs and I suspect it’s something to do with the 2 USB ports on the Surface Book acting as some kind of ‘hub’ which the RealSense doesn’t like.

Going back to my first post then on RealSense with UWP, I made a blank project in that post and I made a MainPage style UI that simply had a Win2D CanvasControl on it;

        <w:CanvasControl
            xmlns:w="using:Microsoft.Graphics.Canvas.UI.Xaml"
            x:Name="canvasControl"
            Draw="OnDraw">
        </w:CanvasControl>

and I wrote code behind to try and display frames from the video camera (NB: code written quite quickly and quite late at night, don’t call it’s just for fun Smile);

  using Intel.RealSense;
  using Microsoft.Graphics.Canvas;
  using Microsoft.Graphics.Canvas.UI.Xaml;
  using System;
  using System.Threading;
  using System.Threading.Tasks;
  using Windows.Foundation;
  using Windows.Graphics.Imaging;
  using Windows.UI.Core;
  using Windows.UI.Popups;
  using Windows.UI.Xaml;
  using Windows.UI.Xaml.Controls;

  public sealed partial class MainPage : Page
  {
    public MainPage()
    {
      this.InitializeComponent();
      this.Loaded += OnLoaded;
    }
    async void OnLoaded(object sender, RoutedEventArgs e)
    {
      var status = Status.STATUS_NO_ERROR;

      // SenseManager is where we always start with RealSense.
      // Very glad to see that the long names of the .NET SDK
      // have gone away here.
      this.senseManager = SenseManager.CreateInstance();

      if (this.senseManager != null)
      {
        // I'm not yet sure how I switch on streams without going via this
        // SampleReader so I'm using it here to switch on colour at 1280x720
        // even though I'm then not going to use it again. In the .NET SDK
        // there's a SenseManager->EnableStream() type method.
        this.sampleReader = SampleReader.Activate(this.senseManager);
        this.sampleReader.EnableStream(StreamType.STREAM_TYPE_COLOR, 1280, 720, 0);

        // Callback for when samples arrive.
        this.sampleReader.SampleArrived += this.OnSampleArrived;

        // Initialise.
        status = await this.senseManager.InitAsync();

        if (status == Status.STATUS_NO_ERROR)
        {
          // Start firing the callbacks as frames arrive.
          status = this.senseManager.StreamFrames();
        }
      }
      if (status != Status.STATUS_NO_ERROR)
      {
        await this.DisplayErrorAsync(status);
      }
    }
    async void OnSampleArrived(object sender, SampleArrivedEventArgs e)
    {
      bool invalidate = false;
      var bitmap = e.Sample?.Color?.SoftwareBitmap;

      if (bitmap != null)
      {
        // A shame that the bitmap format that RealSense is passing me
        // here isn't supported by CanvasBitmap below. Hence the conversion
        // to Rgba8 which is does support.
        // This is also wasteful as we convert it before seeing whether
        // we have a free slot to draw it.
        var convertedBitmap = SoftwareBitmap.Convert(
          bitmap,
          BitmapPixelFormat.Rgba8, BitmapAlphaMode.Premultiplied);

        if (Interlocked.CompareExchange<SoftwareBitmap>(
          ref this.pendingBitmap, convertedBitmap, null) == null)
        {
          invalidate = true;
        }
      }
      if (invalidate)
      {
        await this.InvalidateAsync();
      }
      // TBD
      e.Sample.Dispose();
    }    
    async Task InvalidateAsync()
    {
      await this.Dispatcher.RunAsync(
        CoreDispatcherPriority.Normal,
        this.canvasControl.Invalidate);
    }
    void OnDraw(
      CanvasControl sender, CanvasDrawEventArgs args)
    {
      var bitmap = Interlocked.Exchange<SoftwareBitmap>(
        ref this.pendingBitmap, null);

      if (bitmap != null)
      {
        using (var canvasBitmap = CanvasBitmap.CreateFromSoftwareBitmap(
          args.DrawingSession.Device, bitmap))
        {
          args.DrawingSession.DrawImage(
            canvasBitmap,
            new Rect(0, 0, sender.ActualWidth, sender.ActualHeight));
        }
        bitmap.Dispose();
      }
    }
    async Task DisplayErrorAsync(Status status)
    {
      if (status != Status.STATUS_NO_ERROR)
      {
        var dialog = new MessageDialog(status.ToString(), "RealSense Error");
        await dialog.ShowAsync();
      }
    }
    SampleReader sampleReader;
    SoftwareBitmap pendingBitmap;
    SenseManager senseManager;
  }

and that all works fine and video displays and it seems like it performs fine although I haven’t done anything explicit to try to tune it.

But I had this working (sporadically) in the first post referenced above. Where I struggled was with face detection but that also works fine on my Surface Pro 3.

With the exact same Win2D UI, I can change my code behind to be something like;

  using Intel.RealSense;
  using Intel.RealSense.Face;
  using Microsoft.Graphics.Canvas;
  using Microsoft.Graphics.Canvas.UI.Xaml;
  using System;
  using System.Collections.Generic;
  using System.Threading;
  using System.Threading.Tasks;
  using Windows.Foundation;
  using Windows.Graphics.Imaging;
  using Windows.UI;
  using Windows.UI.Core;
  using Windows.UI.Popups;
  using Windows.UI.Xaml;
  using Windows.UI.Xaml.Controls;
  using System.Linq;

  public sealed partial class MainPage : Page
  {
    public MainPage()
    {
      this.InitializeComponent();
      this.Loaded += OnLoaded;
    }
    async void OnLoaded(object sender, RoutedEventArgs e)
    {
      var status = Status.STATUS_NO_ERROR;

      // SenseManager is where we always start with RealSense.
      // Very glad to see that the long names of the .NET SDK
      // have gone away here.
      this.senseManager = SenseManager.CreateInstance();

      if (this.senseManager != null)
      {
        // I'm not yet sure how I switch on streams without going via this
        // SampleReader so I'm using it here to switch on colour at 1280x720
        // even though I'm then not going to use it again. In the .NET SDK
        // there's a SenseManager->EnableStream() type method.
        this.faceModule = FaceModule.Activate(this.senseManager);

        var config = this.faceModule.CreateActiveConfiguration();

        config.Detection.IsEnabled = true;
        config.Detection.MaxTrackedFaces = 1;
        config.Landmarks.IsEnabled = true;
        config.Landmarks.MaxTrackedFaces = 1;

        // NB: I find this is crucial for getting faces detected.
        // No idea why, maybe it's broken in this preview?
        config.Pose.IsEnabled = false;

        status = config.ApplyChanges();

        if (status == Status.STATUS_NO_ERROR)
        {
          // Callback for when samples arrive.
          this.faceModule.FrameProcessed += OnFrameProcessed;

          // Initialise.
          status = await this.senseManager.InitAsync();

          if (status == Status.STATUS_NO_ERROR)
          {
            // Start firing the callbacks as frames arrive.
            status = this.senseManager.StreamFrames();
          }
        }
      }
      if (status != Status.STATUS_NO_ERROR)
      {
        await this.DisplayErrorAsync(status);
      }
    }
    async void OnFrameProcessed(object sender, FrameProcessedEventArgs e)
    {
      bool invalidate = false;
      var color = this.faceModule.Sample?.Color;

      if ((color != null) && (color.SoftwareBitmap != null))
      {
        // A shame that the bitmap format that RealSense is passing me
        // here isn't supported by CanvasBitmap below. Hence the conversion
        // to Rgba8 which is does support.
        // This is also wasteful as we convert it before seeing whether
        // we have a free slot to draw it.
        var convertedBitmap = SoftwareBitmap.Convert(
          color.SoftwareBitmap,
          BitmapPixelFormat.Rgba8, BitmapAlphaMode.Premultiplied);

        if (Interlocked.CompareExchange<SoftwareBitmap>(
          ref this.pendingBitmap, convertedBitmap, null) == null)
        {
          invalidate = true;
        }
      }
      if (e.Data.Faces.Count > 0)
      {
        float xScale = 1.0f / this.faceModule.Sample.Color.SoftwareBitmap.PixelWidth;
        float yScale = 1.0f / this.faceModule.Sample.Color.SoftwareBitmap.PixelHeight;
        
        // Scale this points to be relative 0..1 values so that we can
        // multiply them back up later. We can also get the bounding
        // rectangle and the average depth of the face here.
        List<Point> relativePoints =
          e.Data.Faces[0].Landmarks.Points.Select(
            point => new Point()
            {
              X = point.image.X * xScale,
              Y = point.image.Y * yScale
            }
          ).ToList();

        this.averageDepth = e.Data.Faces[0].Detection.FaceAverageDepth;

        if (Interlocked.CompareExchange<List<Point>>(
          ref this.landmarkPoints,
          relativePoints,
          null) == null)
        {
          invalidate = true;
        }
      }
      else
      {
        this.averageDepth = float.NaN;
      }
      if (invalidate)
      {
        await this.InvalidateAsync();
      }
    }
    async Task InvalidateAsync()
    {
      await this.Dispatcher.RunAsync(
        CoreDispatcherPriority.Normal,
        this.canvasControl.Invalidate);
    }
    void OnDraw(
      CanvasControl sender, CanvasDrawEventArgs args)
    {
      var bitmap = Interlocked.Exchange<SoftwareBitmap>(
        ref this.pendingBitmap, null);

      if (bitmap != null)
      {
        using (var canvasBitmap = CanvasBitmap.CreateFromSoftwareBitmap(
          args.DrawingSession.Device, bitmap))
        {
          args.DrawingSession.DrawImage(
            canvasBitmap,
            new Rect(0, 0, sender.ActualWidth, sender.ActualHeight));
        }
        bitmap.Dispose();
      }
      var landmarkData = Interlocked.Exchange<List<Point>>(
        ref this.landmarkPoints, null);

      if (landmarkData != null)
      {
        foreach (var landmark in landmarkData)
        {
          args.DrawingSession.FillCircle(
            (float)(landmark.X * sender.ActualWidth),
            (float)(landmark.Y * sender.ActualHeight),
            CIRCLE_RADIUS,
            Colors.White);
        }
        args.DrawingSession.DrawText(
          $"Average depth {(int)this.averageDepth}mm",
          new System.Numerics.Vector2(10, 10),
          Colors.White);
      }
    }
    async Task DisplayErrorAsync(Status status)
    {
      if (status != Status.STATUS_NO_ERROR)
      {
        var dialog = new MessageDialog(status.ToString(), "RealSense Error");
        await dialog.ShowAsync();
      }
    }
    const float CIRCLE_RADIUS = 5.0f;
    float averageDepth;
    FaceModule faceModule;
    List<Point> landmarkPoints;
    SoftwareBitmap pendingBitmap;
    SenseManager senseManager;
  }

Note that I didn’t spend a lot of time thinking about that Interlocked.* stuff being used to move data from one thread to the other so I hope that’s at least “kind of ok” in acting as a form of queue that only has 1 element in it.

Anyway, for a quick hack it seems to work quite nicely and quite smoothly and I get facial landmarks detected – quick video capture of that below;

Note that the video makes it look like the perf is bad, it’s much, much better when I’m not trying to capture the screen at the same time.

I also think that the accuracy and the range seems a lot better than with the F200 – as you’ll spot in the video, I can go to about 1m before it loses the face.

I’m really pleased that I can finally get some of this working inside of the UWP – I think it’s a great step forward for the RealSense SDK and, for me, it makes it a lot easier to include in demos around UWP.

One thought on “Windows 10, UWP, RealSense SR300, Faces and the Surface Pro 3

Comments are closed.