Windows 10 UWP, Intel RealSense SR300, First Steps

Posted on by

I’ve been waiting a little while for the new SR300 RealSense camera to show up from Intel and mine got delivered the other so I thought I’d write an initial post having played with it for around 30 minutes or so.

Firstly, I thought that the camera might look like this;

but it turned out to look exactly like the existing F200 developer kit that I have which looks like this;

I think the first picture is a much better looking camera but I guess it’s not really a beauty contest when it comes to depth cameras (I think the first one is the forthcoming Razr camera).

Secondly, I plugged the new camera into the USB3.0 port on my Surface Book (I have one on loan right now) and it seemed initially to “just work” – that is, a bunch of drivers got installed and then the camera samples seemed to run first time against this camera and with a degree of speed, fluidity that I hadn’t noticed before on the F200.

I’m unsure whether this extra speed/fluidity comes from the new camera or the power of the Surface Book because I haven’t been able to try my existing F200 camera on the Surface Book as it seemed to have some compatibility problems and I’ll come back to that in a moment.

For the SR300, my Device Manager shows;

image

I think the top one is the built-in Surface Book camera and I think the other three are the SR300 (as the names suggest).

There’s a couple of things that I’m really keen to try out with this new camera.

    1. Person tracking – the SDK docs suggest that this ranges up to around 2.5m to 5m and I’d really like to try that out.
    2. The Hand Cursor Module

but I’ve also been looking forward to the preview for Windows 10 UWP SDK support and so I started by I spending a few minutes seeing if I could get that up and running. It’s important to note that this support requires the SR300 camera (not the F200 as confirmed on the Intel forums.) and it doesn’t yet include support for all the areas of functionality but it does have some core functionality and some facial functionality so I thought that I’d try it.

I made a blank project (what else? Smile) and then I went into references and added 2 extensions from the RealSense SDK to my project;

image

I figured that I’d want to do some immediate-mode drawing and so I also added the NuGet package win2d.uwp to my project and I made myself a little UI.

<Page
    x:Class="RealSenseTest.MainPage"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:local="using:RealSenseTest"
    xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
    xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
    mc:Ignorable="d">

    <Grid>
        <w:CanvasControl
            xmlns:w="using:Microsoft.Graphics.Canvas.UI.Xaml"
            x:Name="canvasControl"
            Draw="OnDraw">
        </w:CanvasControl>
    </Grid>
</Page>

which is just using a CanvasControl from Win2d.

At this point, I always feel there’s a decision to be made with these types of APIs in that I seem to be able to do one of two things;

    1. I can constantly invalidate my CanvasControl and then pull frames from the camera with the risk being that I do too much invalidation and might miss the odd frame here and there too.
    2. I can handle events from the camera and then redraw the UI when those occur but then the data involved in drawing (probably a big bitmap) has to be ‘transported’ from the thread that handled the event to the thread that’s doing the drawing and its lifetime preserved for long enough to draw it.

I’ve never really decided which is ‘best’ but I went with approach 1 here in the first instance – pulling frames from the camera in a constant drawing loop.

This code is very sketchy at the time of writing but in terms of simply getting colour video from the camera and onto the screen I wrote some code behind that UI above which looks like this;

 using Intel.RealSense;
  using Microsoft.Graphics.Canvas;
  using Microsoft.Graphics.Canvas.UI.Xaml;
  using System;
  using System.Threading.Tasks;
  using Windows.Foundation;
  using Windows.Graphics.Imaging;
  using Windows.UI.Popups;
  using Windows.UI.Xaml;
  using Windows.UI.Xaml.Controls;

  public sealed partial class MainPage : Page
  {
    public MainPage()
    {
      this.InitializeComponent();
      this.Loaded += OnLoaded;
    }
    async void OnLoaded(object sender, RoutedEventArgs e)
    {
      var status = Status.STATUS_NO_ERROR;

      // SenseManager is where we always start with RealSense.
      // Very glad to see that the long names of the .NET SDK
      // have gone away here.
      this.senseManager = SenseManager.CreateInstance();

      if (this.senseManager != null)
      {
        // I'm not yet sure how I switch on streams without going via this
        // SampleReader so I'm using it here to switch on colour at 1280x720
        // even though I'm then not going to use it again. In the .NET SDK
        // there's a SenseManager->EnableStream() type method.
        this.sampleReader = SampleReader.Activate(this.senseManager);
        this.sampleReader.EnableStream(StreamType.STREAM_TYPE_COLOR, 1280, 720, 0);

        // Initialise.
        status = await this.senseManager.InitAsync();
      }
      if (status != Status.STATUS_NO_ERROR)
      {
        await this.DisplayErrorAsync(status);
      }
    }
    void OnDraw(
      CanvasControl sender, CanvasDrawEventArgs args)
    {
      if (this.senseManager.AcquireFrame(false) == Status.STATUS_NO_ERROR)
      {
        var color = this.senseManager?.Sample?.Color;

        if ((color != null) && (color.SoftwareBitmap != null))
        {
          // Separated out for clarity - trying to avoid copying/converting
          // a frame that we've already got.
          if ((this.lastFrameTime == null) ||
            (this.lastFrameTime.Value != color.RelativeTime))
          {
            this.lastFrameTime = color.RelativeTime;

            // A shame that the bitmap format that RealSense is passing me
            // here isn't supported by CanvasBitmap below. Hence the conversion
            // to Rgba8 which is does support.
            using (var convertedBitmap = SoftwareBitmap.Convert(
              color.SoftwareBitmap,
              BitmapPixelFormat.Rgba8, BitmapAlphaMode.Premultiplied))
            {
              // We keep around the last frame until we get a new frame such
              // that we can draw it.
              this.currentCanvasBitmap?.Dispose();

              this.currentCanvasBitmap = CanvasBitmap.CreateFromSoftwareBitmap(
                args.DrawingSession.Device,
                convertedBitmap);
            }
          }
        }
        this.senseManager.ReleaseFrame();
      }
      // Draw the last frame that we've seen, might cause 'frozen' video
      // if frames stop arriving.
      if (this.currentCanvasBitmap != null)
      {
        args.DrawingSession.DrawImage(
          this.currentCanvasBitmap,
          new Rect(0, 0, sender.ActualWidth, sender.ActualHeight));
      }
      // Ask the control to draw itself again, re-starting the process.
      sender.Invalidate();
    }
    async Task DisplayErrorAsync(Status status)
    {
      if (status != Status.STATUS_NO_ERROR)
      {
        var dialog = new MessageDialog(status.ToString(), "RealSense Error");
        await dialog.ShowAsync();
      }
    }
    TimeSpan? lastFrameTime;
    CanvasBitmap currentCanvasBitmap;
    SenseManager senseManager;
    SampleReader sampleReader;
  }
}

and that seemed to work ok.

It’s possibly also worth noting that I don’t know how to set the Mirror Mode in this SDK just yet like I have done in the past in the .NET SDK and so this screenshot below is actually of my left hand when it looks like my right hand;

image

This feels pretty ‘easy’ in terms of using the SDK to get one of the streams like the colour stream onto the screen but I did also try out the event-based model and the code below tries to wait for a frame to arrive from the camera before handing it off to the UI thread for drawing.

  using Intel.RealSense;
  using Microsoft.Graphics.Canvas;
  using Microsoft.Graphics.Canvas.UI.Xaml;
  using System;
  using System.Threading;
  using System.Threading.Tasks;
  using Windows.Foundation;
  using Windows.Graphics.Imaging;
  using Windows.UI.Core;
  using Windows.UI.Popups;
  using Windows.UI.Xaml;
  using Windows.UI.Xaml.Controls;

  public sealed partial class MainPage : Page
  {
    public MainPage()
    {
      this.InitializeComponent();
      this.Loaded += OnLoaded;
    }
    async void OnLoaded(object sender, RoutedEventArgs e)
    {
      var status = Status.STATUS_NO_ERROR;

      // SenseManager is where we always start with RealSense.
      // Very glad to see that the long names of the .NET SDK
      // have gone away here.
      this.senseManager = SenseManager.CreateInstance();

      if (this.senseManager != null)
      {
        // I'm not yet sure how I switch on streams without going via this
        // SampleReader so I'm using it here to switch on colour at 1280x720
        // even though I'm then not going to use it again. In the .NET SDK
        // there's a SenseManager->EnableStream() type method.
        this.sampleReader = SampleReader.Activate(this.senseManager);
        this.sampleReader.EnableStream(StreamType.STREAM_TYPE_COLOR, 1280, 720, 0);

        if (status == Status.STATUS_NO_ERROR)
        {
          // Callback for when samples arrive.
          this.sampleReader.SampleArrived += OnSampleArrived;

          // Initialise.
          status = await this.senseManager.InitAsync();

          if (status == Status.STATUS_NO_ERROR)
          {
            // Start firing the callbacks as frames arrive.
            status = this.senseManager.StreamFrames();
          }
        }
      }
      if (status != Status.STATUS_NO_ERROR)
      {
        await this.DisplayErrorAsync(status);
      }
    }
    async void OnSampleArrived(object sender, SampleArrivedEventArgs e)
    {
      var color = e.Sample?.Color;

      if ((color != null) && (color.SoftwareBitmap != null))
      {
        // A shame that the bitmap format that RealSense is passing me
        // here isn't supported by CanvasBitmap below. Hence the conversion
        // to Rgba8 which is does support.
        // This is also wasteful as we convert it before seeing whether
        // we have a free slot to draw it.
        var convertedBitmap = SoftwareBitmap.Convert(
          color.SoftwareBitmap,
          BitmapPixelFormat.Rgba8, BitmapAlphaMode.Premultiplied);

        if (Interlocked.CompareExchange<SoftwareBitmap>(
          ref this.pendingBitmap, convertedBitmap, null) == null)
        {
          await this.Dispatcher.RunAsync(
            CoreDispatcherPriority.Normal,
            this.canvasControl.Invalidate);
        }
      }
      e.Sample.Dispose();
    }
    void OnDraw(
      CanvasControl sender, CanvasDrawEventArgs args)
    {
      var bitmap = Interlocked.Exchange<SoftwareBitmap>(
        ref this.pendingBitmap, null);

      if (bitmap != null)
      {
        using (var canvasBitmap = CanvasBitmap.CreateFromSoftwareBitmap(
          args.DrawingSession.Device, bitmap))
        {
          args.DrawingSession.DrawImage(
            canvasBitmap,
            new Rect(0, 0, sender.ActualWidth, sender.ActualHeight));
        }
        bitmap.Dispose();
      }
    }
    async Task DisplayErrorAsync(Status status)
    {
      if (status != Status.STATUS_NO_ERROR)
      {
        var dialog = new MessageDialog(status.ToString(), "RealSense Error");
        await dialog.ShowAsync();
      }
    }
    SoftwareBitmap pendingBitmap;
    SenseManager senseManager;
    SampleReader sampleReader;
  }

and that seemed like it worked reasonably well too and I feel that working this way and only grabbing frames when the SDK tells me to is perhaps better than constantly asking for new frames.

From there, I wanted to open this up and try out some facial data processing but I met a few stumbling blocks that, at the time of writing, I haven’t managed to get over.

Stumbling block one was that it seemed that whenever I tried to use facial data from the camera in my own UWP code, the camera would initialise and its green light would come on but then almost immediately go out again and the device would seem to disappear from Windows.

I found that trying one of the UWP samples from the SDK would behave similarly. I used the DF_FaceViewer_UWP_CS sample and I found that running this up would bring the camera into life and then the light would go off again and no frames would show up.

By contrast, the non-UWP samples for facial seemed to work reliably – e.g. the ‘Face Tracking (C++)’ sample seemed to run up reliably and so I figured that the problem that I’m hitting must be related to something in the UWP preview SDK.

I gave up on the Surface Book and tried another PC – I have a Dell Inspiron 2350 which happens to have an F200 camera embedded in it.

I should say that the ‘recommendation’ for the SR300 is a 6th generation (i.e. Skylake) processor but it’s not 100% clear to me as yet how much that is a ‘recommendation’ and how much it is a requirement.

My 2350 is a bit long in the tooth and it’s not 6th gen but it did run an F200 nicely so I thought I’d try it.

I installed the SR300 camera onto that machine, disabled the built-in F200 and then tried to make sure that I was plugging the SR300 into one of the USB3.0 slots rather than the 2.0 slots.

Again, on this system I could get the (non-UWP) SDK samples around face to work with the SR300 but I couldn’t get my own UWP code (or the UWP samples) to work and it proved to be worse than on the Surface Book because on this system I would find that calls to the SenseManager.InitAsync would always return “Item Not Available” errors which both my code and the SDK samples flag as “no camera is available”.

I then thought I’d try things on a Dell XPS 12 which, again, is a bit long in the tooth but I think it ran the F200 fine. On this one, I couldn’t get any of the samples to run with the SR300 – the green light would come on and then go off immediately.

So, at the time of writing, I’ve managed to get UWP code to display a colour stream on one of my PCs and I’ve not yet managed to get any facial code to display anything on any of my PCs.

Perhaps the disappointing one here is the Surface Book as that’s a Skylake processor so I’d expect it to ‘just work’ but, so far, I’m not having much success with it when it comes to UWP code (beyond just streaming video).

There’s a few other folks on the forums trying to navigate the SR300 too and trying to work with non Skylake processors;

I’ll keep trying and will update the post if I get things working – it’s a preview and I guess there are just some teething troubles in here somewhere but it looks like the Surface Book is going to be my best hope of getting it working. Smile

3 thoughts on “Windows 10 UWP, Intel RealSense SR300, First Steps”

  1. Pingback: Windows 10, RealSense SR300, Person Tracking – Mike Taulty

  2. Yes! Another reason added to my working list: “Why I really need a Surface Book”. Keep ’em coming Mike!

  3. Pingback: Windows 10, UWP, RealSense SR300, Faces and the Surface Pro 3 – Mike Taulty

Comments are closed.