The last 2 posts were really a bit of a catalogue of failures around;
But I kept persevering and I’ve made some progress at least around step 1 above and the main piece of progress that I’ve made is;
Work on the Surface Pro 3
For whatever reason, I’ve found that plugging the SR300 into my Surface Pro 3 with its i7-4650 CPUs seems to work out better than plugging it into my Surface Book with its much newer CPUs and I suspect it’s something to do with the 2 USB ports on the Surface Book acting as some kind of ‘hub’ which the RealSense doesn’t like.
Going back to my first post then on RealSense with UWP, I made a blank project in that post and I made a MainPage style UI that simply had a Win2D CanvasControl on it;
<w:CanvasControl
xmlns:w="using:Microsoft.Graphics.Canvas.UI.Xaml"
x:Name="canvasControl"
Draw="OnDraw">
</w:CanvasControl>
and I wrote code behind to try and display frames from the video camera (NB: code written quite quickly and quite late at night, don’t call it’s just for fun 
);
using Intel.RealSense;
using Microsoft.Graphics.Canvas;
using Microsoft.Graphics.Canvas.UI.Xaml;
using System;
using System.Threading;
using System.Threading.Tasks;
using Windows.Foundation;
using Windows.Graphics.Imaging;
using Windows.UI.Core;
using Windows.UI.Popups;
using Windows.UI.Xaml;
using Windows.UI.Xaml.Controls;
public sealed partial class MainPage : Page
{
public MainPage()
{
this.InitializeComponent();
this.Loaded += OnLoaded;
}
async void OnLoaded(object sender, RoutedEventArgs e)
{
var status = Status.STATUS_NO_ERROR;
// SenseManager is where we always start with RealSense.
// Very glad to see that the long names of the .NET SDK
// have gone away here.
this.senseManager = SenseManager.CreateInstance();
if (this.senseManager != null)
{
// I'm not yet sure how I switch on streams without going via this
// SampleReader so I'm using it here to switch on colour at 1280x720
// even though I'm then not going to use it again. In the .NET SDK
// there's a SenseManager->EnableStream() type method.
this.sampleReader = SampleReader.Activate(this.senseManager);
this.sampleReader.EnableStream(StreamType.STREAM_TYPE_COLOR, 1280, 720, 0);
// Callback for when samples arrive.
this.sampleReader.SampleArrived += this.OnSampleArrived;
// Initialise.
status = await this.senseManager.InitAsync();
if (status == Status.STATUS_NO_ERROR)
{
// Start firing the callbacks as frames arrive.
status = this.senseManager.StreamFrames();
}
}
if (status != Status.STATUS_NO_ERROR)
{
await this.DisplayErrorAsync(status);
}
}
async void OnSampleArrived(object sender, SampleArrivedEventArgs e)
{
bool invalidate = false;
var bitmap = e.Sample?.Color?.SoftwareBitmap;
if (bitmap != null)
{
// A shame that the bitmap format that RealSense is passing me
// here isn't supported by CanvasBitmap below. Hence the conversion
// to Rgba8 which is does support.
// This is also wasteful as we convert it before seeing whether
// we have a free slot to draw it.
var convertedBitmap = SoftwareBitmap.Convert(
bitmap,
BitmapPixelFormat.Rgba8, BitmapAlphaMode.Premultiplied);
if (Interlocked.CompareExchange<SoftwareBitmap>(
ref this.pendingBitmap, convertedBitmap, null) == null)
{
invalidate = true;
}
}
if (invalidate)
{
await this.InvalidateAsync();
}
// TBD
e.Sample.Dispose();
}
async Task InvalidateAsync()
{
await this.Dispatcher.RunAsync(
CoreDispatcherPriority.Normal,
this.canvasControl.Invalidate);
}
void OnDraw(
CanvasControl sender, CanvasDrawEventArgs args)
{
var bitmap = Interlocked.Exchange<SoftwareBitmap>(
ref this.pendingBitmap, null);
if (bitmap != null)
{
using (var canvasBitmap = CanvasBitmap.CreateFromSoftwareBitmap(
args.DrawingSession.Device, bitmap))
{
args.DrawingSession.DrawImage(
canvasBitmap,
new Rect(0, 0, sender.ActualWidth, sender.ActualHeight));
}
bitmap.Dispose();
}
}
async Task DisplayErrorAsync(Status status)
{
if (status != Status.STATUS_NO_ERROR)
{
var dialog = new MessageDialog(status.ToString(), "RealSense Error");
await dialog.ShowAsync();
}
}
SampleReader sampleReader;
SoftwareBitmap pendingBitmap;
SenseManager senseManager;
}
and that all works fine and video displays and it seems like it performs fine although I haven’t done anything explicit to try to tune it.
But I had this working (sporadically) in the first post referenced above. Where I struggled was with face detection but that also works fine on my Surface Pro 3.
With the exact same Win2D UI, I can change my code behind to be something like;
using Intel.RealSense;
using Intel.RealSense.Face;
using Microsoft.Graphics.Canvas;
using Microsoft.Graphics.Canvas.UI.Xaml;
using System;
using System.Collections.Generic;
using System.Threading;
using System.Threading.Tasks;
using Windows.Foundation;
using Windows.Graphics.Imaging;
using Windows.UI;
using Windows.UI.Core;
using Windows.UI.Popups;
using Windows.UI.Xaml;
using Windows.UI.Xaml.Controls;
using System.Linq;
public sealed partial class MainPage : Page
{
public MainPage()
{
this.InitializeComponent();
this.Loaded += OnLoaded;
}
async void OnLoaded(object sender, RoutedEventArgs e)
{
var status = Status.STATUS_NO_ERROR;
// SenseManager is where we always start with RealSense.
// Very glad to see that the long names of the .NET SDK
// have gone away here.
this.senseManager = SenseManager.CreateInstance();
if (this.senseManager != null)
{
// I'm not yet sure how I switch on streams without going via this
// SampleReader so I'm using it here to switch on colour at 1280x720
// even though I'm then not going to use it again. In the .NET SDK
// there's a SenseManager->EnableStream() type method.
this.faceModule = FaceModule.Activate(this.senseManager);
var config = this.faceModule.CreateActiveConfiguration();
config.Detection.IsEnabled = true;
config.Detection.MaxTrackedFaces = 1;
config.Landmarks.IsEnabled = true;
config.Landmarks.MaxTrackedFaces = 1;
// NB: I find this is crucial for getting faces detected.
// No idea why, maybe it's broken in this preview?
config.Pose.IsEnabled = false;
status = config.ApplyChanges();
if (status == Status.STATUS_NO_ERROR)
{
// Callback for when samples arrive.
this.faceModule.FrameProcessed += OnFrameProcessed;
// Initialise.
status = await this.senseManager.InitAsync();
if (status == Status.STATUS_NO_ERROR)
{
// Start firing the callbacks as frames arrive.
status = this.senseManager.StreamFrames();
}
}
}
if (status != Status.STATUS_NO_ERROR)
{
await this.DisplayErrorAsync(status);
}
}
async void OnFrameProcessed(object sender, FrameProcessedEventArgs e)
{
bool invalidate = false;
var color = this.faceModule.Sample?.Color;
if ((color != null) && (color.SoftwareBitmap != null))
{
// A shame that the bitmap format that RealSense is passing me
// here isn't supported by CanvasBitmap below. Hence the conversion
// to Rgba8 which is does support.
// This is also wasteful as we convert it before seeing whether
// we have a free slot to draw it.
var convertedBitmap = SoftwareBitmap.Convert(
color.SoftwareBitmap,
BitmapPixelFormat.Rgba8, BitmapAlphaMode.Premultiplied);
if (Interlocked.CompareExchange<SoftwareBitmap>(
ref this.pendingBitmap, convertedBitmap, null) == null)
{
invalidate = true;
}
}
if (e.Data.Faces.Count > 0)
{
float xScale = 1.0f / this.faceModule.Sample.Color.SoftwareBitmap.PixelWidth;
float yScale = 1.0f / this.faceModule.Sample.Color.SoftwareBitmap.PixelHeight;
// Scale this points to be relative 0..1 values so that we can
// multiply them back up later. We can also get the bounding
// rectangle and the average depth of the face here.
List<Point> relativePoints =
e.Data.Faces[0].Landmarks.Points.Select(
point => new Point()
{
X = point.image.X * xScale,
Y = point.image.Y * yScale
}
).ToList();
this.averageDepth = e.Data.Faces[0].Detection.FaceAverageDepth;
if (Interlocked.CompareExchange<List<Point>>(
ref this.landmarkPoints,
relativePoints,
null) == null)
{
invalidate = true;
}
}
else
{
this.averageDepth = float.NaN;
}
if (invalidate)
{
await this.InvalidateAsync();
}
}
async Task InvalidateAsync()
{
await this.Dispatcher.RunAsync(
CoreDispatcherPriority.Normal,
this.canvasControl.Invalidate);
}
void OnDraw(
CanvasControl sender, CanvasDrawEventArgs args)
{
var bitmap = Interlocked.Exchange<SoftwareBitmap>(
ref this.pendingBitmap, null);
if (bitmap != null)
{
using (var canvasBitmap = CanvasBitmap.CreateFromSoftwareBitmap(
args.DrawingSession.Device, bitmap))
{
args.DrawingSession.DrawImage(
canvasBitmap,
new Rect(0, 0, sender.ActualWidth, sender.ActualHeight));
}
bitmap.Dispose();
}
var landmarkData = Interlocked.Exchange<List<Point>>(
ref this.landmarkPoints, null);
if (landmarkData != null)
{
foreach (var landmark in landmarkData)
{
args.DrawingSession.FillCircle(
(float)(landmark.X * sender.ActualWidth),
(float)(landmark.Y * sender.ActualHeight),
CIRCLE_RADIUS,
Colors.White);
}
args.DrawingSession.DrawText(
$"Average depth {(int)this.averageDepth}mm",
new System.Numerics.Vector2(10, 10),
Colors.White);
}
}
async Task DisplayErrorAsync(Status status)
{
if (status != Status.STATUS_NO_ERROR)
{
var dialog = new MessageDialog(status.ToString(), "RealSense Error");
await dialog.ShowAsync();
}
}
const float CIRCLE_RADIUS = 5.0f;
float averageDepth;
FaceModule faceModule;
List<Point> landmarkPoints;
SoftwareBitmap pendingBitmap;
SenseManager senseManager;
}
Note that I didn’t spend a lot of time thinking about that Interlocked.* stuff being used to move data from one thread to the other so I hope that’s at least “kind of ok” in acting as a form of queue that only has 1 element in it.
Anyway, for a quick hack it seems to work quite nicely and quite smoothly and I get facial landmarks detected – quick video capture of that below;
Note that the video makes it look like the perf is bad, it’s much, much better when I’m not trying to capture the screen at the same time.
I also think that the accuracy and the range seems a lot better than with the F200 – as you’ll spot in the video, I can go to about 1m before it loses the face.
I’m really pleased that I can finally get some of this working inside of the UWP – I think it’s a great step forward for the RealSense SDK and, for me, it makes it a lot easier to include in demos around UWP.
Mike Taulty 
Pingback: Windows 10, WPF, RealSense SR300, Person Tracking–Continued – Mike Taulty