Baby Steps with Spatial Mapping in 2D and 3D Using XAML and SharpDX

NB: The usual blog disclaimer for this site applies to posts around HoloLens. I am not on the HoloLens team. I have no details on HoloLens other than what is on the public web and so what I post here is just from my own experience experimenting with pieces that are publicly available and you should always check out the official developer site for the product documentation.

I’ve been living in fear and hiding from a particular set of APIs Winking smile

Ever since HoloLens and Windows Mixed Reality first came along, I’ve been curious about the realities of the spatial mapping APIs and yet I’ve largely just treated them as a “black box”.

Naturally, that’s not to say that I haven’t benefitted from those APIs because I’ve been using them for many months in Unity via the Mixed Reality Toolkit and its support for spatial mapping and the prefab that’s fairly easy to drop into a Unity project as I first explored in this post last year;

Hitchhiking the HoloToolkit-Unity, Leg 3–Spatial Understanding (& Mapping)

That said, I’ve still had it on my “to do list” for a long time to visit these APIs a little more directly and that’s what this blog post is about.

It’s important to say that the post is mostly meant to be just “for fun” to give me a place to write down some explorations – I’m not planning to do an exhaustive write up of the APIs and what I’ll end up with by the end of this post is going to be pretty “rough”.

It’s also important to say that there are official documentation pages which detail a lot more than I’m about to write up in this post.

Spatial mapping

Spatial mapping in DirectX

but (as usual) I hadn’t really read those documents in nearly enough detail until I started to explore on my own for this post – it’s the exploration that drives the learning.

Additionally, there’s a great official sample that goes along with those documents;

Holographic Spatial Mapping Sample

but, again, I hadn’t actually seen this sample until I got well into writing this post and was trying to figure things out and I realised that I was largely trying to produce a much simpler, less functional piece of code which targeted a different type of application than the one in the sample but there are many similarities between where I ended up and that sample.

So, if you want the definitive views on these topics there are lots of links to visit.

In the meantime, I’m going to write up my own experiments here.

Choosing a Sandbox to Play In

Generally speaking, if I’m wanting to experiment with some .NET APIs then I write a console application. It seems the quickest, easiest thing to spin up.

In a Mixed Reality world, the equivalent seems to be a 2D XAML application. I find it is much quicker to Code->Deploy->Test->Debug when working on a 2D XAML application than when working on (e.g.) a 3D Unity application.

Of course, the output is then a 2D app rather than an immersive app but if you just want to test out some UWP APIs (which the spatial mapping APIs are) then that’s ok.

Specifically, in this case, I found that trying to make use of these APIs in a 2D environment seemed to actually be helpful to gaining some understanding of them as it stopped me from just looking for a quick Unity solution to various challenges and I definitely felt that I wasn’t losing anything by at least starting my journey inside of a 2D XAML application where I could quickly iterate.

Getting Going – Asking for Spatial Mapping API Access

I made a quick, blank 2D XAML UWP application in Visual Studio and made sure that its application manifest gave me the capability to use Spatial Mapping.

When I look in Visual Studio today, I don’t see this listed as an option in the UI and so I hacked the manifest file in the XML editor;

image

where uap2 translates as a namespace to;

xmlns:uap2=http://schemas.microsoft.com/appx/manifest/uap/windows10/2

in case you ever got stuck on that one. From there, I had a blank app where I could write some code to run on the Loaded event of my main XAML page.

Figuring out the SurfaceSpatialObserver

At this point, I had an idea of what I wanted to do and I was fairly sure that I needed to spin up a SpatialSurfaceObserver which does a lot of the work of trying to watch surfaces as they are discovered and refined by HoloLens.

The essence of the class would seem to be to check whether spatial mapping is supported and available via the IsSupported and RequestAccessAsync() methods.

Once support is ascertained, you define some “volumes” for the observer to observe for spatial mapping data via the SetBoundingVolume/s method and then you can interrogate that data via the GetObservedSurfaces method.

Additionally, there’s an event ObservedSurfacesChanged to tell you when the data relating to surfaces has changed because the device has added/removed or updated data.

This didn’t seem too bad and so my code for checking for support ended up looking as below;

  async void OnLoaded(object sender, RoutedEventArgs e)
    {
      bool tryInitialisation = true;

      if (Windows.Foundation.Metadata.ApiInformation.IsApiContractPresent(
          "Windows.Foundation.UniversalApiContract", 4, 0))
      {
        tryInitialisation = SpatialSurfaceObserver.IsSupported();
      }

      if (tryInitialisation)
      {
        var access = await SpatialSurfaceObserver.RequestAccessAsync();

        if (access == SpatialPerceptionAccessStatus.Allowed)
        {
          this.InitialiseSurfaceObservation();
        }
        else
        {
          tryInitialisation = false;
        }
      }
      if (!tryInitialisation)
      {
        var dialog = new MessageDialog(
          "Spatial observation is either not supported or not allowed", "Not Available");

        await dialog.ShowAsync();
      }
    }

Now, as far as I could tell the SpatialSurfaceObserver.IsSupported() method only became available in V4 of the UniversalApiContract so I’m trying to figure out whether it’s safe to call that API or not as you can see above before using it.

The next step would be perhaps to try and define volumes and so I ploughed ahead there…

Volumes, Coordinate Systems, Reference Frames, Locators – Oh My Winking smile

I wanted to keep things as simple as possible and so I chose to look at the SetBoundingVolume method which takes a single SpatialBoundingVolume and there are a number of ways of creating these based on Boxes, Frustrums and Spheres.

I figured that a sphere was a fairly understandable thing and so I went with a sphere and decided I’d use a 5m radius on my sphere hoping to determine all surface information within that radius.

However, to create a volume you first need a SpatialCoordinateSystem and the easiest way I found of getting hold of one of those was to get hold of a frame of reference.

Frames of reference can either be “attached” in the sense of being head-locked and following the device or they can be “stationary” where they don’t follow the device.

A stationary frame of reference seemed easier to think about and so I went that way but to get hold of a frame of reference at all I seemed to need to use a SpatialLocator which has a handy GetDefault() method on it and then I can use the CreateStationaryFrameOfReferenceAtCurrentLocation() method to create my frame.

So…my reasoning here is that I’m creating a frame of reference at the place where the app starts up and that it will never move during the app’s lifetime. Not perhaps the most “flexible” thing in the world, but it seemed simpler than any other options so I went with it.

With that in place, my “start-up” code looks as below;

  void InitialiseSurfaceObservation()
    {
      // We want the default locator.
      this.locator = SpatialLocator.GetDefault();

      // We try to make a frame of reference that is fixed at the current position (i.e. not
      // moving with the user).
      var frameOfReference = this.locator.CreateStationaryFrameOfReferenceAtCurrentLocation();

      this.baseCoordinateSystem = frameOfReference.CoordinateSystem;

      // Make a box which is centred at the origin (the user's startup location)
      // and is hopefully oriented to the Z axis and a certain width/height.
      var boundingVolume = SpatialBoundingVolume.FromSphere(
        this.baseCoordinateSystem,
        new SpatialBoundingSphere()
        {
          Center = new Vector3(0, 0, 0),
          Radius = SPHERE_RADIUS
        }
      );
      this.surfaceObserver = new SpatialSurfaceObserver();
      this.surfaceObserver.SetBoundingVolume(boundingVolume);
    }

Ok…I have got hold of a SpatialSurfaceObserver that’s observing one volume for me defined by a sphere. What next?

Gathering and Monitoring Surfaces Over Time

Having now got my SpatialSurfaceObserver with a defined volume, I wanted some class that took on the responsibility of grabbing any surfaces from it, putting them on a list and then managing that list as the observer fired events to flag that surfaces had been added/removed/updated.

In a real application, it’s likely that you’d need to do this in a highly performant way but I’m more interested in experimentation here than performance and so I wrote a small SurfaceChangeWatcher class which I can pass the SpatialSurfaceObserver to.

Surfaces are identified by GUID and so this watcher class maintains a simple Dictionary<Guid,SpatialSurfaceInfo>. On startup, it calls the GetObservedSurfaces method to initially populate its dictionary and then it handles the ObservedSurfacesChanged event to update its dictionary as data changes over time.

It aggregates up the changes that it sees and fires its own event to tell any interested parties about the changes.

I won’t post the whole source code for the class here but will just link to it instead. It’s not too long and it’s not too complicated.

Source SurfaceChangeWatcher.cs.

Checking for Surface Data

At this point, I’ve enough code to fire up the debugger and debug my 2D app on a HoloLens or an emulator and see if I can get some spatial mapping data into my code.

It’s worth remembering that the HoloLens emulator is good for debugging spatial mapping as by default the emulator places itself into a “default room” and it can switch to a number of other rooms provided with the SDK and also to custom rooms that have been recorded from a HoloLens.

So, debugging on the emulator I can see that in the first instances here I see 22 loaded surfaces coming back from the SpatialSurfaceObserver;

image

and you can see the ID for my first surface and the UpdateTime that it’s associated with.

I also notice that very early on in the application I see the ObservedSurfacesChanged event fire and my code in SurfaceChangeWatcher simply calls back into the LoadSurfaces method shown in the screenshot above which then attempts to figure out which surfaces have been added/removed or updated since they were last queried.

So, getting hold of the surfaces within a volume and responding to their changes as they evolve doesn’t seem too onerous.

But, how to get the actual polygonal mesh data itself?

Getting Mesh Data

Once you have hold of a SpatialSurfaceInfo, you can attempt to get hold of the SpatialSurfaceMesh which it represents via the TryComputeLatestMeshAsync method.

This method wants a “triangle density” in terms of how many triangles it should attempt to bring back per cubic metre. If you’ve used the Unity prefab then you’ll have seen this parameter before and in my code here I chose a value of 100 and stuck with it.

The method is also asynchronous and so you can’t just demand the mesh in realtime but it’s a fairly simple call and here’s a screenshot of me back in the debugger having made that call to get some data;

image

That screenshot shows that I’ve got a SpatialSurfaceMesh and it contains 205 vertices in the R16G16B16A16IntNormalized format and that there are 831 triangle vertices in an R16Uint format and it also gives me the Id and the UpdateTime of the SpatialSurfaceInfo.

It’s also worth noting the VertexPositionScale which needs to be applied to the vertices to reconstruct them.

Rendering Mesh Data

Now, at this point I felt that I had learned a few things about how to get hold of spatial mapping meshes but I thought that it wasn’t really “enough” if I didn’t make at least some attempt to render the meshes produced.

I thought about a few options around how I might do that given that I’m running code inside of a 2D XAML application.

I wondered whether I might somehow flatten the mesh and draw it with a XAML Canvas but that seemed unlikely and I suspected that the best road to go down would be to keep the data in the format that it was already being provided in and try and hand it over to DirectX for rendering.

That led me to wonder whether something from Win2D might be able to draw it for me but Win2D stays true to its name and doesn’t (as far as I know) get into the business of wrapping up Direct3D APIs.

So…I figured that I’d need to bite the bullet and see if I could bring this into my 2D app via the XAML SwapChainPanel integration element with some rendering provided by SharpDX.

It’s worth saying that I’ve hardly ever used SwapChainPanel and I’ve never used SharpDX before so I figured that putting them together with this mesh data might be “fun” Winking smile

A UWP SharpDX SwapChainPanel Sample

In order to try and achieve that, I went on a bit of a search to try and see if I could find a basic sample which illustrated how to integrate SharpDX code inside of a XAML application rendering to a SwapChainPanel.

It took me a little while to find that sample as quite a few of the SharpDX samples seem to be out of date these days and I asked around on Twitter before finding this great sample which uses SharpDX and SwapChainPanel to render a triangle inside of a UWP XAML app;

https://github.com/minhcly/UWP3DTest

That let me drop a few SharpDX packages into my project;

image

and the sample was really useful in that it enabled me to drop a SwapChainPanel into my XAML UI app and, using code that I lifted and reworked out of the sample, I could get that same triangle to render inside of my 2D XAML application.

That gave me a little hope that I might be able to get the mesh data rendered inside of my application too.

Building a SharpDX Renderer (or trying to!)

I wrote a class SwapChainPanelRenderer (source) which essentially takes the SwapChainPanel and my SurfaceChangeWatcher class and it puts them together in order to retrieve/monitor spatial meshes as they are produced by the SpatialSurfaceObserver.

The essence of that class is that it goes through a few steps;

  1. It Initialises D3D via SharpDX largely following the pattern from the sample I found.
  2. It creates a very simple vertex shader and pixel shader much like the sample does although I ended up tweaking them a little.
  3. Whenever a new SpatialSurfaceInfo is provided by the SurfaceChangeWatcher the renderer attempts asks the system to compute the mesh for it and creates a number of data structures from that mesh;
    1. A vertex buffer to match the format provided by the mesh
    2. An index buffer to match the format provided by the mesh
    3. A constant buffer with details of how to transform the vertices provided by the mesh
  4. Whenever the renderer is asked to render, it loads up the right vertex/index/constant buffers for each of the meshes that it knows about and asks the system to render them passing through a few transformation pieces to the vertex shader.

It’s perhaps worth noting a couple of things around how that code works – the first would be;

  • In order to get hold of the actual vertex data, this code relies on using unsafe C# code and IBufferByteAccess in order to be able to grab the real data buffers rather than copying it.

The second point that might be worth mentioning is that I spent quite a bit of time trying to see if I could get the mesh rendering right.

  • I’m not 100% there at the time of writing but what I have managed to get working has been done by consulting back with the official C++ sample which has a more complex pipeline but I specifically consulted it around how to make use of the SpatialSurfaceMesh.VertexPositionScale property and I tried to make my code line up with the sample code around that in as much as possible.

I must admit that I spent a bit of time staring at my code and trying to compare to the sample code as a way of trying to figure out if I could improve the way mine was seeming to render and I think I can easily spend more time on it to make it work better.

The last point I’d make is that there’s nothing in the code at the time of writing which attempts to align the HoloLens position, orientation and view with what’s being shown inside of the 2D app. What that means is;

  • The 2D app starts at a position (0,0.5,0) so half a metre above where the HoloLens is in the world.
  • The 2D app doesn’t know the orientation of the user so could be pointing in the wrong direction with respect to the mesh.

This can make the app a little “disorientating” unless you are familiar with what it’s doing Smile

Trying it Out

At the time of writing, I’ve mostly been trying this code out on the emulator but I have also experimented with it on HoloLens.

Here’s a screenshot of the official sample 3D app with its fancier shader running on the emulator where I’m using the “living room” room;

image

and here’s my 2D XAML app running in a window but, hopefully, rendering a similar thing albeit in wireframe;

image

and, seemingly, there’s something of a mirroring going on in there as well which I still need to dig into!

Wrapping Up & The Source

As I said at the start of the post, this one was very much just “for fun” but I thought I’d write it down so that I can remember it and maybe some pieces of it might be useful to someone else in the future.

If you want the source, it’s all over here on github so feel free to take it, play with it and feel very free to improve it Smile.

Windows 10 Creators Update, UWP Apps–An Experiment with Streaming Installations

I’ve been slowly trying to catch up with what happened at //build via the videos on Channel9 focusing mainly on the topics around Windows 10 UWP and Mixed Reality with a sprinkling of what’s going on in .NET, C# and some of the pieces around Cognitive Services, identity and so on.

Given that //build was a 3 day conference, it generates a video wall worth of content which then takes a very long time to try and catch up with and so I expect I’ll still be doing some of this catching up over the coming weeks and months but I’m slowly making progress.

One of the many sessions that caught my eye was this one on changes to the packaging of UWP apps;

image

which talks about some of the changes that have been made in the Creators Update around breaking up UWP packages into pieces such that they can be installed more intelligently, dynamically and flexibly than perhaps they can today.

It’s well worth watching the session but if I had to summarise it I’d say that it covers;

  • How packages have always been able to be broken into pieces containing different types of resources using the “Modern Resource Technology” (MRT) such that (e.g.) only the resources that are relevant to the user’s language or scale or DirectX level are downloaded for the app.
  • How packages in Creators Update can be broken apart into “Content Groups” and partitioned into those which are required for the application to start up and those which can be deferred and downloaded from the Store at a later point in order to improve the user’s experience. There are APIs to support the developer being aware of which parts of the package are present on the system, to monitor and control download priority, etc.
  • How optional packages can be authored for Creators Update such that one or more apps can optionally make use of a separate package from the Store which can install content (and (native) code) into their application.

As you might expect, there’s lots of additional levels of detail here so if you’re interested in these bits then some links below will provide some of that detail;

and there’s more generally on the App Installer Blog and additional interesting pieces in that //build session around possible future developments and how Microsoft Office ™ is making use of these pieces in order to be deliverable from the Windows Store.

The idea of ‘streaming installations’ seemed immediately applicable to me but I need to spend some more time thinking about optional packages because I was struck by some of the similarities between them and app extensions (more here) and I haven’t quite figured out the boundaries there beyond the ability of an optional package to deliver additional code (native) to an application which extensions can’t do as far as I’m aware.

Having got my head around streaming installations, I wanted to experiment with them and that’s where the rest of this post is going.

I needed an app to play with and so I went and dug one out of the cupboard…

A Simple Pictures App

I wrote this tiny little “app” around the time of the UK “Future Decoded” show in late 2016 in order to demonstrate app extensions.

The essential idea was that I have this app which displays pictures from a group;

image

and there is 1 set of pictures built in – some film posters but I have two more sets of pictures under groupings of ‘Albums’ and ‘BoxSets’.

The original app used app extensions and so the ‘Albums’ and ‘BoxSets’ collections lived in another project providing an ‘extension’ to the content such that when the extension was installed on the system all of the 3 sets of content are loaded and the app looks as below;

image

This was pretty easy to put together using app extensions and it’s similar to what I wrote up in this blog post about app extensions where I used extensions and App Services together to build out a similarly extensible app.

So, having this code kicking around it seemed like an obvious simple project that I could use to try out streaming installations on Creators Update.

Defining Content Groups

Firstly, I brought all 3 of my content folders into the one project (i.e. Posters, Albums, BoxSets) as below;

image

and then I set about authoring a SourceAppxContentGroupMap.xml file as covered in this MSDN article;

Create and convert a source content group map

and I learned a couple of things there which were to firstly make sure that you set the right build action for that XML file;

image

and secondly to make sure that you’re running the right version of makeappx if you expect it to have the new /convertCGM option Smile That right version on my system would come from;

image

at the time of writing although I ultimately let Visual Studio build the content group map and only used makeappx as part of experimenting.

My content group map looked as below – I essentially just define that everything for the application is required apart from the two folders named Albums and BoxSets which are not required to start the application and so can be downloaded post-installation by the system as it sees fit;

<?xml version="1.0" encoding="utf-8"?>
<ContentGroupMap xmlns="http://schemas.microsoft.com/appx/2016/sourcecontentgroupmap" xmlns:s="http://schemas.microsoft.com/appx/2016/sourcecontentgroupmap" >
  <Required>
    <ContentGroup Name="Required">
      <File Name="*"/>
      <File Name="WinMetadata\*"/>
      <File Name="Properties\*"/>
      <File Name="Assets\*"/>
      <File Name="Posters\**"/>
    </ContentGroup>
  </Required>
  <Automatic>
    <ContentGroup Name="BoxSets">
      <File Name="BoxSets\**"/>
    </ContentGroup>
    <ContentGroup Name="Albums">
      <File Name="Albums\**"/>
    </ContentGroup>
  </Automatic>
</ContentGroupMap>

This file is then an input to produce the actual AppxContentGroupMap.xml file and I just used the Visual Studio menu to generate it as per the docs;

image

and after a couple of initial gremlins caused by me, that seemed to work out fine.

Writing Code to Load Content Groups

If the application is going to be installed “in pieces” then my code is going to have to adapt such that it can dynamically load up folders of pictures as they appear post-install.

Because I’d previously written the code to support a similar scenario using app extensions and because the code is very simple it wasn’t particularly difficult to do this. I have a function which attempts to figure out whether the content groups for the Albums and BoxSets have been installed and, if so, it adds them to what the application is displaying. This snippet of code covers it;

    async Task AddStreamedPictureSourcesAsync()
    {
      // Handle any streamed packages that are already installed.
      var groups = await Package.Current.GetContentGroupsAsync();

      // TBD - unsure exactly of the state to check for here in order
      // to be sure that the content group is present.
      foreach (var group in groups.Where(
        g => !g.IsRequired && g.State == PackageContentGroupState.Staged))
      {
        await this.AddPictureSourceAsync(group.Name, group.Name);
      }

      // Now set up handlers to wait for any others to arrive
      this.catalog = PackageCatalog.OpenForCurrentPackage();
      this.catalog.PackageInstalling += OnPackageInstalling;
    }
    async void OnPackageInstalling(
      PackageCatalog sender,
      PackageInstallingEventArgs args)
    {
      if (args.IsComplete)
      {
        await this.Dispatcher.RunAsync(CoreDispatcherPriority.Normal,
          async () =>
          {
            // Warning - untested at time of writing, I need to check
            // whether FullName is the right property here because 
            // I really want the *content group name*.
            await this.AddPictureSourceAsync(args.Package.Id.FullName,
              args.Package.Id.FullName);
          }
        );
      }
    }
    PackageCatalog catalog;

 

and this is making use of APIs that come from either SDK 14393 or 15063 on the PackageCatalog class in order to check what content groups are available and if I find that my Albums/BoxSets groups are available then I have code which goes and adds all the pictures from those folders to the collections which live behind the UI.

The code is also attempting to handle the PackageInstalling event to see if I can dynamically respond to the 2 non-required packages being added while the application is running and note the comment in there about me not actually having seen that code run just yet and I’ll come back to why that is in just one second as it’s the wrong code Smile

Testing…

How to try this out?

In the //build session, there’s a few options listed around how you can test/debug a streaming install without actually putting your application into the Store. One method makes use of the PackageCatalog APIs to programmatically change the installation status of the content groups, another makes use of the Windows Device Portal (although I’m unsure as to whether this one is implemented yet) and there’s an option around using the regular PowerShell add-appxpackage command.

Testing via PowerShell

I thought I’d try the PowerShell option first and so I made a .APPX package for my application via the Store menu in Visual Studio;

image

and then made sure that I wasn’t making an APPX bundle;

image

and then I got hold of the temporary certificate that this generates and made it trusted on my system before going to install the .APPX file via PowerShell;

image

and so the key part here is the new –RequiredContentGroupOnly parameter to the Add-AppxPackage command. With that command executed, I can see that the app only has access to the Posters collection of images from its required content group and so that all seems good;

image

I also found it interesting to go and visit the actual folder on disk where the application is installed and to see what the Albums/BoxSets folders representing the ‘automatic’ content groups look like.

The first thing to say is that those folders do exist and here’s what the content looks like  at this point in the process;

image

so there are “marker files” present in the folders and so (as advised in the //build session) code would have to be careful not to confuse the presence of the folders/files with the content group’s installation status.

I’d hoped to then be able to use the add-appxpackage command again to add the other two content groups (Albums/BoxSets) while the application was running but when I tried to execute that, I saw;

image

Now, this was “very interesting” Smile in that I was reading the section of this page titled “Sideloaded Stream-able App” and it suggested that;

With the debugger attached, you can install the automatic content groups by:

Add-AppxPackage –Path C:\myapp.appx

Which is the exact same command but without the flag (what happens is that the platform will see that the app is already installed and will only stage the files that are missing).

So I attached my debugger to the running app and ran the command again and, sure enough, I could see that the debugger hit a first-chance exception in that piece of untested code that I’d listed earlier;

image

and so, sure enough, my code was being called here as the package started to install but that code wasn’t working because it was confusing the content group name with the application’s full package name.

That didn’t surprise me too much, it had been a bit of a ‘wild guess’ that I might use the PackageCatalog.PackageInstalling event in this way and I was clearly wrong so I went and reworked that code to make use of the far more sensible sounding PackageContentGroupStaging event as below;

 async Task AddStreamedPictureSourcesAsync()
    {
      // Handle any streamed packages that are already installed.
      var groups = await Package.Current.GetContentGroupsAsync();

      // TBD - unsure exactly of the state to check for here in order
      // to be sure that the content group is present.
      foreach (var group in groups.Where(
        g => !g.IsRequired && g.State == PackageContentGroupState.Staged))
      {
        await this.AddPictureSourceAsync(group.Name, group.Name);
      }

      // Now set up handlers to wait for any others to arrive
      this.catalog = PackageCatalog.OpenForCurrentPackage();
      this.catalog.PackageInstalling += OnPackageInstalling;
      this.catalog.PackageContentGroupStaging += OnContentGroupStaging;
    }

    async void OnContentGroupStaging(
      PackageCatalog sender, PackageContentGroupStagingEventArgs args)
    {
      if (args.IsComplete)
      {
          await this.Dispatcher.RunAsync(CoreDispatcherPriority.Normal,
            async () =>
            {
              await this.AddPictureSourceAsync(
                args.ContentGroupName,
                args.ContentGroupName);
            }
          );
      }
    }

    async void OnPackageInstalling(
      PackageCatalog sender,
      PackageInstallingEventArgs args)
    {
      // TODO: Remove this handler, don't think it's useful but leaving
      // it for the moment for debugging.
      Debugger.Break();
    }

This looked like it was far more likely to work but what I found was;

  1. The Add-AppxPackage command would still fail when I tried to add the non-required content groups to the already running app.
  2. From the debugger, I could see that the PackageInstalling event was still firing but the PackageContentGroupStaging event wasn’t. I suspect that the Add-AppxPackage command is quitting out between those 2 stages and so the first event fires and the second doesn’t.

This means that I haven’t been able to use this method just yet to test what happens when the app is running and the additional content groups are installed.

The best that I could find to do here was to install the required content group using the –RequiredContentGroupOnly and then, with the application running, I could install the other groups using the –ForceApplicationShutdown option and, sure enough, the app would go away and come back with all 3 of my content groups rather than just the required one;

image

and so that shows that things are working across app executions but it doesn’t test out how they work when the application is up and running which might well be the case if the user gets the app from Store, runs it and then additional packages show up over the first few minutes of the user’s session with the app.

Testing via the Streaming Install Debugging App

At this point, I went back to this blog post and tried out the steps under the heading of “Using the Streaming Install Debugging App”. This involves going off to download this app from github which then uses the APIs to manipulate the installation status of the content groups within my app.

I uninstalled my app from the system and then reinstalled it by hitting F5 in Visual Studio and then I ran up the debugging app and, sure enough, it showed me the details of my app;

image

and so I can now use this UI to change the status of my two content groups BoxSets and Albums to be ‘not staged’;

image

and then run up my app alongside this one and it correctly just shows the ‘Film Posters’ content;

image

and if I dynamically now switch a content group’s state to Staged then my app updates;

image

and I can repeat that process with the Albums content group;

image

and so that all seems to be working nicely Smile

Wrapping Up

I really like these new sorts of capabilities coming to UWP packaging and the APIs here seem to make it pretty easy to work with although, clearly, you’d need to give quite a lot of early-stage thought to which pieces of your application’s content should be packaged into which content groups.

I put the code that I was playing with here onto github if you’re interested in picking up this (admittedly very simple) sample.

Using OCR to Read an IP Address in a Holographic UWP App

NB: The usual blog disclaimer for this site applies to posts around HoloLens. I am not on the HoloLens team. I have no details on HoloLens other than what is on the public web and so what I post here is just from my own experience experimenting with pieces that are publicly available and you should always check out the official developer site for the product documentation.

Just a short post. I’ve been playing with a situation recently where I needed to get an IP address into a holographic app. There’s lots of different ways that you might do this depending on how frequently the IP address might change;

  • Hard-code it into the app.
  • Put it into a config file (perhaps loaded from some other server).
  • Type it into the app’s UI although I think it’s fair to say that typing IP addresses on HoloLens isn’t so much fun.
  • Magically download it via some sort of bluetooth service
  • Speak it to the application through the microphone and UWP speech recognition
  • Put the IP address into a QR code and have the app scan it through the camera

and I’m sure there’s many more but today I thought I’d experiment with what seemed like a fairly natural idea – when I want to give another person an IP address, I usually write it onto a piece of paper and hand it to them or pin it up on the wall.

So, why not with the device?

Because I already have some code which runs on HoloLens and scans for QR codes (see this blog post and accompanying github) it felt like it would be very little effort to change the QR code scanning that happens in that blog post into some “IP Address” OCR recognition and see how well or badly that works out.

Here’s an example of how well/badly it worked out – I think it works quite well but it’s fair to say that I haven’t managed to get it to work with handwriting although that doesn’t surprise me as I’ve never found the OCR engine in UWP to process handwriting and maybe it’s not meant to so I’m not being critical of it when I say that.

Note – it’s “a bit tricky” to record an app on the HoloLens that is actively taking over the webcam so I’m having to show this with static images;

20170328_144256_HoloLens

20170328_144320_HoloLens

but it feels like a scan of an IP address like that can be done in about 1-2s once the camera has a view of it although, naturally, it’s possible for the camera to get a partial view such as 2.168.0.1 for the above address so a re-scan might be necessary in some circumstances.

In terms of the code here, I simply did some minimal changes to what I already had around QR code processing in that I removed the ZXing library from my project and then changed the public interface so that it looks as it does in the code below and clearly I should do something about renaming the namespace here but this call says “Hey, find me an IP address from the first camera on the system or timeout after 30s”;

MediaFrameQrProcessing.Wrappers.IPAddressScanner.ScanFirstCameraForIPAddress(
        result =>
        {
          UnityEngine.WSA.Application.InvokeOnAppThread(() =>
          {
            // result here is a System.Net.IPAddress...
            this.textMesh.text = result?.ToString() ?? "not found";
          }, 
          false);
        },
        TimeSpan.FromSeconds(30));

and it gives me back an IPAddress instance. In my underlying library, I replaced the QR code frame scanner with a (basic!) IP address frame scanner;

namespace MediaFrameQrProcessing.Processors
{
  using MediaFrameQrProcessing.VideoDeviceFinders;
  using System.Runtime.InteropServices.WindowsRuntime;
  using System.Threading.Tasks;
  using Windows.Devices.Enumeration;
  using Windows.Media.Capture;
  using Windows.Media.Capture.Frames;
  using Windows.Media.Ocr;
  using System;
  using System.Text.RegularExpressions;
  using System.Net;

  public class IPAddressFrameProcessor : MediaCaptureFrameProcessor
  {
    public IPAddress Result { get; private set; }

    public IPAddressFrameProcessor(
      MediaFrameSourceFinder mediaFrameSourceFinder, 
      DeviceInformation videoDeviceInformation, 
      string mediaEncodingSubtype, 
      MediaCaptureMemoryPreference memoryPreference = MediaCaptureMemoryPreference.Cpu) 

      : base(
          mediaFrameSourceFinder, 
          videoDeviceInformation, 
          mediaEncodingSubtype, 
          memoryPreference)
    {
    }
    protected override async Task<bool> ProcessFrameAsync(MediaFrameReference frameReference)
    {
      bool done = false;

      // doc here https://msdn.microsoft.com/en-us/library/windows/apps/xaml/windows.media.capture.frames.videomediaframe.aspx
      // says to dispose this softwarebitmap if you access it.
      using (var bitmap = frameReference.VideoMediaFrame.SoftwareBitmap)
      {
        try
        {
          if (this.ocrEngine == null)
          {
            this.ocrEngine = OcrEngine.TryCreateFromUserProfileLanguages();
            this.regex = new Regex(IP_ADDRESS_PATTERN);
          }
          var results = await this.ocrEngine.RecognizeAsync(bitmap);

          if (results != null)
          {
            var matchingResults = this.regex.Matches(results.Text);

            for (int i = 0; !done && (i < matchingResults.Count); i++)
            {
              IPAddress parsedAddress;

              done = IPAddress.TryParse(matchingResults[i].Value, out parsedAddress);

              if (done)
              {
                this.Result = parsedAddress;
              }
            }
          }
        }
        catch
        {
        }
      }
      return (done);
    }
    Regex regex;
    OcrEngine ocrEngine;

    // Taken from S.O. http://stackoverflow.com/questions/106179/regular-expression-to-match-dns-hostname-or-ip-address
    const string IP_ADDRESS_PATTERN =
      @"(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)){3}";
  }
}

and that all seems to work reasonably well.

The code for this (the library and the 2D XAML test app) is here if anyone wants to play with it.