Exploring the Mixed Reality Design Labs–Introduction and Experiment #1

NB: The usual blog disclaimer for this site applies to posts around HoloLens. I am not on the HoloLens team. I have no details on HoloLens other than what is on the public web and so what I post here is just from my own experience experimenting with pieces that are publicly available and you should always check out the official developer site for the product documentation.

On a recent project, I was trying to avoid having to write a simplified version of the control that the Holograms app offers on selected holograms for moving, rotating and scaling. This control here;

Image result for holograms app rotate scale

It’s essentially doing a job analogous to the resize handles that you get on a Window in the 2D world and I’d imagine that it took quite a bit of work to write, debug and get right.

I didn’t have time to try and reproduce it in full fidelity and so I really wanted to just re-use it because I knew that to write even a limited version would take me a few hours and, even then, it’d not be as good as the original.

I didn’t find such a control and so, between us, myself and Pete wrote one which ended up looking a little like this;

image

and it was pretty functional but it wasn’t as good as the one in the Holograms app and it took a bunch of time to write and debug.

However, just after it was completed someone inside of the company pointed me to a place where I could have taken a better control straight off-the-shelf and that’s what I wanted to write about in this post.

The Mixed Reality Design Labs

This came completely out-of-the-blue to me in that I hadn’t seen the Mixed Reality Design Labs before – here a quick screenshot from their home page on github;

image

I’m going to shorten “Mixed Reality Design Labs” to MRDL (rhymes with TURTLE) in this post because, otherwise, it gets too long.

Straight away I could see that on the home page are listed a number of very interesting sounding controls including these two, the Holobar and the Bounding Box.

 image

and my first thought was as to whether those controls just draw the visuals or do they also implement the behaviour of scale/rotate/translate but, either way, they’d be saving me a tonne of work Smile

I noticed also that this repo is linked from the mixed reality development documentation over here;

image

but it doesn’t look like it’s a simple 1:1 between the MRDL and this document page because (e.g.) the guidance here around ‘Text in Unity’ actually points into pieces from the HoloToolkit-Unity rather than controls within the MRDL.

By contrast, I think the other 4 sections listed here do point into the MRDL so it’s nearly a 1:1 Smile

Including the MRDL

I wanted to try some of these things out and so I spun up a blank Unity project, added the HoloToolkit-Unity and set it up for mixed reality development (i.e. set the project, scene, capability settings using the toolkit) before going to look at how I could start to make use of these pieces.

As there didn’t seem to be any Unity packages or similar in the repo, I figured it was just a matter of copying the MRDesignLabs_Unity folder into my project and I went with that and after Unity had seen those files I got a nice helpful dialog around the HoloLens MDL2 Symbols font;

image

and so I went with downloading and installing that but then I realised that I was perhaps meant to import that .ttf file as an asset into Unity here and then (using the helpful button on the dialog) assign it as the font for buttons as per below;

capture2

That was easy enough but I then dropped off down a bit of a black hole as there seems to be quite a lot in this project;

image

and I find this quite regularly with Unity – when someone delivers you a bunch of bits, how do you figure out which pieces they intended you to use directly and which bits are just supporting infrastructure for what they have built – i.e. there seems to be a lack of [private/public/internal] type visibility which would guide the user as to what was going on.

Rather than get bewildered though, I figured I’d go back to trying to solve my original problem – selecting, rotating, scaling, translating a 3D object…

Manipulations, Bounding Boxes, App Bars

I must admit, I got stuck pretty quickly.

Like the HoloToolkit, there are a lot of pieces here and it’s not easy to figure out how they all sit together and so I also opened up the Examples project and had a bit of a look at the scene called ManipulationGizmo_Examples which gave me more than enough clues to feel confident to play around in my own blank scene.

I started off by dragging out the HoloLens prefab which brings with it a tonne of pieces;

image

I did have a little dig around into where these pieces were coming from and what they might do but I’m not going to attempt to write that up here. What I think I noticed is that these pieces look to be largely independent rather than taking dependencies on the HoloToolkit-Unity but I may be wrong there as I haven’t spent a lot of time on it just yet.

Where I did focus in was on the Manipulation Manager which looked like the piece that I was going to need in order to get my scale/rotate/translate functionality working and it’s a pretty simple script which simply instantiates the ‘Bounding Box Prefab’ and the ‘App Bar Prefab’ and keeps them around as singletons for later use which seems to imply that only one object would have the bounding box wrapped around it at a time (i.e. a single selection model) which seems reasonable to me. I also noticed in that script that it takes a dependency on the Singleton<T> class from the HoloToolkit-Unity so that starts to go against my earlier thoughts around dependencies;

image

To try this out, I added a quick cube into my scene;

image

and (from the example scene) I’d figured that I needed to add a Bounding Box Target component to my cube in order to apply the bounding box behaviour;

image

I like all the options here but I left them as their default values for now and I ran up the code on my HoloLens using the Unity’s Holographic Remoting feature and, sure enough, when I can tap on the cube to select it I get the bounding box and I can then use manipulations in order to translate, rotate and scale matching the functionality that I asked for in the editor.

This screenshot comes from Unity displaying what’s happening remotely on the HoloLens and you can see the AppBar with its Done/Remove buttons and the grab handles which are facilitating scale/rotate/translate. What’s not visible is that the control has animations and highlights which make for a quality, production feel rather than a “Mike just hacked this together in a hurry” feel Smile

image

I’m going to try and find some time to explore more of the MRDL and I’ll write some supplementary posts as/when those work out but if you’re building for Windows Mixed Reality then I think you should have an eye on this project as it has the potential to save you a tonne of work Smile

Windows 10 Creators Update, UWP Apps–An Experiment with Streaming Installations

I’ve been slowly trying to catch up with what happened at //build via the videos on Channel9 focusing mainly on the topics around Windows 10 UWP and Mixed Reality with a sprinkling of what’s going on in .NET, C# and some of the pieces around Cognitive Services, identity and so on.

Given that //build was a 3 day conference, it generates a video wall worth of content which then takes a very long time to try and catch up with and so I expect I’ll still be doing some of this catching up over the coming weeks and months but I’m slowly making progress.

One of the many sessions that caught my eye was this one on changes to the packaging of UWP apps;

image

which talks about some of the changes that have been made in the Creators Update around breaking up UWP packages into pieces such that they can be installed more intelligently, dynamically and flexibly than perhaps they can today.

It’s well worth watching the session but if I had to summarise it I’d say that it covers;

  • How packages have always been able to be broken into pieces containing different types of resources using the “Modern Resource Technology” (MRT) such that (e.g.) only the resources that are relevant to the user’s language or scale or DirectX level are downloaded for the app.
  • How packages in Creators Update can be broken apart into “Content Groups” and partitioned into those which are required for the application to start up and those which can be deferred and downloaded from the Store at a later point in order to improve the user’s experience. There are APIs to support the developer being aware of which parts of the package are present on the system, to monitor and control download priority, etc.
  • How optional packages can be authored for Creators Update such that one or more apps can optionally make use of a separate package from the Store which can install content (and (native) code) into their application.

As you might expect, there’s lots of additional levels of detail here so if you’re interested in these bits then some links below will provide some of that detail;

and there’s more generally on the App Installer Blog and additional interesting pieces in that //build session around possible future developments and how Microsoft Office ™ is making use of these pieces in order to be deliverable from the Windows Store.

The idea of ‘streaming installations’ seemed immediately applicable to me but I need to spend some more time thinking about optional packages because I was struck by some of the similarities between them and app extensions (more here) and I haven’t quite figured out the boundaries there beyond the ability of an optional package to deliver additional code (native) to an application which extensions can’t do as far as I’m aware.

Having got my head around streaming installations, I wanted to experiment with them and that’s where the rest of this post is going.

I needed an app to play with and so I went and dug one out of the cupboard…

A Simple Pictures App

I wrote this tiny little “app” around the time of the UK “Future Decoded” show in late 2016 in order to demonstrate app extensions.

The essential idea was that I have this app which displays pictures from a group;

image

and there is 1 set of pictures built in – some film posters but I have two more sets of pictures under groupings of ‘Albums’ and ‘BoxSets’.

The original app used app extensions and so the ‘Albums’ and ‘BoxSets’ collections lived in another project providing an ‘extension’ to the content such that when the extension was installed on the system all of the 3 sets of content are loaded and the app looks as below;

image

This was pretty easy to put together using app extensions and it’s similar to what I wrote up in this blog post about app extensions where I used extensions and App Services together to build out a similarly extensible app.

So, having this code kicking around it seemed like an obvious simple project that I could use to try out streaming installations on Creators Update.

Defining Content Groups

Firstly, I brought all 3 of my content folders into the one project (i.e. Posters, Albums, BoxSets) as below;

image

and then I set about authoring a SourceAppxContentGroupMap.xml file as covered in this MSDN article;

Create and convert a source content group map

and I learned a couple of things there which were to firstly make sure that you set the right build action for that XML file;

image

and secondly to make sure that you’re running the right version of makeappx if you expect it to have the new /convertCGM option Smile That right version on my system would come from;

image

at the time of writing although I ultimately let Visual Studio build the content group map and only used makeappx as part of experimenting.

My content group map looked as below – I essentially just define that everything for the application is required apart from the two folders named Albums and BoxSets which are not required to start the application and so can be downloaded post-installation by the system as it sees fit;

<?xml version="1.0" encoding="utf-8"?>
<ContentGroupMap xmlns="http://schemas.microsoft.com/appx/2016/sourcecontentgroupmap" xmlns:s="http://schemas.microsoft.com/appx/2016/sourcecontentgroupmap" >
  <Required>
    <ContentGroup Name="Required">
      <File Name="*"/>
      <File Name="WinMetadata\*"/>
      <File Name="Properties\*"/>
      <File Name="Assets\*"/>
      <File Name="Posters\**"/>
    </ContentGroup>
  </Required>
  <Automatic>
    <ContentGroup Name="BoxSets">
      <File Name="BoxSets\**"/>
    </ContentGroup>
    <ContentGroup Name="Albums">
      <File Name="Albums\**"/>
    </ContentGroup>
  </Automatic>
</ContentGroupMap>

This file is then an input to produce the actual AppxContentGroupMap.xml file and I just used the Visual Studio menu to generate it as per the docs;

image

and after a couple of initial gremlins caused by me, that seemed to work out fine.

Writing Code to Load Content Groups

If the application is going to be installed “in pieces” then my code is going to have to adapt such that it can dynamically load up folders of pictures as they appear post-install.

Because I’d previously written the code to support a similar scenario using app extensions and because the code is very simple it wasn’t particularly difficult to do this. I have a function which attempts to figure out whether the content groups for the Albums and BoxSets have been installed and, if so, it adds them to what the application is displaying. This snippet of code covers it;

    async Task AddStreamedPictureSourcesAsync()
    {
      // Handle any streamed packages that are already installed.
      var groups = await Package.Current.GetContentGroupsAsync();

      // TBD - unsure exactly of the state to check for here in order
      // to be sure that the content group is present.
      foreach (var group in groups.Where(
        g => !g.IsRequired && g.State == PackageContentGroupState.Staged))
      {
        await this.AddPictureSourceAsync(group.Name, group.Name);
      }

      // Now set up handlers to wait for any others to arrive
      this.catalog = PackageCatalog.OpenForCurrentPackage();
      this.catalog.PackageInstalling += OnPackageInstalling;
    }
    async void OnPackageInstalling(
      PackageCatalog sender,
      PackageInstallingEventArgs args)
    {
      if (args.IsComplete)
      {
        await this.Dispatcher.RunAsync(CoreDispatcherPriority.Normal,
          async () =>
          {
            // Warning - untested at time of writing, I need to check
            // whether FullName is the right property here because 
            // I really want the *content group name*.
            await this.AddPictureSourceAsync(args.Package.Id.FullName,
              args.Package.Id.FullName);
          }
        );
      }
    }
    PackageCatalog catalog;

 

and this is making use of APIs that come from either SDK 14393 or 15063 on the PackageCatalog class in order to check what content groups are available and if I find that my Albums/BoxSets groups are available then I have code which goes and adds all the pictures from those folders to the collections which live behind the UI.

The code is also attempting to handle the PackageInstalling event to see if I can dynamically respond to the 2 non-required packages being added while the application is running and note the comment in there about me not actually having seen that code run just yet and I’ll come back to why that is in just one second as it’s the wrong code Smile

Testing…

How to try this out?

In the //build session, there’s a few options listed around how you can test/debug a streaming install without actually putting your application into the Store. One method makes use of the PackageCatalog APIs to programmatically change the installation status of the content groups, another makes use of the Windows Device Portal (although I’m unsure as to whether this one is implemented yet) and there’s an option around using the regular PowerShell add-appxpackage command.

Testing via PowerShell

I thought I’d try the PowerShell option first and so I made a .APPX package for my application via the Store menu in Visual Studio;

image

and then made sure that I wasn’t making an APPX bundle;

image

and then I got hold of the temporary certificate that this generates and made it trusted on my system before going to install the .APPX file via PowerShell;

image

and so the key part here is the new –RequiredContentGroupOnly parameter to the Add-AppxPackage command. With that command executed, I can see that the app only has access to the Posters collection of images from its required content group and so that all seems good;

image

I also found it interesting to go and visit the actual folder on disk where the application is installed and to see what the Albums/BoxSets folders representing the ‘automatic’ content groups look like.

The first thing to say is that those folders do exist and here’s what the content looks like  at this point in the process;

image

so there are “marker files” present in the folders and so (as advised in the //build session) code would have to be careful not to confuse the presence of the folders/files with the content group’s installation status.

I’d hoped to then be able to use the add-appxpackage command again to add the other two content groups (Albums/BoxSets) while the application was running but when I tried to execute that, I saw;

image

Now, this was “very interesting” Smile in that I was reading the section of this page titled “Sideloaded Stream-able App” and it suggested that;

With the debugger attached, you can install the automatic content groups by:

Add-AppxPackage –Path C:\myapp.appx

Which is the exact same command but without the flag (what happens is that the platform will see that the app is already installed and will only stage the files that are missing).

So I attached my debugger to the running app and ran the command again and, sure enough, I could see that the debugger hit a first-chance exception in that piece of untested code that I’d listed earlier;

image

and so, sure enough, my code was being called here as the package started to install but that code wasn’t working because it was confusing the content group name with the application’s full package name.

That didn’t surprise me too much, it had been a bit of a ‘wild guess’ that I might use the PackageCatalog.PackageInstalling event in this way and I was clearly wrong so I went and reworked that code to make use of the far more sensible sounding PackageContentGroupStaging event as below;

 async Task AddStreamedPictureSourcesAsync()
    {
      // Handle any streamed packages that are already installed.
      var groups = await Package.Current.GetContentGroupsAsync();

      // TBD - unsure exactly of the state to check for here in order
      // to be sure that the content group is present.
      foreach (var group in groups.Where(
        g => !g.IsRequired && g.State == PackageContentGroupState.Staged))
      {
        await this.AddPictureSourceAsync(group.Name, group.Name);
      }

      // Now set up handlers to wait for any others to arrive
      this.catalog = PackageCatalog.OpenForCurrentPackage();
      this.catalog.PackageInstalling += OnPackageInstalling;
      this.catalog.PackageContentGroupStaging += OnContentGroupStaging;
    }

    async void OnContentGroupStaging(
      PackageCatalog sender, PackageContentGroupStagingEventArgs args)
    {
      if (args.IsComplete)
      {
          await this.Dispatcher.RunAsync(CoreDispatcherPriority.Normal,
            async () =>
            {
              await this.AddPictureSourceAsync(
                args.ContentGroupName,
                args.ContentGroupName);
            }
          );
      }
    }

    async void OnPackageInstalling(
      PackageCatalog sender,
      PackageInstallingEventArgs args)
    {
      // TODO: Remove this handler, don't think it's useful but leaving
      // it for the moment for debugging.
      Debugger.Break();
    }

This looked like it was far more likely to work but what I found was;

  1. The Add-AppxPackage command would still fail when I tried to add the non-required content groups to the already running app.
  2. From the debugger, I could see that the PackageInstalling event was still firing but the PackageContentGroupStaging event wasn’t. I suspect that the Add-AppxPackage command is quitting out between those 2 stages and so the first event fires and the second doesn’t.

This means that I haven’t been able to use this method just yet to test what happens when the app is running and the additional content groups are installed.

The best that I could find to do here was to install the required content group using the –RequiredContentGroupOnly and then, with the application running, I could install the other groups using the –ForceApplicationShutdown option and, sure enough, the app would go away and come back with all 3 of my content groups rather than just the required one;

image

and so that shows that things are working across app executions but it doesn’t test out how they work when the application is up and running which might well be the case if the user gets the app from Store, runs it and then additional packages show up over the first few minutes of the user’s session with the app.

Testing via the Streaming Install Debugging App

At this point, I went back to this blog post and tried out the steps under the heading of “Using the Streaming Install Debugging App”. This involves going off to download this app from github which then uses the APIs to manipulate the installation status of the content groups within my app.

I uninstalled my app from the system and then reinstalled it by hitting F5 in Visual Studio and then I ran up the debugging app and, sure enough, it showed me the details of my app;

image

and so I can now use this UI to change the status of my two content groups BoxSets and Albums to be ‘not staged’;

image

and then run up my app alongside this one and it correctly just shows the ‘Film Posters’ content;

image

and if I dynamically now switch a content group’s state to Staged then my app updates;

image

and I can repeat that process with the Albums content group;

image

and so that all seems to be working nicely Smile

Wrapping Up

I really like these new sorts of capabilities coming to UWP packaging and the APIs here seem to make it pretty easy to work with although, clearly, you’d need to give quite a lot of early-stage thought to which pieces of your application’s content should be packaged into which content groups.

I put the code that I was playing with here onto github if you’re interested in picking up this (admittedly very simple) sample.

Hitchhiking the HoloToolkit-Unity, Leg 14–More with Spatial Understanding

NB: The usual blog disclaimer for this site applies to posts around HoloLens. I am not on the HoloLens team. I have no details on HoloLens other than what is on the public web and so what I post here is just from my own experience experimenting with pieces that are publicly available and you should always check out the official developer site for the product documentation.

I experimented with Spatial Understanding back in this blog post;

Hitchhiking the HoloToolkit-Unity, Leg 3–Spatial Understanding (& Mapping)

but I’ve done more with it in the meantime and wanted to try and write up a small part of that here.

Firstly, let’s outline a basic project with some pieces within it. In the Unity project in the screenshot below I have brought in the HoloToolkit-Unity and I’ve set up the project for HoloLens project settings, scene settings and I’ve made sure that I have the microphone and spatial perception capability settings.

I’ve then brought in the standard SpatialMapping, SpatialUnderstanding and InputManager prefabs.

image

I’ve done little to alter them other than the settings shown below for SpatialMapping;

image

and these settings for SpatialUnderstanding;

image

and then I’ve added to my Placeholder object some components from the toolkit;

image

Including a keyword manager to handle voice commands and Space Visualizer and App State object which I brought in to try and visualise the ‘smoothed’ mesh produced by spatial understanding.

With a small amount of script in place to handle my two voice commands of Scan and Stop;

using HoloToolkit.Unity;
using UnityEngine;

public class Placeholder : MonoBehaviour {


    void Start ()
    {
        SpatialUnderstanding.Instance.ScanStateChanged += OnScanStateChanged;
	}
    void OnScanStateChanged()
    {
        if (SpatialUnderstanding.Instance.ScanState == SpatialUnderstanding.ScanStates.Done)
        {
            this.isScanning = false;
            this.isStopping = false;
        }
    }
    public void OnScan()
    {
        if (!this.isScanning)
        {
            this.isScanning = true;
            SpatialUnderstanding.Instance.RequestBeginScanning();
        }
    }
    public void OnStop()
    {
        if (this.isScanning && !this.isStopping)
        {
            this.isStopping = true;
            SpatialUnderstanding.Instance.RequestFinishScan();
        }
    }
    bool isStopping;
    bool isScanning;
}

 

and that’s enough for me to be able to use Scan and Stop in order to see the smoothed mesh from Spatial Understanding over my environment.

image

and then if I want to (e.g.) find and visualize the largest wall in my environment then I can make use of convenience methods on the SpaceVisualizer class – e.g.

    void OnScanStateChanged()
    {
        if (SpatialUnderstanding.Instance.ScanState == SpatialUnderstanding.ScanStates.Done)
        {
            this.isScanning = false;
            this.isStopping = false;

            // Use Space Visualizer to find a large wall
            SpaceVisualizer.Instance.Query_Topology_FindLargeWall();
        }
    }

and that method in SpaceVisualizer looks like;

        public void Query_Topology_FindLargeWall()
        {
            ClearGeometry();

            // Only if we're enabled
            if (!SpatialUnderstanding.Instance.AllowSpatialUnderstanding)
            {
                return;
            }

            // Query
            IntPtr wallPtr = SpatialUnderstanding.Instance.UnderstandingDLL.PinObject(resultsTopology);
            int wallCount = SpatialUnderstandingDllTopology.QueryTopology_FindLargestWall(
                wallPtr);
            if (wallCount == 0)
            {
                AppState.Instance.SpaceQueryDescription = "Find Largest Wall (0)";
                return;
            }

            // Add the line boxes
            float timeDelay = (float)lineBoxList.Count * AnimatedBox.DelayPerItem;
            lineBoxList.Add(
                new AnimatedBox(
                    timeDelay,
                    resultsTopology[0].position,
                    Quaternion.LookRotation(resultsTopology[0].normal, Vector3.up),
                    Color.magenta,
                    new Vector3(resultsTopology[0].width, resultsTopology[0].length, 0.05f) * 0.5f)
            );
            AppState.Instance.SpaceQueryDescription = "Find Largest Wall (1)";
        }

Now, we have some interesting calls in here referencing SpatialUnderstanding.Instance and then SpatialUnderstandingDllTopology and then SpatialUnderstanding.Instance.UnderstandingDLL and this produces a result such as that blurred out below where the pink lines represent an edge of the largest wall that the device could see;

image

So, the result is fine but what’s going on with the structure of the code here? I think it’s relevant to talk about how some of these pieces are layered together because I think it can be confusing to make use of the spatial understanding pieces and I forget how it works each time that I come to this library and so I’m writing them down as far as I understand them.

SpatialUnderstanding (the Unity script)

The script named SpatialUnderstanding is a singleton accessed via the static SpatialUnderstanding.Instance and is what showed up in the editor as the SpatialUnderstanding component.

It has properties such as AutoBeginScanning, UpdatePeriod_DuringScanning, UpdatePeriod_AfterScanning and it controls the scanning process via methods like RequestBeginScanning/RequestFinishScan and maintains the current state via the ScanState property. It has an Update method which drives the scanning process.

It’s fairly clear what this does for us but where does the functionality ultimately come from?

SpatialUnderstanding (the native DLL)

Inside of the HoloToolkit (rather than the HoloToolkit-Unity) there is a native SpatialUnderstanding project which looks something like this;

image

This builds out into a Windows 8.1 WinRT DLL and ultimately shows up in Unity under the Plugins folder;

image

and it’s essentially a regular, flat DLL with a number of exports – these are defined in the various DLL_*.h header files so for example Dll_Interface.h contains methods like;

	// Init/term
	EXTERN_C __declspec(dllexport) int SpatialUnderstanding_Init();
	EXTERN_C __declspec(dllexport) void SpatialUnderstanding_Term();

	// Scan flow control
	EXTERN_C __declspec(dllexport) void GeneratePlayspace_InitScan(
		float camPos_X, float camPos_Y, float camPos_Z,
		float camFwd_X, float camFwd_Y, float camFwd_Z,
		float camUp_X, float camUp_Y, float  camUp_Z,
		float searchDst, float optimalSize);

and then Dll_Topology.h includes functions such as;

	EXTERN_C __declspec(dllexport) int QueryTopology_FindLargestWall(
		_Inout_ TopologyResult* wall);

Now, calling these functions from C# in Unity is going to require using PInvoke and so…

SpatialUnderstandingDll*.cs in the Toolkit

There’s a number of scripts in the toolkit which then provide PInvoke wrappers over these different functional areas exported from the native DLL;

image

and so if I dig into a file like SpatialUnderstandingDll.cs I’ll find there’s an internal, public class called Imports which has PInvoke signatures like;

            [DllImport("SpatialUnderstanding")]
            public static extern int SpatialUnderstanding_Init();

and so that’s how I might call into that one function exported from the DLL from my C# code and if I dig into SpatialUnderstandingTopologyDll.cs as another example then I’ll find;

        [DllImport("SpatialUnderstanding")]
        public static extern int QueryTopology_FindLargestWall(
            [In, Out] IntPtr wall);             // TopologyResult

and so these provide the wrappers that make these functions callable but there’s another ‘trick’ here…

SpatialUnderstandingDll.cs Script in the Toolkit

When it comes to calling into functions like this one below;

            [DllImport("SpatialUnderstanding")]
            public static extern int QueryPlayspaceStats(
                [In] IntPtr playspaceStats);    // PlayspaceStats

so, what do I provide as an IntPtr here? Well, the same class file defines;

            [StructLayout(LayoutKind.Sequential, Pack = 1)]
            public class PlayspaceStats
            {
                public int IsWorkingOnStats;				// 0 if still working on creating the stats

                public float HorizSurfaceArea;              // In m2 : All horizontal faces UP between Ground - 0.15 and Ground + 1.f (include Ground and convenient horiz surface)
                public float TotalSurfaceArea;              // In m2 : All !
                public float UpSurfaceArea;                 // In m2 : All horizontal faces UP (no constraint => including ground)
                public float DownSurfaceArea;               // In m2 : All horizontal faces DOWN (no constraint => including ceiling)
                public float WallSurfaceArea;               // In m2 : All Vertical faces (not only walls)
                public float VirtualCeilingSurfaceArea;     // In m2 : estimation of surface of virtual Ceiling.
                public float VirtualWallSurfaceArea;        // In m2 : estimation of surface of virtual Walls.

                public int NumFloor;                        // List of Area of each Floor surface (contains count)
                public int NumCeiling;                      // List of Area of each Ceiling surface (contains count)
                public int NumWall_XNeg;                    // List of Area of each Wall XNeg surface (contains count)
                public int NumWall_XPos;                    // List of Area of each Wall XPos surface (contains count)
                public int NumWall_ZNeg;                    // List of Area of each Wall ZNeg surface (contains count)
                public int NumWall_ZPos;                    // List of Area of each Wall ZPos surface (contains count)
                public int NumPlatform;                     // List of Area of each Horizontal not Floor surface (contains count)

                public int CellCount_IsPaintMode;           // Number paint cells (could deduce surface of painted area) => 8cm x 8cm cell
                public int CellCount_IsSeenQualtiy_None;    // Number of not seen cells => 8cm x 8cm cell
                public int CellCount_IsSeenQualtiy_Seen;    // Number of seen cells => 8cm x 8cm cell
                public int CellCount_IsSeenQualtiy_Good;    // Number of seen cells good quality => 8cm x 8cm cell
            };

and so that’s good – I guess that I just have to alloc one of these, pin it and pass it across the boundary before unpinning it after the method completes?

The class helps me with that again. Firstly, it has private members like this one (there are others following the same pattern);

        private Imports.PlayspaceStats reusedPlayspaceStats = new Imports.PlayspaceStats();
        private IntPtr reusedPlayspaceStatsPtr;

and then it provides a method like this one;

        public Imports.PlayspaceStats GetStaticPlayspaceStats()
        {
            return reusedPlayspaceStats;
        }

and so rather than having to make my own instance of PlayspaceStats I can just ‘borrow’ this one when I need it – e.g;

            var ptr = SpatialUnderstanding.Instance.UnderstandingDLL.GetStaticPlayspaceStatsPtr();
            SpatialUnderstandingDll.Imports.QueryPlayspaceStats(ptr);

and that Imports class also has methods to PinObject/String which both pins and puts them onto a list which can later be cleared via UnpinAllObjects so it’s essentially helping with the mechanics involved in calling the underlying DLL.

Extending Functionality

I was looking into the library here because I wanted to extend the functionality provided.

There’s lots of different functionality supported including examples like;

  • Getting hold of meshes
  • Raycasting to determine what type of object the user is looking at (wall, ceiling, etc)
  • Getting the space alignment, floor, ceilings etc.
  • Creating objects of specific sizes on walls, floors, ceilings etc. or randomly away from those obstacles.
  • Finding positions for objects on walls, ceilings, etc.

and quite a lot more. What I wanted though was a simple list of the walls that the library has found within my environment and I didn’t find a method in the library that already did this and so I thought it might be useful to step through how I might add that functionality for my own purposes in the future and for anyone else that wants to do something similar.

The functionality relating to walls seems to reside in these classes in the native SpatialUnderstanding project;

image

and so the first thing I did was to visit TopologyAnalyzer_W.h and add a new method signature;


	void	GetWalls(Vec3fDA& _outPos, Vec3fDA& _outNormal, 
		FloatDA& _outWidths, FloatDA& _outLengths, Bool _bAllowVirtualWall = TRUE);

and then I added an implementation of this in the .cpp file which essentially just copies the centre, normal, width, height of a wall out of the array it resides in;

void TopologyAnalyzer_W::GetWalls(Vec3fDA& _outPos, Vec3fDA& _outNormal, 
	FloatDA& _outWidths, FloatDA&_outLengths, Bool _bAllowVirtualWall)
{
	for (S32 w = 0; w<m_daWalls.GetSize(); w++)
	{
		Wall& wall = m_daWalls[w];

		if (wall.m_bIsVirtual && !_bAllowVirtualWall)
			continue;

		_outPos.Add(wall.m_vCentroid);
		_outNormal.Add(wall.m_vNormal);
		_outLengths.Add(wall.m_fHeight);
		_outWidths.Add(wall.m_fWidth);
	}
}

With that in place, I can modify the Dll_Topology.h and .cpp files to expose that new function;

	EXTERN_C __declspec(dllexport) int QueryTopology_FindWalls(
		_Inout_ TopologyResult* wall);

and;

EXTERN_C __declspec(dllexport) int QueryTopology_FindWalls(
	_In_ int locationCount,
	_Inout_ Dll_Interface::TopologyResult* locationData)
{
	UnderstandingMgr_W &UnderstandingMgr = UnderstandingMgr_W::GetUnderstandingMgr();

	Vec3fDA outPos, outNormal;
	FloatDA outWidths, outLengths;

	UnderstandingMgr.GetPlayspaceInfos().m_TopologyAnalyzer.GetWalls(
		outPos, outNormal, outWidths, outLengths, FALSE);

	return(OutputLocations(locationCount, locationData, outPos, outNormal, outWidths,
		outLengths));
}

and that leans heavily on the existing OutputLocations function within that class.

I can then alter the PInvoke wrapper in SpatialUnderstandingDllTopology.cs in order to surface this new API;

        [DllImport("SpatialUnderstanding")]
        public static extern int QueryTopology_FindWalls(
            [In] int locationCount,             // Pass in the space allocated in locationData
            [In, Out] IntPtr locationData);     // TopologyResult

and then I’m set up to be able to make that call although naturally I need to make sure that I recompile my C++ code, get the resultant DLL and ensure that Unity has picked it up instead of the one that lives in the HoloToolkit-Unity by default.

Making Use of the Extension

I can then perhaps try this new ‘FindWalls’ functionality out by adding a method to the existing SpaceVisualizer to make this call and visualize it with rectangles;

        public void Query_Topology_FindWalls(int top)
        {
            // Only if we're enabled
            if (!SpatialUnderstanding.Instance.AllowSpatialUnderstanding)
            {
                return;
            }
            var resultsTopology = new SpatialUnderstandingDllTopology.TopologyResult[128];

            IntPtr resultsTopologyPtr =
                SpatialUnderstanding.Instance.UnderstandingDLL.PinObject(resultsTopology);
            
            int locationCount = SpatialUnderstandingDllTopology.QueryTopology_FindWalls(
                resultsTopology.Length,
                resultsTopologyPtr);

            if (locationCount != 0)
            {
                var rects = resultsTopology.OrderByDescending(
                    r => r.width * r.length).Take(Math.Max(top, resultsTopology.Length));

                foreach (var rect in rects)
                {
                    float timeDelay = (float)lineBoxList.Count * AnimatedBox.DelayPerItem;
                    lineBoxList.Add(
                        new AnimatedBox(
                            0.0f,
                            rect.position,
                            Quaternion.LookRotation(rect.normal, Vector3.up),
                            Color.blue,
                            new Vector3(rect.width, rect.length, 0.05f) * 0.5f));
                }
            }
        }

and I could then call that from my original code when the status of the scanning goes –> Done;

    void OnScanStateChanged()
    {
        if (SpatialUnderstanding.Instance.ScanState == SpatialUnderstanding.ScanStates.Done)
        {
            this.isScanning = false;
            this.isStopping = false;

            // Use Space Visualizer to find a large wall
            SpaceVisualizer.Instance.Query_Topology_FindWalls(10);

        }
    }

and that produces some ‘interesting’ results in giving me walls around this chair;

image

but, naturally, I could then filter down by size or filter down by angle to remove some of those from my result set but I’ve managed to get the data that I wanted out from the native code and into my C# code so that I can further work on it.

It’s possible that I’ve missed a “get all walls” method in here so feel free to comment and let me know if that’s the case but I thought I’d write up these rough notes in the meantime as I know I’ll be coming back to read this note myself in the future when I revisit spatial understanding again Smile