HoloLens Tracking of an Immersive Headset (or “Manual Spatial Anchoring”)

NB: The usual blog disclaimer for this site applies to posts around HoloLens. I am not on the HoloLens team. I have no details on HoloLens other than what is on the public web and so what I post here is just from my own experience experimenting with pieces that are publicly available and you should always check out the official developer site for the product documentation.

This post falls mainly into the category of “just for fun” but since I first got an immersive Windows Mixed Reality headset (an Acer) I’ve been quite keen to set up a situation where I could track its position using my HoloLens.

I don’t really know why and I don’t know whether there’s a tangible use for this, I just wanted to experiment with it.

What do I mean by track? Here’s a video example to explain. Please keep in mind that this was captured using mixed reality capture on a HoloLens which means that the quality is much lower than the on-device experience would be;

Tracking with Multiple HoloLens Devices

In building shared holographic experiences between multiple HoloLens devices its not too tricky to have multiple HoloLens devices all in one scene, networked together with each device capable of displaying the position, orientation and gaze vector of the other devices or some other shared holograms.

For the purposes of this post, I’m using the term “track” to describe the ability of one HoloLens to know the position and orientation of another device but it’s my own term rather than some official one.

There’s a script in the Mixed Reality Toolkit named RemoteHeadManager which does some of this for you and in previous blog posts like this one I’ve shown examples of doing that as demonstrated in the picture below;

where you can see a HoloLens floating and displaying its gaze ray. In that particular example the participant was remote and so there’s no local human being attached to that HoloLens but, hopefully, you get the idea.

Co-ordinate Systems

Being able to do this piece of magic ultimately comes down to being able to agree a common co-ordinate system between the multiple devices or at least a transformation from the co-ordinate system of one device to that of another.

When you first run an application on a HoloLens the starting device (or head) position is taken as the origin of the Unity scene (i.e. a Vector3(0,0,0)) with the X,Y,Z axes pointing to the right, up and forward in the natural way with respect to the device and/or the user’s head.

This means that if multiple HoloLens devices are present in a location then, unless they all run the application by being placed in the exact same physical start up spot, they are all going to have different positions in that location meaning that their origin point (0,0,0) will be in a different physical position and their X,Y,Z axes are likely to be pointing in different directions.

How to rationalise across these different co-ordinate systems in order to be able to display consistent content? The devices need to agree on something Smile

HoloLens sprinkles in some magic here because the device supports the idea of Spatial Anchors – a blob of data that represents a position and orientation in physical space.

The magic comes when you first learn that a HoloLens can export a spatial anchor, pass it over the network to another HoloLens and then that receiving device can attempt to import the same spatial anchor and locate it in the same space.

If that all works successfully (and generally it does) then the two devices now have an agreement about how a (position, rotation) within the room space is represented in their respective co-ordinate systems – this makes it “relatively easy” to consistently display objects.

A common way of then achieving that is to have each device maintain a GameObject locked to the position and orientation of the spatial anchor and then parent all content to be shared across devices from that GameObject such that all that content effectively has its origin and its axes determined by the anchored object.

This then means that e.g. a co-ordinate of (3,3,3) relative to the spatial anchored object on one device will show up in the same physical place in the world as a co-ordinate of (3,3,3) relative to the spatial anchored object on another device.

So, for HoloLens this is all good because of the magic of Spatial Anchors. What about an immersive headset?

Tracking with a HoloLens and an Immersive Headset

If you’ve looked at the immersive Mixed Reality headsets then you’ll know that they feature inside-out tracking and so it’s perhaps natural to assume that an application running on a PC displaying on an immersive headset would be able to import a spatial anchor from a HoloLens meaning that the code here would be the same as for the HoloLens scenario.

As far as I know, that’s not the case and I don’t believe it’s possible today to share a spatial anchor between an immersive headset and a HoloLens although I can’t quite find the definitive link that tells me that at the time of writing.

I’d be happy to be wrong here and it’d make the rest of the post redundant but that’d be a good thing Smile

Additionally, it’s relevant to consider that on an immersive headset the origin (0,0,0) and axis orientation (X,Y,Z) is not just determined by the place and direction that the headset is physically sitting at the point when the application first runs.

The documentation on coordinate systems explains different the scales of experience as being orientation, seated, standing, room and world and the different frames of reference that make these experiences possible.

One of these is the stage frame of reference where the origin is going to be on the floor of the room at the point that the user defined it when they set up their headset. So, for instance it’s perfectly possible for an app to start on an immersive headset at some position of (2,0.5,2) rather than at (0,0,0) as it would on HoloLens.

So, if I’ve got a HoloLens and an immersive headset operating in the same physical space then they almost certainly will have different origins within the space and differently aligned axes.

In order then for the HoloLens to somehow track the immersive headset in its own co-ordinate system, some form of manual means is going to be needed to agree on some common reference point that can be used to span co-ordinate systems.

Now, one way of doing this might be to use something like a Vuforia tag but the immersive headsets don’t have a web camera on them and so I’m not sure this would be feasible like it would on HoloLens.

With that in mind, I set about an approach of doing this manually along the lines of the following steps;

  1. HoloLens app runs up and displays some marker object that can be positioned in physical space.
    1. The HoloLens app can then create an empty GameObject at this co-ordinate with the same orientation
  2. Immersive headset runs up and is physically moved to the same place as the HoloLens marker object with the same orientation.
    1. The immersive app can then be informed (e.g. via a voice command) to create an empty GameObject at this co-ordinate with the same orientation
  3. The immersive headset sends its subsequent camera positions over the network relative to the GameObject created at step 2.1 above.
  4. The HoloLens headset can now reposition its marker object using the co-ordinates sent from the immersive headset relative to the game object created at step 1.1 above.

and, while quite “manual”, this seems to work out relatively nicely and the human being does the work of telling the devices how to align their respective co-ordinate systems.

It’s like spatial anchors for the generation who remember black and white TV Winking smile

In terms of how that was put together…

Putting it Together

The implementation of this seems relatively simple. I made a new Unity project in Unity 2017.2.0f3, brought in the Mixed Reality Toolkit and set up my project using the provided dialogs for;

  • Project Settings
  • Scene Settings
  • UWP Capabilities (including the microphone)

and so in the scene below, the only part that I created is the GameObject labelled Root with the rest coming from the toolkit dialogs;

image

From there, I wanted a shared experience and was happy to make use of the sharing server and so I brought in the SharingStage prefab from the toolkit and configured it for my local PC’s IP address and I also configured it to use the AutoJoinSessionAndRoom script from the toolkit such that it would automatically join a default session and room on connection;

image

Also on that same object is a script called Logic.cs which simply tries to enable (in a clunky way) one of two child objects named HoloLens and Immersive depending on which type of headset the code is running on;

image

From there, the HoloLens object looks like this;

image

and so it makes use of a modified version of the CustomMessages.cs script taken from the toolkit’s tests project and then also contains this HoloLensLogic.cs script which essentially;

  • Creates the prefab representing the spectacles 1.5m in front of the user and locks them to their gaze (this is a cheap way of positioning them)
  • Waits for a click event and then
    • Creates a new game object at the position where the spectacles are to be used as the parent representing that transform in space
    • Registers to receive broadcasts of the immersive headset position and forward vector
  • On receipt of a broadcast
    • Updates the position of the spectacles (relative to the parent) to reflect the update from the remote immersive headset

On the immersive side, the game object is as below;

image

and so it also uses the same CustomMessages script, it also sets itself up to handle the speech keyword “mark” and has the ImmersiveLogic.cs script set up to provide that handling which;

  • Waits for the speech keyword “mark” and then creates a GameObject to represent the position, orientation of the Camera in space at the point when that keyword is received.
  • Once the common co-ordinate point has been defined, transmits its Camera position and forward vector relative to that GameObject over the network to the HoloLens on every Update().

Wrapping Up

That’s pretty much it – nothing too complicated once I’d decided on an approach although it took me a little while to figure things out initially and I learned a couple of things during the process.

As I said at the start of the post, this was “just for fun” and I’m not yet decided on the use cases for establishing a common co-ordinate system across HoloLens/immersive but something in me would like to take it one step further and add the code to make the immersive headset display the actual position of the HoloLens in its surroundings even if that doesn’t necessarily 100% make sense in an immersive environment.

Maybe I could also then add some code to create other holograms consistently positioned across the two devices. I’m not sure what it would ‘feel’ like to position an object in the real world with HoloLens and then to don an immersive headset and have that object appear “in the same location” given that I couldn’t see that location! Perhaps I need to try Smile

I may update the code to do that at a later point – in the meantime, it’s all over here on github.

Future Decoded 2017 Resources

NB: The usual blog disclaimer for this site applies to posts around HoloLens. I am not on the HoloLens team. I have no details on HoloLens other than what is on the public web and so what I post here is just from my own experience experimenting with pieces that are publicly available and you should always check out the official developer site for the product documentation.

At this year’s Future Decoded show, I presented a couple of sessions with my colleague Pete around Windows Mixed Reality.

  • Windows Mixed Reality – Bootstrapping Your Development
  • Windows Mixed Reality – Developing with Unity

Neither of these were ‘official’ in the sense that Pete and I don’t work on the HoloLens or Mixed Reality teams at Microsoft and we aren’t even on the same continent and so you should always go check the official developer site;

for everything ‘official’ and always refer back to that in case of any doubt.

With the first of these sessions our intention was to provide a list of topics/resources that we’d want to know about if we were starting in Mixed Reality development today. We’ve since screen-captured that session and you can find the video of it below. Apologies on the audio, it’s a little “flat” which comes from recording straight into a PC with a cheap microphone.

In terms of the slides that we made/used for that session – they are downloadable in PDF form from here.

The intention of the second session was to make some of what we’d shown in the first session “come to life” by putting together some rough pieces in Unity with the Mixed Reality Toolkit (MRTK) and the Mixed Reality Design Labs (MRDL) and we’ve screen-captured that session also and it’s available below;

The sharp-eyed amongst you might notice that the video jumps between 1620 and 1920 vertical resolution, apologies for that – we’re aware of it.

As you’ll hear in the session, we worked from a slightly strange starting point because the MRTK and the MRDL were in the process of ‘coming together’ at the time that we made the session.

Since the session, that work has progressed and so we would start from a different place today than we did when we put the session together. See this pull request for the gory/glory details Smile and expect that there is more work to be done to bring some more of the MRDL goodness (e.g. bounding box) into the MRTK.

All mistakes are ours – feel free to let us know about them Smile

Butterflies ‘Hello World’ Demo (across holographic and immersive headsets)

NB: The usual blog disclaimer for this site applies to posts around HoloLens. I am not on the HoloLens team. I have no details on HoloLens other than what is on the public web and so what I post here is just from my own experience experimenting with pieces that are publicly available and you should always check out the official developer site for the product documentation.

Just a quick post to share a ‘hello world’ style demo that I made to try and illustrate a few pieces in the Mixed Reality Toolkit for Unity that could be used to make code which ran across both immersive and a HoloLens headsets with a sprinkling of common features including;

  • Gaze
  • Gesture
  • Voice
  • Spatial mapping
  • Spatial sound
  • Boundary and additional content for an immersive headset

I found a couple of models on Remix3D and on the Unity asset store and I set something up around the theme of butterflies which fly around, banging into walls and responding to a couple of global/local voice commands and gestures. The flying of the butterflies is completely dubious but I quite liked the way it worked out.

I suspect I made a few errors and it wasn’t quite perfect (my teleporting isn’t quite right outside of the editor) but I screen captured it here in case it’s of any use to others;

and you can get the code (feel free to fix it Smile) from here.

There’s nothing innovative in here other than the various mistakes which I made – those are uniquely mine.