NB: The usual blog disclaimer for this site applies to posts around HoloLens. I am not on the HoloLens team. I have no details on HoloLens other than what is on the public web and so what I post here is just from my own experience experimenting with pieces that are publicly available and you should always check out the official developer site for the product documentation.

Preamble

This post builds directly on these two posts;

Windows 10, UWP, HoloLens & A Simple Two-Way Socket Library 

Windows 10, UWP, HoloLens & A Simple Two-Way Socket Library (Part 2)

in that I am going to make use of the socket library that I built in that first post and re-packaged a little in the second post in order to attempt to share a simple holographic experience across devices.

It also more than likely builds on a number of these posts in that I’ll be leaning on some other things that I have previously written about.

Before I get going, I’ll flag that there’s an official lab on this topic;

Holograms 240

and that there’s a really good write up here around the whole concept;

Shared holographic experiences

and some more specifics here around how this works in Unity;

Shared holographic experiences in Unity

and so I hope that has you covered for official documentation links and what I’ll follow with below are just my own experiments in learning how this works without me just picking up someone else’s library and running with it which is largely what I’ve done to date when it comes to ‘shared holographic experiences’.

I chose the title for the blog post reasonably carefully in that I called the post ‘Shared Holographic Experiences’ rather than ‘Shared Holograms’.

It’s perhaps a subtle distinction but to me ‘Shared Holographic Experiences’ implies that it’s the data underpinning the experience that is being  shared across devices rather than some literal copies of the holograms themselves.   This allows for the scenarios outlined in ‘shared holographic experiences’ under the titles of presentation, collaboration and guidance including scenarios where one user may have different visibility/interactions with holograms than another as you might find in a teacher/student scenario.

For this post, I’m going to think about a simple shared holographic experience between two HoloLens devices and I’m going to attempt to give the users of those two devices the same visual representation of the scene but, naturally, each user is likely to be standing in a different place in the scene and interacting in a different way to the other user.

In order to do that, I’m going to explore an approach along the lines of;

1. Connect two HoloLens devices that are present on the same network.
2. Provide a mechanism via which a user can create some basic content (a cube, a sphere or some other model).
3. Ask the device where the content is created to give me some token (a spatial anchor) that contains the details of the location of the content in space.
4. Send that token over the network to the second device.
5. Attempt to redeem that token on that second device in order to come to a common understanding across the two devices of the content’s location.
6. Create the same content on the second device in the same position so as to replicate the experience from the first device.

As a stretch goal, I’d also like to do this in a relatively efficient manner and I’d like the user to be able to manipulate (e.g. move or maybe rotate) the content they have created.

Clearly, the magic here is in being able to do steps (3) and (4) – that is, for a hologram, being able to ask the device to give us some “token” that captures positional information about that hologram which can be understood by another device.

This is handled for us by spatial anchors which I experimented with in this post from a different point of view – that of being able to persist the positions of holograms across instances of an app. That’s quite some feat and, naturally, even more of a leap is to have those anchors be capable of export from one device and import to another but it’s a capability of the device and the SDK

Speaking of which, the APIs involved in this are detailed on that doc page that I referenced earlier and the process looks to run something like;

1. Ensure that the app has the spatial perception capability
2. Ensure that the hologram in question is anchored
3. Use the WorldAnchorTransferBatch.ExportAsync method to export the anchor – this produces a byte[] array which is produced in segments passed to an ‘on progress’ handler before an ‘on completed’ handler is called.
4. Use the WorldAnchorTransferBatch.ImportAsync method to import the anchor on the secondary device.

Naturally, there are going to be scenarios where the first device might not have enough information to do the export and that’s also going to be true for the importing device and so handling failures gracefully would be important in a real world scenario.

With all of that said, it’s time to get going and build up something simple that shares an experience across two HoloLens devices.

Note that I’m using a PC as my ‘second device’ here as I don’t have two HoloLens devices sitting side by side at the time of writing but I’m fairly confident that what I’m doing here will work across two HoloLens devices, I’ve tried similar code in that set up and I’ll try it again on two devices and update the post with the results when I have the opportunity.

I broke this process down into the following steps…

Steps 1 – 4

The initial steps that I mapped out;

• Step 1 – Making A Blank App with Settings and Capabilities

• Step 2 – Importing the HoloToolkit-Unity

• Step 3 – Adding the Networking Pieces from the Previous Blog Post

• Step 4 – Establishing a networking connection

all turned out to be pretty much identical to what I did in the video from my previous blog post and so I’m embedding that video again below rather than repeating it;

and I’ll continue from where that video left off below.

Step 4 – Create a Cube when the User Taps

With those pieces set up, my next steps is to create some object (e.g. a cube) when the user does an air-tap and I work through that in the screen capture below;

Step 5 – Adding and Exporting an Anchor

Once I have an object in place, I want to make sure that it is anchored into the world as per this article and also as per my own blog post here. Once anchored, I want to export the data behind the anchor from one of the two networked devices to the other one and I work through that in the screen capture here;

Step 6 – Importing the Anchor, Creating and Locking a Cube

Once I’ve got an anchor exported from my app running on one device, I need to make sure that the same app running on the second device knows how to import that anchor and recreate the object it represents. Let’s work through that;

Step 7 – Anchoring Every Cube?

At this point, it’s becoming apparent that exporting and importing an anchor takes a little bit of time. In my experiments here, anchors seem to serialize to around 1MB of data but I’ve seen them serialize to over 10MB of data and even in my case here it’s clear that the exporting of the anchor from the system (before the network transfer) can take some time.

Do I really need to create an anchor for every cube that I create or can I be more efficient and create a few anchors and then position subsequent cubes relative to those anchors? The docs suggest that a spatial anchor can be used to anchor content within about 3 metres of it and so perhaps I can be more efficient here and only anchor the first cube (for demo purposes) and hang any subsequent cubes off that.

With that in mind, I want to change my code such that;

1. The first cube created and exported/imported becomes an anchor for both devices.
2. All subsequent cubes will not have world anchors applied to them. Instead, their relative position to the main anchor will be calculated, shipped over the wire and used to recreate that cube on the other side of the wire.

I work through those additions in the screen capture below;

Step 8 – Rotating the Cube

Finally, I wanted to include some manipulation of the cubes and so I decided that I’d add some manipulations to everything but the anchor cubes (i.e. all cubes which are positioned relative to the anchor cube) so that they can be rotated. I work through those changes in the screen capture below;

Wrapping Up

This post got long but I feel that I learned quite a lot about the process of anchoring holograms, exporting those anchors and then bringing them in to another device.

Next steps for me would be to try this on two HoloLens devices at the same time to see how things work there – I’ll do that and will update the post once that’s done and share the code.