Following on from the previous post, it’s time to get some holograms onto the screen and make it such that they are moveable by the user.

The “easiest” way to do this would seem to be add a voice command such that some hologram is created when the user issues a keyword and the easiest thing to create is (always !) a cube and so I started there.

Adding a Voice Command

Adding voice commands is pretty easy with the MRTK.

I went to the Input section of my MRTK profile, cloned the speech commands profile and added in a new “Cube” keyword as below;

and then I added an instance of Speech Input Handler to my Root object as below and wired it up to a new empty method on my PhotonScript named OnCreateCube;

Representing the Root Object

When developing with anchors, it’s always “nice” to have a representation of “where” the anchor is in space and whether it’s been created, located etc.

In my previous post, my anchor was simply represented by a blue beam running through the centre of the anchor location so I improved this slightly have that Root object now contain some 3D axes;

and I also changed the code to add materials such that I could change the colour of the sphere to indicate the anchor status. It starts off white but then;

• if the anchor is created, it turns blue
• if the anchor is located, it turns green
• if there’s an error, it turns red

I was surprised by how useful it is to run the app, see the axes appear at 0,0,0 on my head and then watch the sphere turn green and the axes jump around in space to their previously anchored location – it’s really handy to have a cheap visualisation.

Creating Objects

Now I just need something to create so I made a simple prefab which is a cube scaled to 0.2 x 0.2 x 0.2 along with some MRTK scripts to make it moveable, namely BoundingBox, ManipulationHandler and NearInteractionGrabbable;

Note that the prefab also has the PhotonView component on it so as to make it possible to instantiate this prefab with Photon as a “networked” object.

With that in place, I can add a field to my PhotonScript to store this prefab and then instantiate it in response to the “Cube!” voice command;

    public void OnCreateCube()
    {
        // Position it down the gaze vector
        var position = Camera.main.transform.position + Camera.main.transform.forward.normalized * 1.2f;

        // Create the cube
        var cube = PhotonNetwork.InstantiateSceneObject(this.cubePrefab.name, position, Quaternion.identity);
    }

and that all works quite nicely and I’m creating cubes and my intention with using InstantiateSceneObject is to have those cubes be “owned” by the scene rather than by a particular player so I’m hoping that they will stick around when the player who created them leaves the room.

Parenting Objects Created by Photon

In the editor, though, I notice that those cubes are being created without a parent when I really want them to be parented under my Root object as this is the one which will be anchored so as to sit in the same physical position across devices;

It would be fairly easy for me to grab the return value from PhotonNetwork.InstantiateSceneObject and change the parent relationship but that’s not going to help me if these objects are being created over the network from another user on another device so I need to try a different approach.

It turns out that I can hook into the instantiation of a networked object by implementing the IPunInstantiateMagicCallback ( ! ) interface and so I wrote an (ugly) script called CubeScript which I attached to my prefab in an attempt to pick up the newly created object and parent it back under the Root object in my scene;

using Photon.Pun;
using UnityEngine;

public class CubeScript : MonoBehaviour, IPunInstantiateMagicCallback
{
    public void OnPhotonInstantiate(PhotonMessageInfo info)
    {
        var parent = GameObject.Find("Root");

        this.transform.SetParent(parent.transform, true);
    }
}

Clearly, I need to come up with a better way of doing that then by using GameObject.Find() but this let me experiment.

I deployed that application to a HoloLens, ran it up, created a few cubes and then shut it down and ran it back up again and, sure enough, the cubes came back in the real world where they were originally created and so my assumption is that they would be visible in the same place in the real world to a second, third, etc. HoloLens user of the app.

However, I’ve got scripts on this cube which allow the user to translate, rotate and scale these holograms and, as yet, there’s nothing synchronising those changes to the network. That means that if I create a hologram at point x,y,z in physical space and then move it x1,y1,z1 then another use will not see those changes on their device. Similarly, if I re-run the application on the first device, I will see the hologram back at x,y,z. That needs attention…

Synchronising Hologram Transformations

There seemed to be an obvious way to do this transform sync’ing with Photon and it was calling out to me from the moment that I added the PhotonView script to my prefab;

If I change this “Observed Components” value to point at the object itself then Photon nicely adds for me;

and so it already (via the PhotonTransformView) knows how to synchronise scale, rotate and translate values across networked game objects.

But…I’m not sure that it’s going to do what I want here because, from looking at the script itself it is set up to synchronise the values of Transform.position, Transform.rotation and Transform.localScale.

The challenge with that is that “world” co-ordinates like this are going to correspond to different physical locations on multiple devices. For my scenario, I have my Root object which is spatially anchored to the same place in the real-world so providing a common “origin” for anything parented under it. That means that I need to then synchronise the co-ordinates of my cubes relative to that Root parent object.

That caused me to look for a “local” flag on PhotonTransformView or perhaps a separate PhotonTransformLocalView or similar & I didn’t find one so I simply made one by copying the original script and changing all transform.position and transform.rotation to refer to the localPosition and localScale instead and I configured that on my prefab;

I then tested this by running the application on my HoloLens and in the editor at the same time but I noticed an “interesting” thing in that cubes would be created ok but movement would only be sync’d from the device that created them, not from the other device.

I’d kind of expected this as Photon talks a lot about “ownership” of these networked objects and if you look at the description for RequestOwnership on this page then you’ll see that the “owner” of the object is the client that sends updates to position which implies that non-owners do not.

In configuring my PhotonView, I’d tried to set the “owner” to be “Takeover” intending anyone to own any object they liked but that wasn’t quite enough to make this work.

Photon Object-Control on Focus

I wasn’t sure whether I could actually tell Photon to “not care” about “ownership” but I suspect not and so rather than trying to do that I simply tried to code around it by trying to RequestOwnership of any cube any time the user focused on it.

So, I modified my CubeScript such that it now looked like;

using Microsoft.MixedReality.Toolkit.Input;
using Photon.Pun;
using UnityEngine;

public class CubeScript : MonoBehaviour, IPunInstantiateMagicCallback, IMixedRealityFocusHandler
{
    public void OnFocusEnter(FocusEventData eventData)
    {
        // ask the photonview for permission
        var photonView = this.GetComponent<PhotonView>();

        photonView?.RequestOwnership();
    }

    public void OnFocusExit(FocusEventData eventData)
    {
    }

    public void OnPhotonInstantiate(PhotonMessageInfo info)
    {
        var parent = GameObject.Find("Root");

        this.transform.SetParent(parent.transform, true);
    }
}

and this seemed to work fine for my scenario – I could move the cube on the HoloLens and I could move it in the editor and those movements were sync’d to the other device.

However, I noticed another challenge – my cubes still weren’t always where I expected them to be…

If a cube transforms in an empty room…

By this point, I was running through a set of steps as below;

• Run the app on HoloLens to create the room and the anchor
• Create some cubes
• Run up the app on the editor
• Test to see that cubes could be transformed on the HoloLens and sync’d to the editor and vice versa
• Quit the app on the HoloLens and re-run it to check that it would join the room, locate the anchor and put the cubes back where I left them
• Test again to see that cubes could be transformed on the HoloLens and sync’d to the editor and vice versa

and all was good – everything there seemed to work fine.

Where I had a problem though was in the scenario where a user was alone in the room. In that scenario, I found that leaving/joining the room would result in cubes with transforms reset to their starting values – i.e. any transformations that had been performed on the cube since it was created were lost. I would see the same whether I tried this out from the HoloLens or from the editor.

Initially, I thought that this related to Photon automatically clearing out the events associated with a player when they left the room and so I updated my room creation code to set the CleanupCacheOnLeave option to be false;

    public override void OnConnectedToMaster()
    {
        base.OnConnectedToMaster();

        var roomOptions = new RoomOptions();
        roomOptions.EmptyRoomTtl = this.emptyRoomTimeToLiveSeconds * 1000;
        roomOptions.CleanupCacheOnLeave = false;
        PhotonNetwork.JoinOrCreateRoom(ROOM_NAME, roomOptions, null);
    }

but this seemed to make no difference.

I spent a little time debugging and ultimately confirmed my thought that Photon does not send these messages out to a room of one player. This is from the PhotonNetworkPart.cs script;

and that (very nice and extremely helpful) comment also told me that these OnSerialize messages aren’t buffered by Photon.

Now, if I’d read between the lines a little more carefully on the Photon documentation page;

Synchronization and State

Then I think I would have known this all along because it does point out that RPCs can be buffered but that object synchronizations are not;

” Unlike Object Synchronization, RPCs might be buffered. Any buffered RPC will be sent to players who join later, which can be useful if actions have to be replayed one after another. For example a joining client can replay how someone placed a tool in the scene and how someone else upgraded it. The latter depends on the first action. “

This means that when a user joins the room, they will only get the correct current transforms for any cubes if there is another user in the room that is sending those transforms out to the network. Additionally, I think this is also dependent on the value of the ViewSynchronization value – see the piece in that referenced document that talks about “unreliable” versus “unreliable on change” which details when updates are sent to the network.

That’ll teach me to read the manual properly next time 🙂

Frequencies and Buffering…

It’d be fairly easy to replace the functionality that the PhotonTransferLocalView is currently providing for me with Photon RPCs that could be buffered but I might then fall into the trap of having lots (probably too many) RPCs being buffered every time the user scales, rotates or moves an object. That’s not likely to be a great choice – I’d perhaps rather rely on the unbuffered behaviour that I have already.

What I really want is some sort of bufferedLatest option such that we do buffering but only for the last update sent but I don’t know that Photon has that type of functionality.

So, for the scenario were > 1 users are in a room manipulating holograms I’m going to keep the immediacy offered by my PhotonTransferLocalView.

For the scenario where users leave the room and return, I need to have some other approach and I thought that I would go back to using custom properties on the room with the idea being to;

• watch for when a user completes a manipulation and use that to update a custom property on the room, using the ID of the PhotonView as the key for the property and a string value representing localPosition, localScale, localRotation as the value of the property.

then, whenever Photon instantiates a cube, I can check to see if this property is present for that cube’s PhotonView ID and, if so, apply these local transform values.

What does that do to my code? Firstly, I set up the scripts on my cube objects such that they handled the end of manipulations.

I did this for my BoundingBox for Rotate Stopped and Scale Stopped;

and I also did it for ManipulationHandler;

Why do this in both places? Because of this MRTK “issue”;

OnManipulationEnded doesn’t fire for rotation or scale

With that in place, I can use that event to serialize the local transform and put it into a custom property on the room;

    string ViewIDAsString => this.GetComponent<PhotonView>().ViewID.ToString();

    public void OnManipulationEnded()
    {
        var photonView = this.GetComponent<PhotonView>();

        if (photonView != null)
        {
            var transformStringValue = LocalTransformToString(this.transform);

            PhotonNetwork.CurrentRoom.SetCustomProperties(
                new Hashtable()
                {
                    {  this.ViewIDAsString, transformStringValue }
                }
            );
        }
    }

I’ll spare you the details of the LocalTransformToString method, it’s just capturing position, rotation, scale into a string.

Then, when Photon instantiates a networked cube I can add a little extra code to the method that I already had which reparents it in order access the custom property value from the room and use it to put the transform on the cube back to how it was at the last recorded manipulation;

    public void OnPhotonInstantiate(PhotonMessageInfo info)
    {
        var parent = GameObject.Find("Root");
        this.transform.SetParent(parent.transform, true);

        // Do we have a stored transform for this cube within the room?
        if (PhotonNetwork.CurrentRoom.CustomProperties.Keys.Contains(this.ViewIDAsString))
        {
            var transformStringValue = PhotonNetwork.CurrentRoom.CustomProperties[this.ViewIDAsString] as string;

            StringToLocalTransform(this.transform, transformStringValue);
        }
    }

and that seems to work out pretty nicely – using the PhotonTransformView for the non-buffered, frequently changing values and using “buffered” custom room properties for values that will change less frequently.

Wrapping Up

As always, I learned a few things while trying to put this post and the previous one together and, mainly, I learned about Photon because I don’t have a tonne of familiarity with it.

That said, getting the basics of a shared holographic experience up and running wasn’t too difficult and if I needed to spin up another examples those learnings would mean that I could get back to it pretty quickly.

I put the Unity project here on github in case you (or a future version of me) wanted to do anything with it – naturally, apply a pinch of salt as I put it together purely for the experiments in this post. Just one note – the keys for Azure Spatial Anchors embedded in that project won’t work, you’ll need to update to provide your own configuration.

I’ve written a lot in the past about shared holograms and I’ve also written about Photon networking a couple of years ago;

Experiments with Shared Holographic Experiences and Photon Unity Networking

but I recently was working through this new tutorial around shared experiences with Photon;

Multi-user Capabilities Tutorials

and, while it’s great, it had me wondering what the minimal set of pieces might be for getting a shared experience up and running on the current versions of Unity, the Mixed Reality Toolkit for Unity and HoloLens and so I set about trying that out and I’m going to jot down notes here in case anyone is looking at this for the first time.

What I found surprisingly good to see is that it is fairly simple to get to the point where you have shared holograms using a networking technology like Photon.

Let’s get going.

Versions

I am using the following pieces;

• Unity 2018.4.3f1
• Visual Studio 2019 (16.0.3)
• Mixed Reality Toolkit for Unity 2.0.0 RC1
• Photon Unity Networking (PUN) V2.13

Making a Project

To start with, I made a new 3D project in Unity using the “New Project” dialog box;

I then set this up for Mixed Reality development by doing what I think of as the bare minimum;

• Switch the platform to UWP
• Switch on Virtual Reality SDK support
• Set some items in the application manifest

that means using this dialog (File->Build Settings);

and this set of settings (Player Settings->XR Settings);

and this set of settings (Player Settings->Publishing Settings);

and, with that, my project is good to go.

Importing Toolkits

I then used the regular right mouse menu on my Project Assets to import the MRTK Foundation package downloaded from the previous link as below;

note that I import all of the assets here to get all of the toolkit. I then used the Mixed Reality Toolkit->Add to Scene and Configure menu and selected the default HoloLens profile;

I then went to the Unity Asset Store and searched for “PUN” to find this package which I then downloaded and imported;

When it came to the import here I wasn’t 100% sure that I needed all of PUN and so I deselected PhotonChat as I don’t think I need it;

and imported that into my application.

Importing Azure Spatial Anchors

In order to have a shared holographic experience, it’s crucial to establish a common co-ordinate system across the participating devices.

There’s lots of ways of establishing a shared co-ordinate system across MR devices with perhaps the usual one being to use a spatial anchor.

When I originally wrote about shared holograms with PUN in this blog post, I ended up writing a bunch of code to share spatial anchors using Azure blob storage because PUN doesn’t make it so easy to pass around large binary objects.

Here in 2019, though, we’ve got new options and Azure Spatial Anchors to help out with the heavy lifting and so I wanted to make use of Azure Spatial Anchors here to establish a common co-ordinate system.

I wrote a little about Azure Spatial Anchors here;

Baby Steps with the Azure Spatial Anchors Service

and so I won’t repeat everything that I said but will, instead, try and keep undertaking doing the minimal to get what I want up and running.

In that blog post, I talked about how I take only these pieces (5 files) from the official SDK sample here;

But I wanted a simpler wrapper class to make this “easier” for my code to work with and so I wrote the class below. Note that this is most definitely making some very simplistic trade-offs in terms of sacrificing failure handling and error conditions for simplicity and especially in a couple of places;

• By not following the guidance and providing callbacks here such that an app using this class can provide UI to tell the user what to do to improve the capture of an anchor
• By potentially having loops that might execute for ever – I’m being very optimistic here.

With those caveats in place, here’s my little helper class;

using Microsoft.Azure.SpatialAnchors;
using Microsoft.MixedReality.Toolkit.Utilities;
using System;
using System.Threading.Tasks;
using UnityEngine;
using UnityEngine.XR.WSA;

namespace AzureSpatialAnchors
{
    public class AzureSpatialAnchorService : MonoBehaviour
    {
        [Serializable]
        public class AzureSpatialAnchorServiceProfile
        {
            [SerializeField]
            [Tooltip("The account id from the Azure portal for the Azure Spatial Anchors service")]
            string azureAccountId;
            public string AzureAccountId => this.azureAccountId;

            [SerializeField]
            [Tooltip("The access key from the Azure portal for the Azure Spatial Anchors service (for Key authentication)")]
            string azureServiceKey;
            public string AzureServiceKey => this.azureServiceKey;
        }

        [SerializeField]
        [Tooltip("The configuration for the Azure Spatial Anchors Service")]
        AzureSpatialAnchorServiceProfile profile = new AzureSpatialAnchorServiceProfile();
        public AzureSpatialAnchorServiceProfile Profile => this.profile;

        TaskCompletionSource<CloudSpatialAnchor> taskWaitForAnchorLocation;

        CloudSpatialAnchorSession cloudSpatialAnchorSession;

        public AzureSpatialAnchorService()
        {
        }
        public async Task<string> CreateAnchorOnObjectAsync(GameObject gameObjectForAnchor)
        {
            string anchorId = string.Empty;
            try
            {
                this.StartSession();

                var worldAnchor = gameObjectForAnchor.GetComponent<WorldAnchor>();

                if (worldAnchor == null)
                {
                    throw new ArgumentException("Expected a world anchor on the game object parameter");
                }

                // Note - these next 2 waits are highly dubious as they may never happen so
                // a real world solution would have to do more but I'm trying to be 
                // minimal here
                await new WaitUntil(() => worldAnchor.isLocated);

                // As per previous comment.
                while (true)
                {
                    var status = await this.cloudSpatialAnchorSession.GetSessionStatusAsync();

                    if (status.ReadyForCreateProgress >= 1.0f)
                    {
                        break;
                    }
                    await Task.Delay(250);
                }
                var cloudAnchor = new CloudSpatialAnchor();

                cloudAnchor.LocalAnchor = worldAnchor.GetNativeSpatialAnchorPtr();

                await this.cloudSpatialAnchorSession.CreateAnchorAsync(cloudAnchor);

                anchorId = cloudAnchor?.Identifier;
            }
            catch (Exception ex) // TODO: reasonable exceptions here.
            {
                Debug.Log($"Caught {ex.Message}");
            }
            return (anchorId);
        }
        public async Task<bool> PopulateAnchorOnObjectAsync(string anchorId, GameObject gameObjectForAnchor)
        {
            bool anchorLocated = false;

            try
            {
                this.StartSession();

                this.taskWaitForAnchorLocation = new TaskCompletionSource<CloudSpatialAnchor>();

                var watcher = this.cloudSpatialAnchorSession.CreateWatcher(
                    new AnchorLocateCriteria()
                    {
                        Identifiers = new string[] { anchorId },
                        BypassCache = true,
                        Strategy = LocateStrategy.AnyStrategy,
                        RequestedCategories = AnchorDataCategory.Spatial
                    }
                );

                var cloudAnchor = await this.taskWaitForAnchorLocation.Task;

                anchorLocated = cloudAnchor != null;

                if (anchorLocated)
                {
                    gameObjectForAnchor.GetComponent<WorldAnchor>().SetNativeSpatialAnchorPtr(cloudAnchor.LocalAnchor);
                }
                watcher.Stop();
            }
            catch (Exception ex) // TODO: reasonable exceptions here.
            {
                Debug.Log($"Caught {ex.Message}");
            }
            return (anchorLocated);
        }
        /// <summary>
        /// Start the Azure Spatial Anchor Service session
        /// This must be called before calling create, populate or delete methods.
        /// </summary>
        public void StartSession()
        {
            if (this.cloudSpatialAnchorSession == null)
            {
                Debug.Assert(this.cloudSpatialAnchorSession == null);

                this.ThrowOnBadAuthConfiguration();
                // setup the session
                this.cloudSpatialAnchorSession = new CloudSpatialAnchorSession();
                // set the Azure configuration parameters
                this.cloudSpatialAnchorSession.Configuration.AccountId = this.Profile.AzureAccountId;
                this.cloudSpatialAnchorSession.Configuration.AccountKey = this.Profile.AzureServiceKey;
                // register event handlers
                this.cloudSpatialAnchorSession.Error += this.OnCloudSessionError;
                this.cloudSpatialAnchorSession.AnchorLocated += OnAnchorLocated;
                this.cloudSpatialAnchorSession.LocateAnchorsCompleted += OnLocateAnchorsCompleted;

                // start the session
                this.cloudSpatialAnchorSession.Start();
            }
        }
        /// <summary>
        /// Stop the Azure Spatial Anchor Service session
        /// </summary>
        public void StopSession()
        {
            if (this.cloudSpatialAnchorSession != null)
            {
                // stop session
                this.cloudSpatialAnchorSession.Stop();
                // clear event handlers
                this.cloudSpatialAnchorSession.Error -= this.OnCloudSessionError;
                this.cloudSpatialAnchorSession.AnchorLocated -= OnAnchorLocated;
                this.cloudSpatialAnchorSession.LocateAnchorsCompleted -= OnLocateAnchorsCompleted;
                // cleanup
                this.cloudSpatialAnchorSession.Dispose();
                this.cloudSpatialAnchorSession = null;
            }
        }
        void OnLocateAnchorsCompleted(object sender, LocateAnchorsCompletedEventArgs args)
        {
            Debug.Log("On Locate Anchors Completed");
            Debug.Assert(this.taskWaitForAnchorLocation != null);

            if (!this.taskWaitForAnchorLocation.Task.IsCompleted)
            {
                this.taskWaitForAnchorLocation.TrySetResult(null);
            }
        }
        void OnAnchorLocated(object sender, AnchorLocatedEventArgs args)
        {
            Debug.Log($"On Anchor Located, status is {args.Status} anchor is {args.Anchor?.Identifier}, pointer is {args.Anchor?.LocalAnchor}");
            Debug.Assert(this.taskWaitForAnchorLocation != null);

            this.taskWaitForAnchorLocation.SetResult(args.Anchor);
        }
        void OnCloudSessionError(object sender, SessionErrorEventArgs args)
        {
            Debug.Log($"On Cloud Session Error: {args.ErrorMessage}");
        }
        void ThrowOnBadAuthConfiguration()
        {
            if (string.IsNullOrEmpty(this.Profile.AzureAccountId) ||
                string.IsNullOrEmpty(this.Profile.AzureServiceKey))
            {
                throw new ArgumentNullException("Missing required configuration to connect to service");
            }
        }
    }
}

It’s perhaps worth saying that while I packaged this as a MonoBehaviour here, I have other variants of this code that would package it as a Mixed Reality extension service which would make it available across the entire application rather than to a set of components that happen to be configured on a particular GameObject. In this case, I went with a MonoBehaviour and configured this into my scene as below;

Setting up PUN to Run with a Cloud Session

It’s possible to use PUN either using a local network server or a cloud server but I want things to be simple and with minimal configuration so I decided to run from the cloud.

With that in mind, I visited the PUN portal here;

https://dashboard.photonengine.com/en-US/publiccloud

and used the Create New App button to create a new app of type Photon PUN with a suitable name. I did not fill in the description or Url properties. I called it TestApp and you can see that the portal then gives me a GUID to represent that app as below;

Once the Photon package has imported into Unity, it conveniently pops up a dialog where I can enter this GUID to link the app with that cloud instance;

and that’s all I need to have this up and running.

Getting a Connection

Getting a connection is dead simple. There are, no doubt, a million options that you can use but all I did was to create an empty GameObject in my scene (named Root) and then write a script which inherits from the PUN base class MonoBehaviourPunCallbacks which provides overrides for network events and so I wrote;

    public class PhotonScript : MonoBehaviourPunCallbacks
    {
        void Start()
        {
            PhotonNetwork.ConnectUsingSettings();
        }
        public override void OnConnectedToMaster()
        {
            base.OnConnectedToMaster();
        }
    }

and trying that out in the editor and seeing it run through in the debugger all seemed to be working nicely.

Getting a Room

The “boundary” for communications in PUN seems to be the “room” and, as you might expect, there’s potential for a lot of functionality, capability and configuration around picking rooms and, optionally, using “lobbies” to select these rooms.

For my purposes, I’m going to pretend that none of this matters and it’s ok to just hard-code a room to avoid any of these additional steps.

Consequently, I can write some simple code to create/or join a room once the network connection is made;

    public class PhotonScript : MonoBehaviourPunCallbacks
    {
        void Start()
        {
            PhotonNetwork.ConnectUsingSettings();
        }
        public override void OnConnectedToMaster()
        {
            base.OnConnectedToMaster();

            PhotonNetwork.JoinOrCreateRoom("HardCodedRoom", null, null);
        }
        public override void OnJoinedRoom()
        {
            base.OnJoinedRoom();
        }
        public override void OnCreatedRoom()
        {
            base.OnCreatedRoom();
        }
    }

and, again, in the debugger attached to the editor I can see both the OnCreatedRoom and OnJoinedRoom overrides being called so things seem fine.

If I run the code again, I see that the room is once again created and joined and this comes down to the ‘time to live’ specified for the RoomOptions for the room. At the moment, my code does not pass any RoomOptions and so the room seems to get torn down pretty quickly whereas I could leave the room ‘alive’ for longer if I changed that value

There are also options on the room around whether objects that are created by a particular player are to be removed when that player leaves the room and around how much time needs to pass before a player is considered to have left the room. For my purposes, I’m not too worried about those details just yet so I tidied up my script and simply set the time to live value on the room itself such that it would give me enough time to join the same room more than once from a single device if I needed to;

    public class PhotonScript : MonoBehaviourPunCallbacks
    {
        enum RoomStatus
        {
            None,
            CreatedRoom,
            JoinedRoom
        }

        public int emptyRoomTimeToLiveSeconds = 120;

        RoomStatus roomStatus = RoomStatus.None;

        void Start()
        {
            PhotonNetwork.ConnectUsingSettings();
        }
        public override void OnConnectedToMaster()
        {
            base.OnConnectedToMaster();

            var roomOptions = new RoomOptions();
            roomOptions.EmptyRoomTtl = this.emptyRoomTimeToLiveSeconds * 1000;
            PhotonNetwork.JoinOrCreateRoom(ROOM_NAME, roomOptions, null);
        }
        public async override void OnJoinedRoom()
        {
            base.OnJoinedRoom();

            if (this.roomStatus == RoomStatus.None)
            {
                this.roomStatus = RoomStatus.JoinedRoom;
            }
        }
        public async override void OnCreatedRoom()
        {
            base.OnCreatedRoom();

            this.roomStatus = RoomStatus.CreatedRoom;
        }
        static readonly string ROOM_NAME = "HardCodedRoomName";
    }

Establishing a Common Co-ordinate System

The heart of a shared holographic experience revolves around using some mechanism to establish a co-ordinate system that’s common across all the devices that are participating in the experience.

I want to do this using world anchors on HoloLens and in conjunction with Azure Spatial Anchors in the cloud which is going to provide me with a mechanism to share the anchor from one device to another.

To keep things simple, I’m going to try and establish a 1:1 relationship between 1 anchor and 1 room in Photon. This probably isn’t realistic for a real-world application but it’s more than enough for my sample here.

The way that I want things to work is as below;

• When a user creates the Photon room, it will be assumed that user should also create the spatial anchor and post it to the Azure Spatial Anchors (ASA) service.
• When a user joins a room, it will be assumed that the user should attempt to find the anchor at the ASA service and import it if it exists and, otherwise, wait to be notified that changes have occurred and it should try that process again.

The object that I want to anchor in my scene is the Root object and I have added a WorldAnchor to it as below;

In terms of Photon, the mechanism that I chose to use to try and implement this was to use custom room properties as described here.

My process then becomes;

• For a user who creates a room, ensure that the WorldAnchor on the Root object isLocated and then use the ASA pieces to create an Azure Spatial Anchor from this getting the ID and storing it in a custom room property named anchorId
• For a user who joins a room they check the room properties to look for a property named anchorId
  • If present, use ASA to download the anchor and import it to the WorldAnchor on the Root object
  • If not present, assume that we are too early in the process and wait for Photon to call OnUpdateRoomProperties, letting us know that the anchorId property has now been set by the user who created the room & we can now access the value, call the ASA service & get the anchor.

and so my script ended up looking as below;

using System;
using System.Threading.Tasks;
using AzureSpatialAnchors;
using ExitGames.Client.Photon;
using Photon.Pun;
using Photon.Realtime;

public class PhotonScript : MonoBehaviourPunCallbacks
{
    enum RoomStatus
    {
        None,
        CreatedRoom,
        JoinedRoom,
        JoinedRoomDownloadedAnchor
    }

    public int emptyRoomTimeToLiveSeconds = 120;

    RoomStatus roomStatus = RoomStatus.None;

    void Start()
    {
        PhotonNetwork.ConnectUsingSettings();
    }
    public override void OnConnectedToMaster()
    {
        base.OnConnectedToMaster();

        var roomOptions = new RoomOptions();
        roomOptions.EmptyRoomTtl = this.emptyRoomTimeToLiveSeconds * 1000;
        PhotonNetwork.JoinOrCreateRoom(ROOM_NAME, roomOptions, null);
    }
    public async override void OnJoinedRoom()
    {
        base.OnJoinedRoom();

        // Note that the creator of the room also joins the room...
        if (this.roomStatus == RoomStatus.None)
        {
            this.roomStatus = RoomStatus.JoinedRoom;
        }
        await this.PopulateAnchorAsync();
    }
    public async override void OnCreatedRoom()
    {
        base.OnCreatedRoom();
        this.roomStatus = RoomStatus.CreatedRoom;
        await this.CreateAnchorAsync();
    }
    async Task CreateAnchorAsync()
    {
        // If we created the room then we will attempt to create an anchor for the parent
        // of the cubes that we are creating.
        var anchorService = this.GetComponent<AzureSpatialAnchorService>();

        var anchorId = await anchorService.CreateAnchorOnObjectAsync(this.gameObject);

        // Put this ID into a custom property so that other devices joining the
        // room can get hold of it.
        PhotonNetwork.CurrentRoom.SetCustomProperties(
            new Hashtable()
            {
                { ANCHOR_ID_CUSTOM_PROPERTY, anchorId }
            }
        );
    }
    async Task PopulateAnchorAsync()
    {
        if (this.roomStatus == RoomStatus.JoinedRoom)
        {
            object keyValue = null;

            // First time around, this property may not be here so we see if is there.
            if (PhotonNetwork.CurrentRoom.CustomProperties.TryGetValue(
                ANCHOR_ID_CUSTOM_PROPERTY, out keyValue))
            {
                // If the anchorId property is present then we will try and get the
                // anchor but only once so change the status.
                this.roomStatus = RoomStatus.JoinedRoomDownloadedAnchor;

                // If we didn't create the room then we want to try and get the anchor
                // from the cloud and apply it.
                var anchorService = this.GetComponent<AzureSpatialAnchorService>();

                await anchorService.PopulateAnchorOnObjectAsync(
                    (string)keyValue, this.gameObject);
            }
        }
    }
    public async override void OnRoomPropertiesUpdate(Hashtable propertiesThatChanged)
    {
        base.OnRoomPropertiesUpdate(propertiesThatChanged);

        await this.PopulateAnchorAsync();
    }
    static readonly string ANCHOR_ID_CUSTOM_PROPERTY = "anchorId";
    static readonly string ROOM_NAME = "HardCodedRoomName";
}

At this point, in order to try and “test” whether this worked or not I added a simple elongated cube under my Root object in the scene;

with the aim being to try this out on a single HoloLens device by performing;

1. Run the application standing in a particular position and orientation to establish an origin.
2. Wait a little while for the anchor to get created and sync’d to ASA.
3. Close the application.
4. Step a few metres to one side.
5. Re-run the application within 2 minutes to attempt to join the same room.
6. Wait a little for the anchor to get download and located.
7. Expect that the blue bar will ‘jump’ to the position that it had at step 1 above.

and that worked out fine on the first run of that project and so I have the “core” of a shared holographic experience up and running on Photon in that I can establish a common co-ordinate system across multiple devices with very little code indeed.

The next step (in the next blog post) would be to see if I can create some holograms and move them around…