Simple Shared Holograms with Photon Networking (Part 1)

I’ve written a lot in the past about shared holograms and I’ve also written about Photon networking a couple of years ago;

Experiments with Shared Holographic Experiences and Photon Unity Networking

but I recently was working through this new tutorial around shared experiences with Photon;

Multi-user Capabilities Tutorials

and, while it’s great, it had me wondering what the minimal set of pieces might be for getting a shared experience up and running on the current versions of Unity, the Mixed Reality Toolkit for Unity and HoloLens and so I set about trying that out and I’m going to jot down notes here in case anyone is looking at this for the first time.

What I found surprisingly good to see is that it is fairly simple to get to the point where you have shared holograms using a networking technology like Photon.

Let’s get going.

Versions

I am using the following pieces;

Making a Project

To start with, I made a new 3D project in Unity using the “New Project” dialog box;

I then set this up for Mixed Reality development by doing what I think of as the bare minimum;

  • Switch the platform to UWP
  • Switch on Virtual Reality SDK support
  • Set some items in the application manifest

that means using this dialog (File->Build Settings);

and this set of settings (Player Settings->XR Settings);

and this set of settings (Player Settings->Publishing Settings);

and, with that, my project is good to go.

Importing Toolkits

I then used the regular right mouse menu on my Project Assets to import the MRTK Foundation package downloaded from the previous link as below;

note that I import all of the assets here to get all of the toolkit. I then used the Mixed Reality Toolkit->Add to Scene and Configure menu and selected the default HoloLens profile;

I then went to the Unity Asset Store and searched for “PUN” to find this package which I then downloaded and imported;

When it came to the import here I wasn’t 100% sure that I needed all of PUN and so I deselected PhotonChat as I don’t think I need it;

and imported that into my application.

Importing Azure Spatial Anchors

In order to have a shared holographic experience, it’s crucial to establish a common co-ordinate system across the participating devices.

There’s lots of ways of establishing a shared co-ordinate system across MR devices with perhaps the usual one being to use a spatial anchor.

When I originally wrote about shared holograms with PUN in this blog post, I ended up writing a bunch of code to share spatial anchors using Azure blob storage because PUN doesn’t make it so easy to pass around large binary objects.

Here in 2019, though, we’ve got new options and Azure Spatial Anchors to help out with the heavy lifting and so I wanted to make use of Azure Spatial Anchors here to establish a common co-ordinate system.

I wrote a little about Azure Spatial Anchors here;

Baby Steps with the Azure Spatial Anchors Service

and so I won’t repeat everything that I said but will, instead, try and keep undertaking doing the minimal to get what I want up and running.

In that blog post, I talked about how I take only these pieces (5 files) from the official SDK sample here;

But I wanted a simpler wrapper class to make this “easier” for my code to work with and so I wrote the class below. Note that this is most definitely making some very simplistic trade-offs in terms of sacrificing failure handling and error conditions for simplicity and especially in a couple of places;

  • By not following the guidance and providing callbacks here such that an app using this class can provide UI to tell the user what to do to improve the capture of an anchor
  • By potentially having loops that might execute for ever – I’m being very optimistic here.

With those caveats in place, here’s my little helper class;

using Microsoft.Azure.SpatialAnchors;
using Microsoft.MixedReality.Toolkit.Utilities;
using System;
using System.Threading.Tasks;
using UnityEngine;
using UnityEngine.XR.WSA;

namespace AzureSpatialAnchors
{
    public class AzureSpatialAnchorService : MonoBehaviour
    {
        [Serializable]
        public class AzureSpatialAnchorServiceProfile
        {
            [SerializeField]
            [Tooltip("The account id from the Azure portal for the Azure Spatial Anchors service")]
            string azureAccountId;
            public string AzureAccountId => this.azureAccountId;

            [SerializeField]
            [Tooltip("The access key from the Azure portal for the Azure Spatial Anchors service (for Key authentication)")]
            string azureServiceKey;
            public string AzureServiceKey => this.azureServiceKey;
        }

        [SerializeField]
        [Tooltip("The configuration for the Azure Spatial Anchors Service")]
        AzureSpatialAnchorServiceProfile profile = new AzureSpatialAnchorServiceProfile();
        public AzureSpatialAnchorServiceProfile Profile => this.profile;

        TaskCompletionSource<CloudSpatialAnchor> taskWaitForAnchorLocation;

        CloudSpatialAnchorSession cloudSpatialAnchorSession;

        public AzureSpatialAnchorService()
        {
        }
        public async Task<string> CreateAnchorOnObjectAsync(GameObject gameObjectForAnchor)
        {
            string anchorId = string.Empty;
            try
            {
                this.StartSession();

                var worldAnchor = gameObjectForAnchor.GetComponent<WorldAnchor>();

                if (worldAnchor == null)
                {
                    throw new ArgumentException("Expected a world anchor on the game object parameter");
                }

                // Note - these next 2 waits are highly dubious as they may never happen so
                // a real world solution would have to do more but I'm trying to be 
                // minimal here
                await new WaitUntil(() => worldAnchor.isLocated);

                // As per previous comment.
                while (true)
                {
                    var status = await this.cloudSpatialAnchorSession.GetSessionStatusAsync();

                    if (status.ReadyForCreateProgress >= 1.0f)
                    {
                        break;
                    }
                    await Task.Delay(250);
                }
                var cloudAnchor = new CloudSpatialAnchor();

                cloudAnchor.LocalAnchor = worldAnchor.GetNativeSpatialAnchorPtr();

                await this.cloudSpatialAnchorSession.CreateAnchorAsync(cloudAnchor);

                anchorId = cloudAnchor?.Identifier;
            }
            catch (Exception ex) // TODO: reasonable exceptions here.
            {
                Debug.Log($"Caught {ex.Message}");
            }
            return (anchorId);
        }
        public async Task<bool> PopulateAnchorOnObjectAsync(string anchorId, GameObject gameObjectForAnchor)
        {
            bool anchorLocated = false;

            try
            {
                this.StartSession();

                this.taskWaitForAnchorLocation = new TaskCompletionSource<CloudSpatialAnchor>();

                var watcher = this.cloudSpatialAnchorSession.CreateWatcher(
                    new AnchorLocateCriteria()
                    {
                        Identifiers = new string[] { anchorId },
                        BypassCache = true,
                        Strategy = LocateStrategy.AnyStrategy,
                        RequestedCategories = AnchorDataCategory.Spatial
                    }
                );

                var cloudAnchor = await this.taskWaitForAnchorLocation.Task;

                anchorLocated = cloudAnchor != null;

                if (anchorLocated)
                {
                    gameObjectForAnchor.GetComponent<WorldAnchor>().SetNativeSpatialAnchorPtr(cloudAnchor.LocalAnchor);
                }
                watcher.Stop();
            }
            catch (Exception ex) // TODO: reasonable exceptions here.
            {
                Debug.Log($"Caught {ex.Message}");
            }
            return (anchorLocated);
        }
        /// <summary>
        /// Start the Azure Spatial Anchor Service session
        /// This must be called before calling create, populate or delete methods.
        /// </summary>
        public void StartSession()
        {
            if (this.cloudSpatialAnchorSession == null)
            {
                Debug.Assert(this.cloudSpatialAnchorSession == null);

                this.ThrowOnBadAuthConfiguration();
                // setup the session
                this.cloudSpatialAnchorSession = new CloudSpatialAnchorSession();
                // set the Azure configuration parameters
                this.cloudSpatialAnchorSession.Configuration.AccountId = this.Profile.AzureAccountId;
                this.cloudSpatialAnchorSession.Configuration.AccountKey = this.Profile.AzureServiceKey;
                // register event handlers
                this.cloudSpatialAnchorSession.Error += this.OnCloudSessionError;
                this.cloudSpatialAnchorSession.AnchorLocated += OnAnchorLocated;
                this.cloudSpatialAnchorSession.LocateAnchorsCompleted += OnLocateAnchorsCompleted;

                // start the session
                this.cloudSpatialAnchorSession.Start();
            }
        }
        /// <summary>
        /// Stop the Azure Spatial Anchor Service session
        /// </summary>
        public void StopSession()
        {
            if (this.cloudSpatialAnchorSession != null)
            {
                // stop session
                this.cloudSpatialAnchorSession.Stop();
                // clear event handlers
                this.cloudSpatialAnchorSession.Error -= this.OnCloudSessionError;
                this.cloudSpatialAnchorSession.AnchorLocated -= OnAnchorLocated;
                this.cloudSpatialAnchorSession.LocateAnchorsCompleted -= OnLocateAnchorsCompleted;
                // cleanup
                this.cloudSpatialAnchorSession.Dispose();
                this.cloudSpatialAnchorSession = null;
            }
        }
        void OnLocateAnchorsCompleted(object sender, LocateAnchorsCompletedEventArgs args)
        {
            Debug.Log("On Locate Anchors Completed");
            Debug.Assert(this.taskWaitForAnchorLocation != null);

            if (!this.taskWaitForAnchorLocation.Task.IsCompleted)
            {
                this.taskWaitForAnchorLocation.TrySetResult(null);
            }
        }
        void OnAnchorLocated(object sender, AnchorLocatedEventArgs args)
        {
            Debug.Log($"On Anchor Located, status is {args.Status} anchor is {args.Anchor?.Identifier}, pointer is {args.Anchor?.LocalAnchor}");
            Debug.Assert(this.taskWaitForAnchorLocation != null);

            this.taskWaitForAnchorLocation.SetResult(args.Anchor);
        }
        void OnCloudSessionError(object sender, SessionErrorEventArgs args)
        {
            Debug.Log($"On Cloud Session Error: {args.ErrorMessage}");
        }
        void ThrowOnBadAuthConfiguration()
        {
            if (string.IsNullOrEmpty(this.Profile.AzureAccountId) ||
                string.IsNullOrEmpty(this.Profile.AzureServiceKey))
            {
                throw new ArgumentNullException("Missing required configuration to connect to service");
            }
        }
    }
}

It’s perhaps worth saying that while I packaged this as a MonoBehaviour here, I have other variants of this code that would package it as a Mixed Reality extension service which would make it available across the entire application rather than to a set of components that happen to be configured on a particular GameObject. In this case, I went with a MonoBehaviour and configured this into my scene as below;

Setting up PUN to Run with a Cloud Session

It’s possible to use PUN either using a local network server or a cloud server but I want things to be simple and with minimal configuration so I decided to run from the cloud.

With that in mind, I visited the PUN portal here;

https://dashboard.photonengine.com/en-US/publiccloud

and used the Create New App button to create a new app of type Photon PUN with a suitable name. I did not fill in the description or Url properties. I called it TestApp and you can see that the portal then gives me a GUID to represent that app as below;

Once the Photon package has imported into Unity, it conveniently pops up a dialog where I can enter this GUID to link the app with that cloud instance;

and that’s all I need to have this up and running.

Getting a Connection

Getting a connection is dead simple. There are, no doubt, a million options that you can use but all I did was to create an empty GameObject in my scene (named Root) and then write a script which inherits from the PUN base class MonoBehaviourPunCallbacks which provides overrides for network events and so I wrote;

    public class PhotonScript : MonoBehaviourPunCallbacks
    {
        void Start()
        {
            PhotonNetwork.ConnectUsingSettings();
        }
        public override void OnConnectedToMaster()
        {
            base.OnConnectedToMaster();
        }
    }

and trying that out in the editor and seeing it run through in the debugger all seemed to be working nicely.

Getting a Room

The “boundary” for communications in PUN seems to be the “room” and, as you might expect, there’s potential for a lot of functionality, capability and configuration around picking rooms and, optionally, using “lobbies” to select these rooms.

For my purposes, I’m going to pretend that none of this matters and it’s ok to just hard-code a room to avoid any of these additional steps.

Consequently, I can write some simple code to create/or join a room once the network connection is made;

    public class PhotonScript : MonoBehaviourPunCallbacks
    {
        void Start()
        {
            PhotonNetwork.ConnectUsingSettings();
        }
        public override void OnConnectedToMaster()
        {
            base.OnConnectedToMaster();

            PhotonNetwork.JoinOrCreateRoom("HardCodedRoom", null, null);
        }
        public override void OnJoinedRoom()
        {
            base.OnJoinedRoom();
        }
        public override void OnCreatedRoom()
        {
            base.OnCreatedRoom();
        }
    }

and, again, in the debugger attached to the editor I can see both the OnCreatedRoom and OnJoinedRoom overrides being called so things seem fine.

If I run the code again, I see that the room is once again created and joined and this comes down to the ‘time to live’ specified for the RoomOptions for the room. At the moment, my code does not pass any RoomOptions and so the room seems to get torn down pretty quickly whereas I could leave the room ‘alive’ for longer if I changed that value

There are also options on the room around whether objects that are created by a particular player are to be removed when that player leaves the room and around how much time needs to pass before a player is considered to have left the room. For my purposes, I’m not too worried about those details just yet so I tidied up my script and simply set the time to live value on the room itself such that it would give me enough time to join the same room more than once from a single device if I needed to;

    public class PhotonScript : MonoBehaviourPunCallbacks
    {
        enum RoomStatus
        {
            None,
            CreatedRoom,
            JoinedRoom
        }

        public int emptyRoomTimeToLiveSeconds = 120;

        RoomStatus roomStatus = RoomStatus.None;

        void Start()
        {
            PhotonNetwork.ConnectUsingSettings();
        }
        public override void OnConnectedToMaster()
        {
            base.OnConnectedToMaster();

            var roomOptions = new RoomOptions();
            roomOptions.EmptyRoomTtl = this.emptyRoomTimeToLiveSeconds * 1000;
            PhotonNetwork.JoinOrCreateRoom(ROOM_NAME, roomOptions, null);
        }
        public async override void OnJoinedRoom()
        {
            base.OnJoinedRoom();

            if (this.roomStatus == RoomStatus.None)
            {
                this.roomStatus = RoomStatus.JoinedRoom;
            }
        }
        public async override void OnCreatedRoom()
        {
            base.OnCreatedRoom();

            this.roomStatus = RoomStatus.CreatedRoom;
        }
        static readonly string ROOM_NAME = "HardCodedRoomName";
    }

Establishing a Common Co-ordinate System

The heart of a shared holographic experience revolves around using some mechanism to establish a co-ordinate system that’s common across all the devices that are participating in the experience.

I want to do this using world anchors on HoloLens and in conjunction with Azure Spatial Anchors in the cloud which is going to provide me with a mechanism to share the anchor from one device to another.

To keep things simple, I’m going to try and establish a 1:1 relationship between 1 anchor and 1 room in Photon. This probably isn’t realistic for a real-world application but it’s more than enough for my sample here.

The way that I want things to work is as below;

  • When a user creates the Photon room, it will be assumed that user should also create the spatial anchor and post it to the Azure Spatial Anchors (ASA) service.
  • When a user joins a room, it will be assumed that the user should attempt to find the anchor at the ASA service and import it if it exists and, otherwise, wait to be notified that changes have occurred and it should try that process again.

The object that I want to anchor in my scene is the Root object and I have added a WorldAnchor to it as below;

In terms of Photon, the mechanism that I chose to use to try and implement this was to use custom room properties as described here.

My process then becomes;

  • For a user who creates a room, ensure that the WorldAnchor on the Root object isLocated and then use the ASA pieces to create an Azure Spatial Anchor from this getting the ID and storing it in a custom room property named anchorId
  • For a user who joins a room they check the room properties to look for a property named anchorId
    • If present, use ASA to download the anchor and import it to the WorldAnchor on the Root object
    • If not present, assume that we are too early in the process and wait for Photon to call OnUpdateRoomProperties, letting us know that the anchorId property has now been set by the user who created the room & we can now access the value, call the ASA service & get the anchor.

and so my script ended up looking as below;

using System;
using System.Threading.Tasks;
using AzureSpatialAnchors;
using ExitGames.Client.Photon;
using Photon.Pun;
using Photon.Realtime;

public class PhotonScript : MonoBehaviourPunCallbacks
{
    enum RoomStatus
    {
        None,
        CreatedRoom,
        JoinedRoom,
        JoinedRoomDownloadedAnchor
    }

    public int emptyRoomTimeToLiveSeconds = 120;

    RoomStatus roomStatus = RoomStatus.None;

    void Start()
    {
        PhotonNetwork.ConnectUsingSettings();
    }
    public override void OnConnectedToMaster()
    {
        base.OnConnectedToMaster();

        var roomOptions = new RoomOptions();
        roomOptions.EmptyRoomTtl = this.emptyRoomTimeToLiveSeconds * 1000;
        PhotonNetwork.JoinOrCreateRoom(ROOM_NAME, roomOptions, null);
    }
    public async override void OnJoinedRoom()
    {
        base.OnJoinedRoom();

        // Note that the creator of the room also joins the room...
        if (this.roomStatus == RoomStatus.None)
        {
            this.roomStatus = RoomStatus.JoinedRoom;
        }
        await this.PopulateAnchorAsync();
    }
    public async override void OnCreatedRoom()
    {
        base.OnCreatedRoom();
        this.roomStatus = RoomStatus.CreatedRoom;
        await this.CreateAnchorAsync();
    }
    async Task CreateAnchorAsync()
    {
        // If we created the room then we will attempt to create an anchor for the parent
        // of the cubes that we are creating.
        var anchorService = this.GetComponent<AzureSpatialAnchorService>();

        var anchorId = await anchorService.CreateAnchorOnObjectAsync(this.gameObject);

        // Put this ID into a custom property so that other devices joining the
        // room can get hold of it.
        PhotonNetwork.CurrentRoom.SetCustomProperties(
            new Hashtable()
            {
                { ANCHOR_ID_CUSTOM_PROPERTY, anchorId }
            }
        );
    }
    async Task PopulateAnchorAsync()
    {
        if (this.roomStatus == RoomStatus.JoinedRoom)
        {
            object keyValue = null;

            // First time around, this property may not be here so we see if is there.
            if (PhotonNetwork.CurrentRoom.CustomProperties.TryGetValue(
                ANCHOR_ID_CUSTOM_PROPERTY, out keyValue))
            {
                // If the anchorId property is present then we will try and get the
                // anchor but only once so change the status.
                this.roomStatus = RoomStatus.JoinedRoomDownloadedAnchor;

                // If we didn't create the room then we want to try and get the anchor
                // from the cloud and apply it.
                var anchorService = this.GetComponent<AzureSpatialAnchorService>();

                await anchorService.PopulateAnchorOnObjectAsync(
                    (string)keyValue, this.gameObject);
            }
        }
    }
    public async override void OnRoomPropertiesUpdate(Hashtable propertiesThatChanged)
    {
        base.OnRoomPropertiesUpdate(propertiesThatChanged);

        await this.PopulateAnchorAsync();
    }
    static readonly string ANCHOR_ID_CUSTOM_PROPERTY = "anchorId";
    static readonly string ROOM_NAME = "HardCodedRoomName";
}

At this point, in order to try and “test” whether this worked or not I added a simple elongated cube under my Root object in the scene;

with the aim being to try this out on a single HoloLens device by performing;

  1. Run the application standing in a particular position and orientation to establish an origin.
  2. Wait a little while for the anchor to get created and sync’d to ASA.
  3. Close the application.
  4. Step a few metres to one side.
  5. Re-run the application within 2 minutes to attempt to join the same room.
  6. Wait a little for the anchor to get download and located.
  7. Expect that the blue bar will ‘jump’ to the position that it had at step 1 above.

and that worked out fine on the first run of that project and so I have the “core” of a shared holographic experience up and running on Photon in that I can establish a common co-ordinate system across multiple devices with very little code indeed.

The next step (in the next blog post) would be to see if I can create some holograms and move them around…

2 thoughts on “Simple Shared Holograms with Photon Networking (Part 1)

Comments are closed.