Baby Steps with the Mixed Reality Portal & Simulator on Windows 10 Creators Update

NB: The usual blog disclaimer for this site applies to posts around HoloLens. I am not on the HoloLens team. I have no details on HoloLens other than what is on the public web and so what I post here is just from my own experience experimenting with pieces that are publicly available and you should always check out the official developer site for the product documentation.

Creators Update and the Windows Mixed Reality Portal/Simulator

I’ve been trying out the Creators Update (get it here) on an older Surface Pro 3 for quite a while but I haven’t given it a tonne of focus just yet but with the release I thought it was time to upgrade both my home PCs and my work PCs and I did that in a few spare hours the other day.

One of the things that the Creators Update brings with it is the Windows Mixed Reality Portal and that portal also comes with the Windows Mixed Reality Simulator.

I’ve got very familiar with running an app like the ‘Holograms’ app on my HoloLens and using it to position objects in the real world – here’s a screenshot of the Spaceman floating in mid-air inside a huge atrium at Manchester Metropolitan University the other month;

C5VkK6GWAAEWLj2

While I don’t have an immersive Windows Mixed Reality headset available to me today, the Creators Update makes it possible for me to make use of the Holograms app in an entirely virtual world shown to me via the Windows Mixed Reality Portal. So, if I run up the portal on my device and set it up in Developer Mode for Simulation as detailed in this document then I can use the simulator to see the Spaceman in a whole new ‘world’;

image

So now I have my HoloLens which blends virtual objects into my real world and I have the Mixed Reality Simulator which puts virtual objects into a virtual world projected onto a flat 2D screen and they’re running what looks like a very similar Holograms app Smile

Can I Play RoboRaid?

My first thought was ‘Hey, maybe I can now go and play RoboRaid in this environment?’ but I don’t see that listed today in the Store that I access on my device whether from inside or outside of the Windows Mixed Reality Simulator.

That makes sense to me because I could see the RoboRaid developers perhaps needing to make some changes to their app in order to run inside of this environment rather than on HoloLens.

There are, clearly, differences between HoloLens hardware and immersive headset hardware with the grids on this page listing out different features including Gestures, Motion Controllers, Spatial Mapping and so it wouldn’t be surprising if an app like RoboRaid which makes extensive use of spatial mapping and gestures needed some tweaks to run on an immersive headset.

From looking into the Store and the docs, my curiosity was sparked enough to want to have a think about UWP device families, contracts, APIs for detecting headsets and so on and I wrote a few notes around those below based purely on my experiments in the debugger with the public bits. I have no ‘special’ knowledge here and, clearly, if I did then I wouldn’t have to conduct these types of experiments in a debugger to try and figure things out Smile

What Device (Family) am I Running On?

I wondered what device family I was running on when I run an app inside of the Windows Mixed Reality Simulator on my PC. I think I know but I wanted to check.

If I write a basic ‘Hello World’ app that I run on the HoloLens emulator (Windows 14393) and the Mixed Reality Simulator (Windows 15063) choosing my platforms a little carefully;

image

then I can query the device family property (NB: this is rarely the right way to code for device specific code but it’s a cheap/cheerful thing to do here) then I see this from the Mixed Reality Simulator;

image

and this from the HoloLens emulator;

image

which lines up with what I’d expect – in the HoloLens case, the device is its own computer whereas in the other case it’s my PC that’s the computer. This was the output from this line of code;

            this.txtDeviceFamily.Text = AnalyticsInfo.VersionInfo.DeviceFamily;

Where Do My APIs Come From?

From the point of view of the UWP, this would suggest that when I’m running on the Mixed Reality Simulator I should have access to the Universal API Contract and the Desktop API Extensions.

What about on HoloLens? I would have access to the Universal API Contract alone – as far as I’m aware, there isn’t a set of ‘Mixed Reality’ extension APIs for the UWP and when I look into the folder on my machine I see;

image

although I’m not 100% certain that this is an exhaustive list but if I go and check the documentation then I see that APIs like these ones;

https://docs.microsoft.com/en-us/uwp/api/windows.graphics.holographic

reside in the Universal API contract and so have the potential to be called on any device family albeit with the developer understanding that they need to check for support prior to calling those APIs.

How then does a developer make a call to know whether they are running inside of a mixed reality headset environment?

Do I Have a Mixed Reality Headset?

I can check whether I have a mixed reality headset or not by making a call to the APIs IsSupported and IsAvailable on this HolographicSpace class.

The former tells me (from the docs) whether the OS supports headsets and the latter tells me whether one is attached to the PC and ready for rendering (post set-up by the user).

I was a bit puzzled because the docs seemed to suggest that the IsSupported API was present in 10586 and yet Lucian’s super-helpful plugin told me different;

image

and the code crashed on the HoloLens emulator running 14393 so that seemed to suggest that these APIs weren’t part of 14393.

This wouldn’t be a problem for me except that my HoloLens and emulator are on 14393 and so I ended up with code that looked like this for the moment;

        void UpdateHolographicHeadset()
        {
            var text = "Holographic not supported";

            if (Windows.Foundation.Metadata.ApiInformation.IsPropertyPresent(
                "Windows.Graphics.Holographic.HolographicSpace", "IsSupported") &&
                HolographicSpace.IsSupported)
            {
                text =
                    HolographicSpace.IsAvailable ?
                    "Holographic space available" :
                    "Holographic space not available";

                HolographicSpace.IsAvailableChanged += OnHeadsetAvailabilityChanged;
            }
            else if (AnalyticsInfo.VersionInfo.DeviceFamily == "Windows.Holographic")
            {
                text = "Holographic space supported on HoloLens";
            }
            this.txtHeadset.Text = text;
        }
        async void OnHeadsetAvailabilityChanged(object sender, object e)
        {
            await this.Dispatcher.RunAsync(CoreDispatcherPriority.Normal,
                () =>
                {
                    this.UpdateHolographicHeadset();
                }
            );
        }

and that seems to display reasonable results on the HoloLens emulator (14393);

image

and gives the same answer on my PC whether I launch the app inside/outside of the Mixed Reality Simulator;

image

and I could get this flag to change by switching simulation on/off and then running the app on my desktop;

image

although I’m not sure that I saw the app update dynamically from the “not available” –> “available” state when I switched Simulation back on but I’d have to test that again.

So, assuming that I was running everything on 15063 and could simply call these APIs, how would I differentiate between immersive/HoloLens headsets?

Immersive Headset?

There’s another flag to be tested on the HolographicDisplay class called IsOpaque which gives information about the type of display present. I could make a call to this (once again, factoring in the missing API on 14393);

     void UpdateDisplayType()
        {
            var displayType = "no display";

            if (Windows.Foundation.Metadata.ApiInformation.IsTypePresent(
                "Windows.Graphics.Holographic.HolographicDisplay"))
            {
                displayType =
                    (HolographicDisplay.GetDefault()?.IsOpaque == true) ?
                    "opaque lenses" : "transparent lenses";
            }
            else if (AnalyticsInfo.VersionInfo.DeviceFamily == "Windows.Holographic")
            {
                displayType = "transparent lenses";
            }
            this.txtDisplayType.Text = displayType;
        }

and on the Mixed Reality Simulator I see;

image

and I get the expected result on the HoloLens emulator;

image

but this could be a little confusing because I don’t actually have an immersive headset and I’m just running this app on a flat screen rather than with stereoscopic projection but the IsStereo flag can help me with that too;

        void UpdateStereoDisplay()
        {
            var stereoType = "unknown display";

            if (Windows.Foundation.Metadata.ApiInformation.IsTypePresent(
                "Windows.Graphics.Holographic.HolographicDisplay"))
            {
                stereoType = HolographicDisplay.GetDefault()?.IsStereo == true ?
                    "stereoscopic display" : "2D display";
            }
            this.txtStereoDisplay.Text = stereoType;
        }

but I don’t yet understand the output here because I somewhat expected the app inside of the simulator to say “2D” and it doesn’t seem to do that;

image

and it doesn’t when I run it purely on the desktop either;

image

so perhaps I need to think on that one a little bit more.

A Unity Example?

I wanted to see how this worked with a simple Unity example and so I made a ‘blank’ project with the HoloToolkit and just had a red cube and a wall and the idea is that when the user clicks on the cube, the red cube goes green.

I have this scene;

image

and then a script on the cube tries to handle a click to change the cube to be green and it also tries to hide the wall on headsets with transparent lenses;

using HoloToolkit.Unity.InputModule;
using UnityEngine;
using Windows.Graphics.Holographic;
using Windows.System.Profile;

public class CubeScript : MonoBehaviour, IInputClickHandler
{
    private void Awake()
    {
        var wall = GameObject.Find("Wall");

        if (wall != null)
        {
#if UNITY_UWP && !UNITY_EDITOR
            var opaque = true;

            if (Windows.Foundation.Metadata.ApiInformation.IsTypePresent(
                "Windows.Graphics.Holographic.HolographicDisplay"))
            {
                opaque = (HolographicDisplay.GetDefault()?.IsOpaque == true);
            }
            else if (AnalyticsInfo.VersionInfo.DeviceFamily == "Windows.Holographic")
            {
                opaque = false;
            }
            wall.SetActive(opaque);
#endif // UNITY_UWP
        }
    }
    public void OnInputClicked(InputClickedEventData eventData)
    {
        this.gameObject.GetComponent<Renderer>().material.color = Color.green;
    }
}

This seemed to work out fine on the HoloLens emulator (with the app manifest requesting a minimum platform of 14393, maximum of 15063 and targeting the Universal family);

image

with no wall which is what I was hoping for.

If I then deployed this to my local machine and ran it up inside of the Mixed Reality Portal using the simulator then I saw;

MR

and so I get the wall and the cube running there although it is worth saying that I did experience some glitches (drivers perhaps?) around getting this to display in that sometimes it would display but on a few occasions I noticed the Mixed Reality Portal seemed to reset itself before I got to the Unity splash screen for my app.

Naturally, I’m sure there are better ways of doing what I’m doing here but I felt like I learned a few things and so thought I’d share as part of experimentation…it’s exciting to see these bits in the Creators Update and it’ll be even more exciting to get my hands on a headset and see what the experience is like there.

Don’t forget that you can sign up for news on those headsets here.

Hitchhiking the HoloToolkit-Unity, Leg 13–Continuing with Shared Experiences

NB: The usual blog disclaimer for this site applies to posts around HoloLens. I am not on the HoloLens team. I have no details on HoloLens other than what is on the public web and so what I post here is just from my own experience experimenting with pieces that are publicly available and you should always check out the official developer site for the product documentation.

Continuing with what I’d published in this previous post, I wanted to add some more functionality to the shared experience that I’ve been experimenting with and so I’m going to take this (and the codebase) further in this post.

The first thing that I wanted to add was the ability to download the model that’s going to be viewed from a web server rather than have it hard-baked into the application’s binary.

The ‘Unity’ way of doing this seems to be to use asset bundles and so, in order to achieve this, I combined what I’d learned in this post into my code base such that my project now contains the code that’s needed to both;

  1. Build asset bundles
  2. Download asset bundles at runtime

Note that (as per that post) this involves me changing Unity’s demo scripts somewhat to work with HoloLens (only tiny changes as it happens). Wherever I’ve done that, I’ve used a #if MIKET_CHANGE to flag what I’ve been doing to someone else’s script.

This code now lives here in my project;

image

and I have both the scripts for runtime and the scripts for the Editor so that I can build the asset bundle for the model from within my project.

Loading the Model from an Asset Bundle at Startup

With those scripts added, I needed to do a little work to get this dynamic model loading implemented and so I made a prefab out of the model, took it out of my scene and made it into an asset bundle;

image

and then built that bundle;

image

and deployed it to my test web server in Azure;

image

I made a small structural change in that I moved my ‘Coordinator’ script which handles the essential ‘flow’ of the app so that it was a component of the SharedObjects game object rather than a stand alone script;

image

and I added a new script here which I called ‘Bundle Downloader’ which is intended to the bare bones of loading up an asset from a bundle on a web server, leaning very heavily on my earlier post and the code that Unity ships as part of their demo project and with some very basic support around providing a fallback prefab for the case where the download doesn’t succeed or the developer has simply switched it off in the editor;

using AssetBundles;
using HoloToolkit.Unity;
using System;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class BundleDownloadedEventArgs : EventArgs
{
  public bool DownloadSucceeded { get; set; }
}

public class BundleDownloader : Singleton<BundleDownloader>
{
  [SerializeField]
  string downloadUrl;

  [SerializeField]
  string bundleName;

  [SerializeField]
  string prefabName;

  [SerializeField]
  GameObject fallbackPrefab;

  [SerializeField]
  bool isActive = true;

  public GameObject LoadedPrefab
  {
    get; set;
  }

  public event EventHandler<BundleDownloadedEventArgs> Downloaded;

  public void StartAsyncDownload()
  {
    StartCoroutine(this.DownloadAsync());
  }
  IEnumerator DownloadAsync()
  {
    var prefabObject = this.fallbackPrefab;
    var succeeded = false;

#if !UNITY_EDITOR

    if (this.isActive &&
      !string.IsNullOrEmpty(this.downloadUrl) &&
      !string.IsNullOrEmpty(this.bundleName) &&
      !string.IsNullOrEmpty(this.prefabName))
    {
      AssetBundleManager.SetSourceAssetBundleURL(this.downloadUrl);

      var initializeOperation = AssetBundleManager.Initialize();

      if (initializeOperation != null)
      {
        yield return StartCoroutine(initializeOperation);

        AssetBundleLoadAssetOperation loadOperation = null;

        try
        {
          loadOperation = AssetBundleManager.LoadAssetAsync(
            this.bundleName, this.prefabName, typeof(GameObject));
        }
        catch
        {

        }
        if (loadOperation != null)
        {
          yield return StartCoroutine(loadOperation);

          var loadedPrefab = loadOperation.GetAsset<GameObject>();

          if (loadedPrefab != null)
          {
            prefabObject = loadedPrefab;
            succeeded = true;
          }
        }
      }
    }
#else
    succeeded = true;
#endif

    this.LoadedPrefab = prefabObject;

    if (this.Downloaded != null)
    {
      this.Downloaded(
        this, new global::BundleDownloadedEventArgs()
        {
          DownloadSucceeded = succeeded
        }
      );
    }
    yield break;
  }
}

and I set that up in the Unity editor such that is properties were set to point to the right URLs etc;

image

and I then modified my Coordinator script such that it had two new status values which are represented at the top of this enum here and I also modified the display text that is displayed on start up;

 enum CurrentStatus
  {
    Start,
    WaitingForModelToLoad,
    WaitingToConnectToStage,
    WaitingForRoomApiToStabilise,
    WaitingForModelPositioning,
    WaitingForWorldAnchorExport,
    WaitingForWorldAnchorImport
  }
  void Start()
  {
    StatusTextDisplay.Instance.SetStatusText("waiting for model to load");
  }

and then added a case to my switch statement to check for that Start status and move the app on to the ‘waiting for network’ status;

        case CurrentStatus.Start:
          this.MoveToStatus(CurrentStatus.WaitingForModelToLoad);

          Debug.Log("Coordinator: starting to load model from web server");
          StatusTextDisplay.Instance.SetStatusText("loading model from web server");

          this.GetComponent<BundleDownloader>().Downloaded += this.OnModelDownloaded;
          this.GetComponent<BundleDownloader>().StartAsyncDownload();
          break;

and so this code now gets hold of the new BundleDownloader, asks it to download the model from the web server and when that completes we get this Downloaded event which we handle by adding;

  void OnModelDownloaded(object sender, BundleDownloadedEventArgs e)
  {
    var bundleDownloader = this.GetComponent<BundleDownloader>();

    bundleDownloader.Downloaded -= this.OnModelDownloaded;

    Debug.Log(
      string.Format(
        "Coordinator: download of model from web server has completed and {0}",
        e.DownloadSucceeded ? "succeeded" : "failed or wasn't tried"));

    StatusTextDisplay.Instance.SetStatusText(
      string.Format(
        "{0} model from web server",
        e.DownloadSucceeded ? "loaded" : "failed to load"));

    // Create the model and parent it off this object.
    this.model = Instantiate(bundleDownloader.LoadedPrefab);
    this.model.transform.parent = this.modelParent.transform;

    // Move the world locked parent so that it's in a 'reasonable'
    // place to start with
    this.modelParent.transform.SetPositionAndRotation(
      WORLD_LOCKED_STARTING_POSITION, Quaternion.identity);

    Debug.Log(
      string.Format(
        "Coordinator: waiting for network connection",
        e.DownloadSucceeded ? "succeeded" : "failed or wasn't tried"));

    StatusTextDisplay.Instance.SetStatusText("connecting to room server");

    this.MoveToStatus(CurrentStatus.WaitingToConnectToStage);
  }

with that in place, I now have the same app as previously but with the addition that this version dynamically loads its model from a web server – as in the screenshot below;

image

followed by the model arriving from the web;

image

Adding More to the Remote Head Manager (the T-shirt)

In the existing code, I have the idea that users can be in different rooms based on the name of their WiFi network.

The flow is something like;

  • A room is created if there is not a room on the server for the WiFi network name.
  • The first user into the room positions the model.
  • Subsequent users into the room see the model already positioned by the first user.
  • Users across all rooms can see the HoloLens position and movement of other users regardless of room.

This was working in the previous blog post but I wanted a better representation of a HoloLens user that was remote and so I found a simple model of a T-shirt out on the web;

image

and then I modified the Remote Head Manager script primarily to add another GameObject for the body to be displayed set to be this T-shirt below;

image

and I modified the script in order to support both a head and a body object for a user that is not determined to be in the same room as the user running the app. I’ve included the entire script here but it’s perhaps better to check the github repo for the details;

#define MIKET_CHANGE
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License. See LICENSE in the project root for license information.

using System;
using System.Collections.Generic;
using UnityEngine;
using HoloToolkit.Unity;
using HoloToolkit.Unity.InputModule;

namespace HoloToolkit.Sharing.Tests
{
  /// <summary>
  /// Broadcasts the head transform of the local user to other users in the session,
  /// and adds and updates the head transforms of remote users.
  /// Head transforms are sent and received in the local coordinate space of the GameObject this component is on.
  /// </summary>
  public class RemoteHeadManager : Singleton<RemoteHeadManager>
  {
    public class RemoteHeadInfo
    {
      public long UserID;
      public GameObject HeadObject;
#if MIKET_CHANGE
      public GameObject BodyObject;
#endif     
    }

#if MIKET_CHANGE
    public GameObject remoteHeadPrefab;
    public GameObject remoteBodyPrefab;
#endif

    /// <summary>
    /// Keep a list of the remote heads, indexed by XTools userID
    /// </summary>
    private Dictionary<long, RemoteHeadInfo> remoteHeads = new Dictionary<long, RemoteHeadInfo>();

#if MIKET_CHANGE
    private void OnEnable()
    {
      this.roomId = -1;

      CustomMessages.Instance.MessageHandlers[CustomMessages.TestMessageID.HeadTransform] =
        UpdateHeadTransform;

      SharingStage.Instance.SessionUsersTracker.UserJoined += UserJoinedSession;
      SharingStage.Instance.SessionUsersTracker.UserLeft += UserLeftSession;
    }
#else
    private void Start()
    {
      CustomMessages.Instance.MessageHandlers[CustomMessages.TestMessageID.HeadTransform] = UpdateHeadTransform;

      // SharingStage should be valid at this point, but we may not be connected.
      if (SharingStage.Instance.IsConnected)
      {
        Connected();
      }
      else
      {
        SharingStage.Instance.SharingManagerConnected += Connected;
      }
    }
    private void Connected(object sender = null, EventArgs e = null)
    {
      SharingStage.Instance.SharingManagerConnected -= Connected;

      SharingStage.Instance.SessionUsersTracker.UserJoined += UserJoinedSession;
      SharingStage.Instance.SessionUsersTracker.UserLeft += UserLeftSession;
    }
#endif

    private void Update()
    {
#if MIKET_CHANGE
      this.DetermineCurrentRoom();
#endif
      // Grab the current head transform and broadcast it to all the other users in the session
      Transform headTransform = Camera.main.transform;

      // Transform the head position and rotation from world space into local space
      Vector3 headPosition = transform.InverseTransformPoint(headTransform.position);

      Quaternion headRotation = Quaternion.Inverse(transform.rotation) * headTransform.rotation;

#if MIKET_CHANGE
      CustomMessages.Instance.SendHeadTransform(headPosition, headRotation,
         this.roomId);
#endif
    }
#if MIKET_CHANGE
    void DetermineCurrentRoom()
    {
      if (this.roomId == -1)
      {
        var roomManager = SharingStage.Instance.Manager.GetRoomManager();

        if (roomManager != null)
        {
          var room = roomManager.GetCurrentRoom();
          this.roomId = room.GetID();
        }
      }
    }
#endif

    protected override void OnDestroy()
    {
      if (SharingStage.Instance != null)
      {
        if (SharingStage.Instance.SessionUsersTracker != null)
        {
          SharingStage.Instance.SessionUsersTracker.UserJoined -= UserJoinedSession;
          SharingStage.Instance.SessionUsersTracker.UserLeft -= UserLeftSession;
        }
      }

      base.OnDestroy();
    }

    /// <summary>
    /// Called when a new user is leaving the current session.
    /// </summary>
    /// <param name="user">User that left the current session.</param>
    private void UserLeftSession(User user)
    {
      int userId = user.GetID();
      if (userId != SharingStage.Instance.Manager.GetLocalUser().GetID())
      {
        RemoveRemoteHead(remoteHeads[userId].HeadObject);
        remoteHeads.Remove(userId);
      }
    }

    /// <summary>
    /// Called when a user is joining the current session.
    /// </summary>
    /// <param name="user">User that joined the current session.</param>
    private void UserJoinedSession(User user)
    {
      if (user.GetID() != SharingStage.Instance.Manager.GetLocalUser().GetID())
      {
        GetRemoteHeadInfo(user.GetID());
      }
    }

    /// <summary>
    /// Gets the data structure for the remote users' head position.
    /// </summary>
    /// <param name="userId">User ID for which the remote head info should be obtained.</param>
    /// <returns>RemoteHeadInfo for the specified user.</returns>
    public RemoteHeadInfo GetRemoteHeadInfo(long userId)
    {
      RemoteHeadInfo headInfo;

      // Get the head info if its already in the list, otherwise add it
      if (!remoteHeads.TryGetValue(userId, out headInfo))
      {
        headInfo = new RemoteHeadInfo();
        headInfo.UserID = userId;
        headInfo.HeadObject = CreateRemoteHead();

#if MIKET_CHANGE
        headInfo.BodyObject = Instantiate(this.remoteBodyPrefab);
        headInfo.BodyObject.transform.parent = this.gameObject.transform;
#endif
        remoteHeads.Add(userId, headInfo);
      }

      return headInfo;
    }

    /// <summary>
    /// Called when a remote user sends a head transform.
    /// </summary>
    /// <param name="msg"></param>
    private void UpdateHeadTransform(NetworkInMessage msg)
    {
      // Parse the message
      long userID = msg.ReadInt64();

      Vector3 headPos = CustomMessages.Instance.ReadVector3(msg);

      Quaternion headRot = CustomMessages.Instance.ReadQuaternion(msg);

#if MIKET_CHANGE
      long remoteRoomId = msg.ReadInt64();
#endif

      RemoteHeadInfo headInfo = GetRemoteHeadInfo(userID);
      headInfo.HeadObject.transform.localPosition = headPos;
      headInfo.HeadObject.transform.localRotation = headRot;

#if MIKET_CHANGE
      var rayLength = maxRayDistance;

      RaycastHit hitInfo;

      if (Physics.Raycast(
        headInfo.HeadObject.transform.position,
        headInfo.HeadObject.transform.forward,
        out hitInfo))
      {
        rayLength = hitInfo.distance;
      }
      var lineRenderer = headInfo.HeadObject.GetComponent<LineRenderer>();
      lineRenderer.SetPosition(1, Vector3.forward * rayLength);

      if ((remoteRoomId == -1) || (this.roomId == -1) ||
        (remoteRoomId != this.roomId))
      {
        headInfo.BodyObject.SetActive(true);
        headInfo.BodyObject.transform.localPosition = headPos;
        headInfo.BodyObject.transform.localRotation = headRot;
      }
      else
      {
        headInfo.BodyObject.SetActive(false);
      }
#endif
    }

    /// <summary>
    /// Creates a new game object to represent the user's head.
    /// </summary>
    /// <returns></returns>
    private GameObject CreateRemoteHead()
    {
      GameObject newHeadObj = Instantiate(this.remoteHeadPrefab);
      newHeadObj.transform.parent = gameObject.transform;

#if MIKET_CHANGE
      this.AddLineRenderer(newHeadObj);
#endif
      return newHeadObj;
    }
#if MIKET_CHANGE
    void AddLineRenderer(GameObject headObject)
    {
      var lineRenderer = headObject.AddComponent<LineRenderer>();
      lineRenderer.useWorldSpace = false;
      lineRenderer.startWidth = 0.01f;
      lineRenderer.endWidth = 0.05f;
      lineRenderer.positionCount = 2;
      lineRenderer.SetPosition(0, Vector3.forward * 0.1f);
      var material = new Material(Shader.Find("Diffuse"));
      material.color = colors[this.colorIndex++ % colors.Length];

      lineRenderer.material = material;
    }
#endif

    /// <summary>
    /// When a user has left the session this will cleanup their
    /// head data.
    /// </summary>
    /// <param name="remoteHeadObject"></param>
    private void RemoveRemoteHead(GameObject remoteHeadObject)
    {
      DestroyImmediate(remoteHeadObject);
    }
#if MIKET_CHANGE
    long roomId;
    const float maxRayDistance = 5.0f;
    int colorIndex;
    static Color[] colors =
    {
      Color.red,
      Color.green,
      Color.blue,
      Color.cyan,
      Color.magenta,
      Color.yellow
    };
#endif
  }
}

Most of the modifications here are to try and figure out whether the user is present in the same room (i.e. WiFi network) as the current user – this seems to work ok but I’d not be surprised to see bugs and the way that I’m doing it currently means that you can certainly see a T-shirt for a local user until the point where the code realises that it really is a local user.

The net effect of all this looks something like the picture below and, currently, I move and rotate the T-shirt just as much as the head and that’s an area that could definitely be refined;

image

Moving the Model After It Has Been Positioned

Previously, the app has allowed one user to position a model in a room, say the word “lock” and then anchor that model such that another user can view it at the same place in the same room or remotely from another room.

What I haven’t previously made possible was the idea that the users might be able to continue to manipulate the model after it has been placed into the room.

In order to progress this, I need to modify the current code base because right now the ModelParent object takes on two roles;

image

The first role is that this object is the one which ultimately becomes world anchored to provide a common parent for users to view the same content.

The second role is that this object is the one with the “User Moveable” script on it allowing the user to move it around and then say “lock” to world anchor it.

These two roles are compatible in a situation where I no longer want to move the model once it has been “locked”. Hover, if the model is to move after that locking process then I need to make changes because the world anchor stops it responding to movement.

One way around this is to dynamically add/remove the world anchor but I preferred what seemed like a simpler way which is to leave this code alone and dynamically to add a “User Moveable” behaviour to the model as it is created and so I modified my coordinator script to do this.

I have code that runs [as/when] the world anchor is [imported/exported] from the server and so it seemed sensible to dynamically add that “User Moveable” behaviour at this point;

  void OnImportOrExportCompleted(bool succeeded)
  {
    StatusTextDisplay.Instance.SetStatusText("room in sync");

    // Allow the child to be moved now that it is positioned in
    // the right place.
    this.model.AddComponent<UserMoveable>();

    // Switch on the remote head management.
    this.modelParent.GetComponent<RemoteHeadManager>().enabled = true;
  }

and so now I have solved perhaps “half” the problem in that the model can be moved once it has been locked.

However, those movements won’t be reflected on any of the other devices in the room. That needs more of a change…

Baby Steps Into the SyncModel

To synchronize the movement of the model, I need a component which will monitor its transform, note any changes and then send them over the network to other devices to be applied locally.

I’ve seen this sort of behaviour in the SyncSpawnManager and PrefabSpawnManager pieces before which I wrote a little about in this post and, from unpicking that code, I know that the HoloToolkit-Unity has a component named TransformSynchronizer which does exactly this work and that it works with the SyncTransform class to do this;

image

Here, the TransformSynchronizer does the monitoring of the data held by the underlying SyncTransform which the code refers to as ‘the data model’. As such, it derives from SyncObject which lives within the whole SyncModel part of the toolkit;

image

At the time of writing, I’m not sure that I entirely understand how all of this ‘synchronized data model’ part of the sharing toolkit works to the ultimate level of detail but I think that I now know that the SharingStage component which is used to group up the networking functionality has a root object;

image

and that’s of type SyncRoot and it’s the data (broken down into SyncObject derived types) which is to be sync’d across devices with change notification support as you’d expect.

I noticed that the Prefab Spawn Manager has some data in this SyncRoot which it uses to synchronize the list of objects that have been spawned across devices via its various routines;

image

In this particular project, I’m not using the Prefab Spawn Manager because I am not dynamically adding objects into the scene once they are loaded from the web server.

However, I figure that I can use the same mechanism that the Prefab Spawn Manager uses in order to try and keep the transformations on my model in sync and so I added a property here;

#define MIKET_CHANGE
//
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License. See LICENSE in the project root for license information.
//

using HoloToolkit.Sharing.Spawning;
using HoloToolkit.Sharing.SyncModel;

namespace HoloToolkit.Sharing
{
  /// <summary>
  /// Root of the synchronization data model used by this application.
  /// </summary>
  public class SyncRoot : SyncObject
  {
#if MIKET_CHANGE
    [SyncData]
    public SyncSpawnedObject ModelObject;
#endif

    /// <summary>
    /// Children of the root.
    /// </summary>
    [SyncData]
    public SyncArray<SyncSpawnedObject> InstantiatedPrefabs;

    /// <summary>
    /// Constructor.
    /// </summary>
    /// <param name="rootElement">Root Element from Sharing Stage</param>
    public SyncRoot(ObjectElement rootElement)
    {
      Element = rootElement;
      FieldName = Element.GetName().GetString();
      InitializeSyncSettings();
      InitializeDataModel();
    }

    private void InitializeSyncSettings()
    {
      SyncSettings.Instance.Initialize();
    }

    /// <summary>
    /// Initializes any data models that need to have a local state.
    /// </summary>
    private void InitializeDataModel()
    {
      InstantiatedPrefabs.InitializeLocal(Element);

#if MIKET_CHANGE
      this.ModelObject.InitializeLocal(Element);
#endif
    }
  }
}

and I wanted to see if I could then make use of this to synchronize the model after it had already been created.

I modified the code in my Coordinator script so as to make sure that this was set up appropriately at the point when the world anchor’d object was either exported or imported from/into the current device. That changed the existing method that dealt with this to;

 void OnImportOrExportCompleted(bool succeeded)
  {
    StatusTextDisplay.Instance.SetStatusText("room in sync");

    // Allow the child to be moved now that it is positioned in
    // the right place.
    this.model.AddComponent<UserMoveable>();

    var dataModel = SharingStage.Instance.Root.ModelObject;
    dataModel.GameObject = this.model.gameObject;
    dataModel.Initialize(this.model.gameObject.name, this.model.transform.GetFullPath());
    dataModel.Transform.Position.Value = this.model.transform.localPosition;
    dataModel.Transform.Rotation.Value = this.model.transform.localRotation;
    dataModel.Transform.Scale.Value = this.model.transform.localScale;

    var synchronizer = this.model.EnsureComponent<TransformSynchronizer>();
    synchronizer.TransformDataModel = dataModel.Transform;

    // Switch on the remote head management.
    this.modelParent.GetComponent<RemoteHeadManager>().enabled = true;
  }

and that seemed to enable the experience that I was looking for as demonstrated in this little test video below;

and so that works reasonably well but it led me to think that there might be another option here where a user might want to move the four cubes of the model as a single group or, alternatively, as four separate objects.

Moving the Parent or the Children

In the solution at this point, I have a model (ultimately served from a web server) and that model is treated as a single thing. There’s a box collider wrapped around it so that it can be ‘hit tested’ and all the cubes behave as a group.

image

But there might be an occasion where a user wanted to treat each individual cube separately as they also have colliders on them. That could be taken further by allowing the user to do this recursively and let them choose levels of objects to manipulate but that might be going “a little far” for this blog post and so I’ve chosen just to think about the direct children of the model and I’ve made an assumption that the model designer will be kind and will have put colliders onto them for my code to hit against.

I don’t necessarily want to lose the original ‘grouped’ mode of operation though and so I figured it might be nice to have that be the default and then add some voice command (‘split’) which switches into the child-focused mode. At the time of writing, I haven’t attempted to implement the idea of reversing this choice as it’s a bit more tricky.

With that in mind, I decided that the way that I would approach this would be to make the model and its direct children all capable of being moved in a trackable way that’s synchronized across devices. If I assume this then the act of switching from ‘group mode’ to ‘child mode’ simply involves taking the collider off the model object in order to let the child colliders ‘come through’.

To make these changes, I first went and changed my SyncRoot again such that I had a SyncArray rather than a single SyncSpawnedObject instance;

#define MIKET_CHANGE
//
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License. See LICENSE in the project root for license information.
//

using HoloToolkit.Sharing.Spawning;
using HoloToolkit.Sharing.SyncModel;

namespace HoloToolkit.Sharing
{
  /// <summary>
  /// Root of the synchronization data model used by this application.
  /// </summary>
  public class SyncRoot : SyncObject
  {
#if MIKET_CHANGE
    [SyncData]
    public SyncArray<SyncSpawnedObject> ModelObjects;
#endif

    /// <summary>
    /// Children of the root.
    /// </summary>
    [SyncData]
    public SyncArray<SyncSpawnedObject> InstantiatedPrefabs;

    /// <summary>
    /// Constructor.
    /// </summary>
    /// <param name="rootElement">Root Element from Sharing Stage</param>
    public SyncRoot(ObjectElement rootElement)
    {
      Element = rootElement;
      FieldName = Element.GetName().GetString();
      InitializeSyncSettings();
      InitializeDataModel();
    }

    private void InitializeSyncSettings()
    {
      SyncSettings.Instance.Initialize();
    }

    /// <summary>
    /// Initializes any data models that need to have a local state.
    /// </summary>
    private void InitializeDataModel()
    {
      InstantiatedPrefabs.InitializeLocal(Element);

#if MIKET_CHANGE
      this.ModelObjects.InitializeLocal(Element);
#endif
    }
  }
}

and then I modified the code which runs when the world anchor is either imported or exported to populate this array in a very similar way to the single-object case. Note that I made the slightly dubious decision to store both the parent model and all of its children in the same array which would likely lead some future maintainer (i.e. me) to make ‘off by 1’ type errors here;

  void OnImportOrExportCompleted(bool succeeded)
  {
    StatusTextDisplay.Instance.SetStatusText(
      string.Format("room import/export {0}", succeeded ? "succeeded" : "failed"));

    if (succeeded)
    {
      var isExporter = this.currentStatus == CurrentStatus.WaitingForWorldAnchorExport;

      StatusTextDisplay.Instance.SetStatusText("room is now in sync");

      // First, make sure the model itself is set up to be moveable in a trackable way.
      // NB: We use index 0 for the model itself.
      this.MakeModelPartMoveableAndTrackable(this.model, 0, isExporter);

      // And all of its children are also moveable.
      var childCount = this.model.transform.childCount;

      // NB: 1 to N because the have the model in slot 0.
      for (int i = 1; i <= childCount; i++)
      {
        // NB: We add 1 because the model itself is in slot 1.
        var child = this.model.transform.GetChild(i - 1);
        this.MakeModelPartMoveableAndTrackable(child.gameObject, i, isExporter);
      }
      // Switch on the remote head management.
      this.modelParent.GetComponent<RemoteHeadManager>().enabled = true;

      // Switch on the keyword recognizer listening for 'join' and 'split'
      this.gameObject.GetComponent<KeywordManager>().StartKeywordRecognizer();
    }
  }
  SyncSpawnedObject MakeModelPartMoveableAndTrackable(
    GameObject objectInstance, int indexIntoRootSyncStore, bool isExporter)
  {
    SyncSpawnedObject dataModel = null;

    if (isExporter)
    {
      dataModel = new SyncSpawnedObject();
      dataModel.GameObject = objectInstance; ;
      dataModel.Initialize(objectInstance.name, objectInstance.transform.GetFullPath());
      dataModel.Transform.Position.Value = objectInstance.transform.localPosition;
      dataModel.Transform.Rotation.Value = objectInstance.transform.localRotation;
      dataModel.Transform.Scale.Value = objectInstance.transform.localScale;

      SharingStage.Instance.Root.ModelObjects.AddObject(dataModel);
    }
    else
    {
      dataModel = 
        SharingStage.Instance.Root.ModelObjects.GetDataArray()[indexIntoRootSyncStore];
    }
    objectInstance.EnsureComponent<UserMoveable>();
    var synchronizer = objectInstance.EnsureComponent<TransformSynchronizer>();
    synchronizer.TransformDataModel = dataModel.Transform;

    return (dataModel);
  }

and so now my code is tracking all of the N objects that live at level 0 (the model) and level 1 (its direct children) in the scene which is loaded from the web server.

With that in place, I just need to add a new KeywordManager component to listen for the word “split”

image

and then I can write the code on my Coordinator script to ensure this takes away any top collider on the model;

  public void OnSplit()
  {
    this.GetComponent<KeywordManager>().enabled = false;
    this.model.GetComponent<Collider>().enabled = false;
  }

and that’s pretty much it for this post.

Testing and Wrapping Up

To test this last piece of ‘splitting up’ code, I had to be a little crafty and make use of HoloLens+Emulator as two separate users as I only had the one HoloLens. I’ll try it with multiple devices at the first opportunity but, for now, here’s a capture of it running across my device and my emulator and it seems to work reasonably well;

What I’d like to do next is to capture how this now looks when I have multiple users in a single room and also multiple users split across rooms represented by the T-shirt.

I’ll come back to that in a follow on post…

The code for all of this is here on github, keep in mind that you’ll need to set up IP addresses or host names to make it work.

Experimenting with Unity’s Asset Bundle Loading in Holographic UWP Projects

NB: The usual blog disclaimer for this site applies to posts around HoloLens. I am not on the HoloLens team. I have no details on HoloLens other than what is on the public web and so what I post here is just from my own experience experimenting with pieces that are publicly available and you should always check out the official developer site for the product documentation.

I wanted to experiment with Asset Bundles from Unity running on HoloLens in the sense that I wanted to make sure that I had the right pieces to dynamically load up something like some 3D models bundled on a web server dynamically at runtime and display them in a scene.

This is, no doubt, a very basic scenario for experienced Unity folks but I hadn’t tried it out on HoloLens so I thought I’d do that and share the write-up here.

Step 1 – Reading the Unity Asset Bundle Docs

I had a decent read of Unity’s docs around asset bundles, serialization graphs and that type of thing all from this set of resources;

A guide to AssetBundles and Resources

and that was pretty useful and led me through to the demo and companion scripts that Unity publishes on this repo;

Asset Bundle Demo Repo

and so I cloned that.

Step 2 – Building Asset Bundles

I then opened up that demo project in Unity and used the Assets menu to make the asset bundles;

image

which dropped the assets out into a folder called Windows;

image

and so I just need to drop these onto a web server somewhere and serve them up. I’ve heard that Azure thing is pretty good at this Winking smile

Step 3 – A Web Server for the Asset Bundles

I popped over to Azure and created a basic web server using the regular ‘Add App Service’ button;

image

and then it turned out that I didn’t actually have the Visual Studio web tools installed (woops, I should fix that) so I just went with the “edit this in the browser” option;

image

and hacked the web.config so as to try and make sure that it would serve up the files in question;

image

and then used the “Upload files” option to upload the files – note that this means that these files live at /AssetBundles/Windows and there’s another file in that folder also called Windows;

image

Step 4 – Trying the Desktop Demo

To see if this was “working” ok, I opened up the AssetLoader scene provided in the demo project, opened up the LoadAssets script associated with the Loader GameObject as below;

image

noting that it is loading the bundle named cube-bundle and the asset named MyCube;

image

and I hacked the accompanying script such that it loaded the asset from my new web site;

image

and, sure enough, I built and ran the desktop app and it displayed the cube as expected;

image

and Fiddler shows me where it’s coming from;

image

and so that’s all good, what about doing this on HoloLens?

Step 5 – Starting with a Blank HoloLens Project

I made a new Unity project much like I do in this video and imported the HoloToolkit-Unity;

Step 6 – Adding in the Asset Bundle Manager

I took some of the scripts from the demo asset bundle project and added them to my project, trying to be as ‘minimal’ as I could be so I brought in;

image

Bringing in the Asset Bundle Manager caused me a slight problem in that there are a couple of functions that don’t build in there on the HoloLens/UWP platform;

    //private static string GetStreamingAssetsPath()
    //{
    //  if (Application.isEditor)
    //    return "file://" + System.Environment.CurrentDirectory.Replace("\\", "/"); // Use the build output folder directly.
    //  else if (Application.isWebPlayer)
    //    return System.IO.Path.GetDirectoryName(Application.absoluteURL).Replace("\\", "/") + "/StreamingAssets";
    //  else if (Application.isMobilePlatform || Application.isConsolePlatform)
    //    return Application.streamingAssetsPath;
    //  else // For standalone player.
    //    return "file://" + Application.streamingAssetsPath;
    //}


    /// <summary>
    /// Sets base downloading URL to a directory relative to the streaming assets directory.
    /// Asset bundles are loaded from a local directory.
    /// </summary>
    public static void SetSourceAssetBundleDirectory(string relativePath)
    {
      throw new NotImplementedException();
      // BaseDownloadingURL = GetStreamingAssetsPath() + relativePath;
    }

one is public and calls the other and nothing seemed to call the public function so I figured that I was safe to comment these out at the moment (as above).

I continued by making a small “UI” with a button on it which would respond to a click;

image

and added some code to respond to the click;

using AssetBundles;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class Placeholder : MonoBehaviour
{
  public void OnClick()
  {
    if (!this.loaded)
    {
      StartCoroutine(this.LoadModelAsync());
      this.loaded = true;
    }
  }
  IEnumerator LoadModelAsync()
  {
    AssetBundleManager.SetSourceAssetBundleURL("http://mttestbundles.azurewebsites.net/AssetBundles/");

    var initializeOperation = AssetBundleManager.Initialize();

    if (initializeOperation == null)
    {
      yield break;
    }
    yield return StartCoroutine(initializeOperation);

    var loadOperation = AssetBundleManager.LoadAssetAsync(
      "cube-bundle", "MyCube", typeof(GameObject));

    if (loadOperation == null)
    {
      yield break;
    }
    yield return StartCoroutine(loadOperation);

    var prefab = loadOperation.GetAsset<GameObject>();

    if (prefab != null)
    {
      var cube = GameObject.Instantiate(prefab);

      // Put the cube somewhere obvious.
      cube.transform.position = new Vector3(0, 1.0f, 2.0f);

      // The cube in the asset bundle is quite big.
      cube.transform.localScale = new Vector3(0.2f, 0.2f, 0.2f);
    }
  }
  bool loaded;
}

This code is really just me taking what seemed like the “essential” code out of the Unity sample and seeing if I could get it running on its own here inside of my HoloLens project.

Not surprisingly, I couldn’t Smile The code crashed.

Step 7 – Adding the Platform Name

Fixing the crash wasn’t too difficult – there are two functions in the supplied Utility.cs code file named GetPlatformForAssetBundles (one for the editor and one for the non-editor) and they are just a big switch statement that turns an enumeration for the build platform into a string. They needed me to add;

image

although I can’t be 100% certain that this is the right thing to do but it seemed like a reasonable place to start.

However, this didn’t work either Smile

Step 8 – The Wrong Build Target

With that change in place, my code no longer crashed but I did some debugging and some watching of the Unity debug spew and ultimately I could see that Unity was taking a look at my asset bundle and rejecting it. That is, it was saying (and I quote Winking smile);

The file can not be loaded because it was created for another build target that is not compatible with this platform.
Please make sure to build AssetBundles using the build target platform that it is used by.
File’s Build target is: 5

Ok – so clearly, here, I’ve built the asset bundles using the demo project supplied and they emit some target identifier of “5” into the output and that “5” is not viewed as being compatible with my current build target in my HoloLens project.

I left the assets in the demo project and simply changed the build target to be Windows Store;

image

as the script which does the building (BuildScript) seemed to be making use of EditorUserBuildSettings.activeBuildTarget so that felt like it might work. Because the build platform was now UWP, I needed to make the same changes to the scripts that I’d made in my own project in order to add the “Windows” name for the WSA… runtime platform enumerations.

With that done, I could click “Build Asset Bundles…” and get some new bundles and I overwrote the ones on my web server with these new variants.

That worked out fine and on the HoloLens I can then click the button in order to dynamically pull the cube asset bundle from the web server and display it on the device as below;

20170328_120919_HoloLens

Wrapping Up

With a couple of minor script changes, this workflow around asset bundles seems to work pretty well for the HoloLens. I could perhaps experiment a little further and try out working with different variants of assets and so on but, for my quick experiment, I’m happy that it’s working reasonably well and without too much of a deviation from what the Unity docs say.

What I haven’t yet tried is using the local server that Unity provides and the simulation mode to see if I can get that working for a HoloLens project but it feels like it should work to me…maybe that’s one for another post?