Baby Steps with the Mixed Reality Portal & Simulator on Windows 10 Creators Update

NB: The usual blog disclaimer for this site applies to posts around HoloLens. I am not on the HoloLens team. I have no details on HoloLens other than what is on the public web and so what I post here is just from my own experience experimenting with pieces that are publicly available and you should always check out the official developer site for the product documentation.

Creators Update and the Windows Mixed Reality Portal/Simulator

I’ve been trying out the Creators Update (get it here) on an older Surface Pro 3 for quite a while but I haven’t given it a tonne of focus just yet but with the release I thought it was time to upgrade both my home PCs and my work PCs and I did that in a few spare hours the other day.

One of the things that the Creators Update brings with it is the Windows Mixed Reality Portal and that portal also comes with the Windows Mixed Reality Simulator.

I’ve got very familiar with running an app like the ‘Holograms’ app on my HoloLens and using it to position objects in the real world – here’s a screenshot of the Spaceman floating in mid-air inside a huge atrium at Manchester Metropolitan University the other month;

C5VkK6GWAAEWLj2

While I don’t have an immersive Windows Mixed Reality headset available to me today, the Creators Update makes it possible for me to make use of the Holograms app in an entirely virtual world shown to me via the Windows Mixed Reality Portal. So, if I run up the portal on my device and set it up in Developer Mode for Simulation as detailed in this document then I can use the simulator to see the Spaceman in a whole new ‘world’;

image

So now I have my HoloLens which blends virtual objects into my real world and I have the Mixed Reality Simulator which puts virtual objects into a virtual world projected onto a flat 2D screen and they’re running what looks like a very similar Holograms app Smile

Can I Play RoboRaid?

My first thought was ‘Hey, maybe I can now go and play RoboRaid in this environment?’ but I don’t see that listed today in the Store that I access on my device whether from inside or outside of the Windows Mixed Reality Simulator.

That makes sense to me because I could see the RoboRaid developers perhaps needing to make some changes to their app in order to run inside of this environment rather than on HoloLens.

There are, clearly, differences between HoloLens hardware and immersive headset hardware with the grids on this page listing out different features including Gestures, Motion Controllers, Spatial Mapping and so it wouldn’t be surprising if an app like RoboRaid which makes extensive use of spatial mapping and gestures needed some tweaks to run on an immersive headset.

From looking into the Store and the docs, my curiosity was sparked enough to want to have a think about UWP device families, contracts, APIs for detecting headsets and so on and I wrote a few notes around those below based purely on my experiments in the debugger with the public bits. I have no ‘special’ knowledge here and, clearly, if I did then I wouldn’t have to conduct these types of experiments in a debugger to try and figure things out Smile

What Device (Family) am I Running On?

I wondered what device family I was running on when I run an app inside of the Windows Mixed Reality Simulator on my PC. I think I know but I wanted to check.

If I write a basic ‘Hello World’ app that I run on the HoloLens emulator (Windows 14393) and the Mixed Reality Simulator (Windows 15063) choosing my platforms a little carefully;

image

then I can query the device family property (NB: this is rarely the right way to code for device specific code but it’s a cheap/cheerful thing to do here) then I see this from the Mixed Reality Simulator;

image

and this from the HoloLens emulator;

image

which lines up with what I’d expect – in the HoloLens case, the device is its own computer whereas in the other case it’s my PC that’s the computer. This was the output from this line of code;

            this.txtDeviceFamily.Text = AnalyticsInfo.VersionInfo.DeviceFamily;

Where Do My APIs Come From?

From the point of view of the UWP, this would suggest that when I’m running on the Mixed Reality Simulator I should have access to the Universal API Contract and the Desktop API Extensions.

What about on HoloLens? I would have access to the Universal API Contract alone – as far as I’m aware, there isn’t a set of ‘Mixed Reality’ extension APIs for the UWP and when I look into the folder on my machine I see;

image

although I’m not 100% certain that this is an exhaustive list but if I go and check the documentation then I see that APIs like these ones;

https://docs.microsoft.com/en-us/uwp/api/windows.graphics.holographic

reside in the Universal API contract and so have the potential to be called on any device family albeit with the developer understanding that they need to check for support prior to calling those APIs.

How then does a developer make a call to know whether they are running inside of a mixed reality headset environment?

Do I Have a Mixed Reality Headset?

I can check whether I have a mixed reality headset or not by making a call to the APIs IsSupported and IsAvailable on this HolographicSpace class.

The former tells me (from the docs) whether the OS supports headsets and the latter tells me whether one is attached to the PC and ready for rendering (post set-up by the user).

I was a bit puzzled because the docs seemed to suggest that the IsSupported API was present in 10586 and yet Lucian’s super-helpful plugin told me different;

image

and the code crashed on the HoloLens emulator running 14393 so that seemed to suggest that these APIs weren’t part of 14393.

This wouldn’t be a problem for me except that my HoloLens and emulator are on 14393 and so I ended up with code that looked like this for the moment;

        void UpdateHolographicHeadset()
        {
            var text = "Holographic not supported";

            if (Windows.Foundation.Metadata.ApiInformation.IsPropertyPresent(
                "Windows.Graphics.Holographic.HolographicSpace", "IsSupported") &&
                HolographicSpace.IsSupported)
            {
                text =
                    HolographicSpace.IsAvailable ?
                    "Holographic space available" :
                    "Holographic space not available";

                HolographicSpace.IsAvailableChanged += OnHeadsetAvailabilityChanged;
            }
            else if (AnalyticsInfo.VersionInfo.DeviceFamily == "Windows.Holographic")
            {
                text = "Holographic space supported on HoloLens";
            }
            this.txtHeadset.Text = text;
        }
        async void OnHeadsetAvailabilityChanged(object sender, object e)
        {
            await this.Dispatcher.RunAsync(CoreDispatcherPriority.Normal,
                () =>
                {
                    this.UpdateHolographicHeadset();
                }
            );
        }

and that seems to display reasonable results on the HoloLens emulator (14393);

image

and gives the same answer on my PC whether I launch the app inside/outside of the Mixed Reality Simulator;

image

and I could get this flag to change by switching simulation on/off and then running the app on my desktop;

image

although I’m not sure that I saw the app update dynamically from the “not available” –> “available” state when I switched Simulation back on but I’d have to test that again.

So, assuming that I was running everything on 15063 and could simply call these APIs, how would I differentiate between immersive/HoloLens headsets?

Immersive Headset?

There’s another flag to be tested on the HolographicDisplay class called IsOpaque which gives information about the type of display present. I could make a call to this (once again, factoring in the missing API on 14393);

     void UpdateDisplayType()
        {
            var displayType = "no display";

            if (Windows.Foundation.Metadata.ApiInformation.IsTypePresent(
                "Windows.Graphics.Holographic.HolographicDisplay"))
            {
                displayType =
                    (HolographicDisplay.GetDefault()?.IsOpaque == true) ?
                    "opaque lenses" : "transparent lenses";
            }
            else if (AnalyticsInfo.VersionInfo.DeviceFamily == "Windows.Holographic")
            {
                displayType = "transparent lenses";
            }
            this.txtDisplayType.Text = displayType;
        }

and on the Mixed Reality Simulator I see;

image

and I get the expected result on the HoloLens emulator;

image

but this could be a little confusing because I don’t actually have an immersive headset and I’m just running this app on a flat screen rather than with stereoscopic projection but the IsStereo flag can help me with that too;

        void UpdateStereoDisplay()
        {
            var stereoType = "unknown display";

            if (Windows.Foundation.Metadata.ApiInformation.IsTypePresent(
                "Windows.Graphics.Holographic.HolographicDisplay"))
            {
                stereoType = HolographicDisplay.GetDefault()?.IsStereo == true ?
                    "stereoscopic display" : "2D display";
            }
            this.txtStereoDisplay.Text = stereoType;
        }

but I don’t yet understand the output here because I somewhat expected the app inside of the simulator to say “2D” and it doesn’t seem to do that;

image

and it doesn’t when I run it purely on the desktop either;

image

so perhaps I need to think on that one a little bit more.

A Unity Example?

I wanted to see how this worked with a simple Unity example and so I made a ‘blank’ project with the HoloToolkit and just had a red cube and a wall and the idea is that when the user clicks on the cube, the red cube goes green.

I have this scene;

image

and then a script on the cube tries to handle a click to change the cube to be green and it also tries to hide the wall on headsets with transparent lenses;

using HoloToolkit.Unity.InputModule;
using UnityEngine;
using Windows.Graphics.Holographic;
using Windows.System.Profile;

public class CubeScript : MonoBehaviour, IInputClickHandler
{
    private void Awake()
    {
        var wall = GameObject.Find("Wall");

        if (wall != null)
        {
#if UNITY_UWP && !UNITY_EDITOR
            var opaque = true;

            if (Windows.Foundation.Metadata.ApiInformation.IsTypePresent(
                "Windows.Graphics.Holographic.HolographicDisplay"))
            {
                opaque = (HolographicDisplay.GetDefault()?.IsOpaque == true);
            }
            else if (AnalyticsInfo.VersionInfo.DeviceFamily == "Windows.Holographic")
            {
                opaque = false;
            }
            wall.SetActive(opaque);
#endif // UNITY_UWP
        }
    }
    public void OnInputClicked(InputClickedEventData eventData)
    {
        this.gameObject.GetComponent<Renderer>().material.color = Color.green;
    }
}

This seemed to work out fine on the HoloLens emulator (with the app manifest requesting a minimum platform of 14393, maximum of 15063 and targeting the Universal family);

image

with no wall which is what I was hoping for.

If I then deployed this to my local machine and ran it up inside of the Mixed Reality Portal using the simulator then I saw;

MR

and so I get the wall and the cube running there although it is worth saying that I did experience some glitches (drivers perhaps?) around getting this to display in that sometimes it would display but on a few occasions I noticed the Mixed Reality Portal seemed to reset itself before I got to the Unity splash screen for my app.

Naturally, I’m sure there are better ways of doing what I’m doing here but I felt like I learned a few things and so thought I’d share as part of experimentation…it’s exciting to see these bits in the Creators Update and it’ll be even more exciting to get my hands on a headset and see what the experience is like there.

Don’t forget that you can sign up for news on those headsets here.

Hitchhiking the HoloToolkit-Unity, Leg 13–Continuing with Shared Experiences

NB: The usual blog disclaimer for this site applies to posts around HoloLens. I am not on the HoloLens team. I have no details on HoloLens other than what is on the public web and so what I post here is just from my own experience experimenting with pieces that are publicly available and you should always check out the official developer site for the product documentation.

Continuing with what I’d published in this previous post, I wanted to add some more functionality to the shared experience that I’ve been experimenting with and so I’m going to take this (and the codebase) further in this post.

The first thing that I wanted to add was the ability to download the model that’s going to be viewed from a web server rather than have it hard-baked into the application’s binary.

The ‘Unity’ way of doing this seems to be to use asset bundles and so, in order to achieve this, I combined what I’d learned in this post into my code base such that my project now contains the code that’s needed to both;

  1. Build asset bundles
  2. Download asset bundles at runtime

Note that (as per that post) this involves me changing Unity’s demo scripts somewhat to work with HoloLens (only tiny changes as it happens). Wherever I’ve done that, I’ve used a #if MIKET_CHANGE to flag what I’ve been doing to someone else’s script.

This code now lives here in my project;

image

and I have both the scripts for runtime and the scripts for the Editor so that I can build the asset bundle for the model from within my project.

Loading the Model from an Asset Bundle at Startup

With those scripts added, I needed to do a little work to get this dynamic model loading implemented and so I made a prefab out of the model, took it out of my scene and made it into an asset bundle;

image

and then built that bundle;

image

and deployed it to my test web server in Azure;

image

I made a small structural change in that I moved my ‘Coordinator’ script which handles the essential ‘flow’ of the app so that it was a component of the SharedObjects game object rather than a stand alone script;

image

and I added a new script here which I called ‘Bundle Downloader’ which is intended to the bare bones of loading up an asset from a bundle on a web server, leaning very heavily on my earlier post and the code that Unity ships as part of their demo project and with some very basic support around providing a fallback prefab for the case where the download doesn’t succeed or the developer has simply switched it off in the editor;

using AssetBundles;
using HoloToolkit.Unity;
using System;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;

public class BundleDownloadedEventArgs : EventArgs
{
  public bool DownloadSucceeded { get; set; }
}

public class BundleDownloader : Singleton<BundleDownloader>
{
  [SerializeField]
  string downloadUrl;

  [SerializeField]
  string bundleName;

  [SerializeField]
  string prefabName;

  [SerializeField]
  GameObject fallbackPrefab;

  [SerializeField]
  bool isActive = true;

  public GameObject LoadedPrefab
  {
    get; set;
  }

  public event EventHandler<BundleDownloadedEventArgs> Downloaded;

  public void StartAsyncDownload()
  {
    StartCoroutine(this.DownloadAsync());
  }
  IEnumerator DownloadAsync()
  {
    var prefabObject = this.fallbackPrefab;
    var succeeded = false;

#if !UNITY_EDITOR

    if (this.isActive &&
      !string.IsNullOrEmpty(this.downloadUrl) &&
      !string.IsNullOrEmpty(this.bundleName) &&
      !string.IsNullOrEmpty(this.prefabName))
    {
      AssetBundleManager.SetSourceAssetBundleURL(this.downloadUrl);

      var initializeOperation = AssetBundleManager.Initialize();

      if (initializeOperation != null)
      {
        yield return StartCoroutine(initializeOperation);

        AssetBundleLoadAssetOperation loadOperation = null;

        try
        {
          loadOperation = AssetBundleManager.LoadAssetAsync(
            this.bundleName, this.prefabName, typeof(GameObject));
        }
        catch
        {

        }
        if (loadOperation != null)
        {
          yield return StartCoroutine(loadOperation);

          var loadedPrefab = loadOperation.GetAsset<GameObject>();

          if (loadedPrefab != null)
          {
            prefabObject = loadedPrefab;
            succeeded = true;
          }
        }
      }
    }
#else
    succeeded = true;
#endif

    this.LoadedPrefab = prefabObject;

    if (this.Downloaded != null)
    {
      this.Downloaded(
        this, new global::BundleDownloadedEventArgs()
        {
          DownloadSucceeded = succeeded
        }
      );
    }
    yield break;
  }
}

and I set that up in the Unity editor such that is properties were set to point to the right URLs etc;

image

and I then modified my Coordinator script such that it had two new status values which are represented at the top of this enum here and I also modified the display text that is displayed on start up;

 enum CurrentStatus
  {
    Start,
    WaitingForModelToLoad,
    WaitingToConnectToStage,
    WaitingForRoomApiToStabilise,
    WaitingForModelPositioning,
    WaitingForWorldAnchorExport,
    WaitingForWorldAnchorImport
  }
  void Start()
  {
    StatusTextDisplay.Instance.SetStatusText("waiting for model to load");
  }

and then added a case to my switch statement to check for that Start status and move the app on to the ‘waiting for network’ status;

        case CurrentStatus.Start:
          this.MoveToStatus(CurrentStatus.WaitingForModelToLoad);

          Debug.Log("Coordinator: starting to load model from web server");
          StatusTextDisplay.Instance.SetStatusText("loading model from web server");

          this.GetComponent<BundleDownloader>().Downloaded += this.OnModelDownloaded;
          this.GetComponent<BundleDownloader>().StartAsyncDownload();
          break;

and so this code now gets hold of the new BundleDownloader, asks it to download the model from the web server and when that completes we get this Downloaded event which we handle by adding;

  void OnModelDownloaded(object sender, BundleDownloadedEventArgs e)
  {
    var bundleDownloader = this.GetComponent<BundleDownloader>();

    bundleDownloader.Downloaded -= this.OnModelDownloaded;

    Debug.Log(
      string.Format(
        "Coordinator: download of model from web server has completed and {0}",
        e.DownloadSucceeded ? "succeeded" : "failed or wasn't tried"));

    StatusTextDisplay.Instance.SetStatusText(
      string.Format(
        "{0} model from web server",
        e.DownloadSucceeded ? "loaded" : "failed to load"));

    // Create the model and parent it off this object.
    this.model = Instantiate(bundleDownloader.LoadedPrefab);
    this.model.transform.parent = this.modelParent.transform;

    // Move the world locked parent so that it's in a 'reasonable'
    // place to start with
    this.modelParent.transform.SetPositionAndRotation(
      WORLD_LOCKED_STARTING_POSITION, Quaternion.identity);

    Debug.Log(
      string.Format(
        "Coordinator: waiting for network connection",
        e.DownloadSucceeded ? "succeeded" : "failed or wasn't tried"));

    StatusTextDisplay.Instance.SetStatusText("connecting to room server");

    this.MoveToStatus(CurrentStatus.WaitingToConnectToStage);
  }

with that in place, I now have the same app as previously but with the addition that this version dynamically loads its model from a web server – as in the screenshot below;

image

followed by the model arriving from the web;

image

Adding More to the Remote Head Manager (the T-shirt)

In the existing code, I have the idea that users can be in different rooms based on the name of their WiFi network.

The flow is something like;

  • A room is created if there is not a room on the server for the WiFi network name.
  • The first user into the room positions the model.
  • Subsequent users into the room see the model already positioned by the first user.
  • Users across all rooms can see the HoloLens position and movement of other users regardless of room.

This was working in the previous blog post but I wanted a better representation of a HoloLens user that was remote and so I found a simple model of a T-shirt out on the web;

image

and then I modified the Remote Head Manager script primarily to add another GameObject for the body to be displayed set to be this T-shirt below;

image

and I modified the script in order to support both a head and a body object for a user that is not determined to be in the same room as the user running the app. I’ve included the entire script here but it’s perhaps better to check the github repo for the details;

#define MIKET_CHANGE
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License. See LICENSE in the project root for license information.

using System;
using System.Collections.Generic;
using UnityEngine;
using HoloToolkit.Unity;
using HoloToolkit.Unity.InputModule;

namespace HoloToolkit.Sharing.Tests
{
  /// <summary>
  /// Broadcasts the head transform of the local user to other users in the session,
  /// and adds and updates the head transforms of remote users.
  /// Head transforms are sent and received in the local coordinate space of the GameObject this component is on.
  /// </summary>
  public class RemoteHeadManager : Singleton<RemoteHeadManager>
  {
    public class RemoteHeadInfo
    {
      public long UserID;
      public GameObject HeadObject;
#if MIKET_CHANGE
      public GameObject BodyObject;
#endif     
    }

#if MIKET_CHANGE
    public GameObject remoteHeadPrefab;
    public GameObject remoteBodyPrefab;
#endif

    /// <summary>
    /// Keep a list of the remote heads, indexed by XTools userID
    /// </summary>
    private Dictionary<long, RemoteHeadInfo> remoteHeads = new Dictionary<long, RemoteHeadInfo>();

#if MIKET_CHANGE
    private void OnEnable()
    {
      this.roomId = -1;

      CustomMessages.Instance.MessageHandlers[CustomMessages.TestMessageID.HeadTransform] =
        UpdateHeadTransform;

      SharingStage.Instance.SessionUsersTracker.UserJoined += UserJoinedSession;
      SharingStage.Instance.SessionUsersTracker.UserLeft += UserLeftSession;
    }
#else
    private void Start()
    {
      CustomMessages.Instance.MessageHandlers[CustomMessages.TestMessageID.HeadTransform] = UpdateHeadTransform;

      // SharingStage should be valid at this point, but we may not be connected.
      if (SharingStage.Instance.IsConnected)
      {
        Connected();
      }
      else
      {
        SharingStage.Instance.SharingManagerConnected += Connected;
      }
    }
    private void Connected(object sender = null, EventArgs e = null)
    {
      SharingStage.Instance.SharingManagerConnected -= Connected;

      SharingStage.Instance.SessionUsersTracker.UserJoined += UserJoinedSession;
      SharingStage.Instance.SessionUsersTracker.UserLeft += UserLeftSession;
    }
#endif

    private void Update()
    {
#if MIKET_CHANGE
      this.DetermineCurrentRoom();
#endif
      // Grab the current head transform and broadcast it to all the other users in the session
      Transform headTransform = Camera.main.transform;

      // Transform the head position and rotation from world space into local space
      Vector3 headPosition = transform.InverseTransformPoint(headTransform.position);

      Quaternion headRotation = Quaternion.Inverse(transform.rotation) * headTransform.rotation;

#if MIKET_CHANGE
      CustomMessages.Instance.SendHeadTransform(headPosition, headRotation,
         this.roomId);
#endif
    }
#if MIKET_CHANGE
    void DetermineCurrentRoom()
    {
      if (this.roomId == -1)
      {
        var roomManager = SharingStage.Instance.Manager.GetRoomManager();

        if (roomManager != null)
        {
          var room = roomManager.GetCurrentRoom();
          this.roomId = room.GetID();
        }
      }
    }
#endif

    protected override void OnDestroy()
    {
      if (SharingStage.Instance != null)
      {
        if (SharingStage.Instance.SessionUsersTracker != null)
        {
          SharingStage.Instance.SessionUsersTracker.UserJoined -= UserJoinedSession;
          SharingStage.Instance.SessionUsersTracker.UserLeft -= UserLeftSession;
        }
      }

      base.OnDestroy();
    }

    /// <summary>
    /// Called when a new user is leaving the current session.
    /// </summary>
    /// <param name="user">User that left the current session.</param>
    private void UserLeftSession(User user)
    {
      int userId = user.GetID();
      if (userId != SharingStage.Instance.Manager.GetLocalUser().GetID())
      {
        RemoveRemoteHead(remoteHeads[userId].HeadObject);
        remoteHeads.Remove(userId);
      }
    }

    /// <summary>
    /// Called when a user is joining the current session.
    /// </summary>
    /// <param name="user">User that joined the current session.</param>
    private void UserJoinedSession(User user)
    {
      if (user.GetID() != SharingStage.Instance.Manager.GetLocalUser().GetID())
      {
        GetRemoteHeadInfo(user.GetID());
      }
    }

    /// <summary>
    /// Gets the data structure for the remote users' head position.
    /// </summary>
    /// <param name="userId">User ID for which the remote head info should be obtained.</param>
    /// <returns>RemoteHeadInfo for the specified user.</returns>
    public RemoteHeadInfo GetRemoteHeadInfo(long userId)
    {
      RemoteHeadInfo headInfo;

      // Get the head info if its already in the list, otherwise add it
      if (!remoteHeads.TryGetValue(userId, out headInfo))
      {
        headInfo = new RemoteHeadInfo();
        headInfo.UserID = userId;
        headInfo.HeadObject = CreateRemoteHead();

#if MIKET_CHANGE
        headInfo.BodyObject = Instantiate(this.remoteBodyPrefab);
        headInfo.BodyObject.transform.parent = this.gameObject.transform;
#endif
        remoteHeads.Add(userId, headInfo);
      }

      return headInfo;
    }

    /// <summary>
    /// Called when a remote user sends a head transform.
    /// </summary>
    /// <param name="msg"></param>
    private void UpdateHeadTransform(NetworkInMessage msg)
    {
      // Parse the message
      long userID = msg.ReadInt64();

      Vector3 headPos = CustomMessages.Instance.ReadVector3(msg);

      Quaternion headRot = CustomMessages.Instance.ReadQuaternion(msg);

#if MIKET_CHANGE
      long remoteRoomId = msg.ReadInt64();
#endif

      RemoteHeadInfo headInfo = GetRemoteHeadInfo(userID);
      headInfo.HeadObject.transform.localPosition = headPos;
      headInfo.HeadObject.transform.localRotation = headRot;

#if MIKET_CHANGE
      var rayLength = maxRayDistance;

      RaycastHit hitInfo;

      if (Physics.Raycast(
        headInfo.HeadObject.transform.position,
        headInfo.HeadObject.transform.forward,
        out hitInfo))
      {
        rayLength = hitInfo.distance;
      }
      var lineRenderer = headInfo.HeadObject.GetComponent<LineRenderer>();
      lineRenderer.SetPosition(1, Vector3.forward * rayLength);

      if ((remoteRoomId == -1) || (this.roomId == -1) ||
        (remoteRoomId != this.roomId))
      {
        headInfo.BodyObject.SetActive(true);
        headInfo.BodyObject.transform.localPosition = headPos;
        headInfo.BodyObject.transform.localRotation = headRot;
      }
      else
      {
        headInfo.BodyObject.SetActive(false);
      }
#endif
    }

    /// <summary>
    /// Creates a new game object to represent the user's head.
    /// </summary>
    /// <returns></returns>
    private GameObject CreateRemoteHead()
    {
      GameObject newHeadObj = Instantiate(this.remoteHeadPrefab);
      newHeadObj.transform.parent = gameObject.transform;

#if MIKET_CHANGE
      this.AddLineRenderer(newHeadObj);
#endif
      return newHeadObj;
    }
#if MIKET_CHANGE
    void AddLineRenderer(GameObject headObject)
    {
      var lineRenderer = headObject.AddComponent<LineRenderer>();
      lineRenderer.useWorldSpace = false;
      lineRenderer.startWidth = 0.01f;
      lineRenderer.endWidth = 0.05f;
      lineRenderer.positionCount = 2;
      lineRenderer.SetPosition(0, Vector3.forward * 0.1f);
      var material = new Material(Shader.Find("Diffuse"));
      material.color = colors[this.colorIndex++ % colors.Length];

      lineRenderer.material = material;
    }
#endif

    /// <summary>
    /// When a user has left the session this will cleanup their
    /// head data.
    /// </summary>
    /// <param name="remoteHeadObject"></param>
    private void RemoveRemoteHead(GameObject remoteHeadObject)
    {
      DestroyImmediate(remoteHeadObject);
    }
#if MIKET_CHANGE
    long roomId;
    const float maxRayDistance = 5.0f;
    int colorIndex;
    static Color[] colors =
    {
      Color.red,
      Color.green,
      Color.blue,
      Color.cyan,
      Color.magenta,
      Color.yellow
    };
#endif
  }
}

Most of the modifications here are to try and figure out whether the user is present in the same room (i.e. WiFi network) as the current user – this seems to work ok but I’d not be surprised to see bugs and the way that I’m doing it currently means that you can certainly see a T-shirt for a local user until the point where the code realises that it really is a local user.

The net effect of all this looks something like the picture below and, currently, I move and rotate the T-shirt just as much as the head and that’s an area that could definitely be refined;

image

Moving the Model After It Has Been Positioned

Previously, the app has allowed one user to position a model in a room, say the word “lock” and then anchor that model such that another user can view it at the same place in the same room or remotely from another room.

What I haven’t previously made possible was the idea that the users might be able to continue to manipulate the model after it has been placed into the room.

In order to progress this, I need to modify the current code base because right now the ModelParent object takes on two roles;

image

The first role is that this object is the one which ultimately becomes world anchored to provide a common parent for users to view the same content.

The second role is that this object is the one with the “User Moveable” script on it allowing the user to move it around and then say “lock” to world anchor it.

These two roles are compatible in a situation where I no longer want to move the model once it has been “locked”. Hover, if the model is to move after that locking process then I need to make changes because the world anchor stops it responding to movement.

One way around this is to dynamically add/remove the world anchor but I preferred what seemed like a simpler way which is to leave this code alone and dynamically to add a “User Moveable” behaviour to the model as it is created and so I modified my coordinator script to do this.

I have code that runs [as/when] the world anchor is [imported/exported] from the server and so it seemed sensible to dynamically add that “User Moveable” behaviour at this point;

  void OnImportOrExportCompleted(bool succeeded)
  {
    StatusTextDisplay.Instance.SetStatusText("room in sync");

    // Allow the child to be moved now that it is positioned in
    // the right place.
    this.model.AddComponent<UserMoveable>();

    // Switch on the remote head management.
    this.modelParent.GetComponent<RemoteHeadManager>().enabled = true;
  }

and so now I have solved perhaps “half” the problem in that the model can be moved once it has been locked.

However, those movements won’t be reflected on any of the other devices in the room. That needs more of a change…

Baby Steps Into the SyncModel

To synchronize the movement of the model, I need a component which will monitor its transform, note any changes and then send them over the network to other devices to be applied locally.

I’ve seen this sort of behaviour in the SyncSpawnManager and PrefabSpawnManager pieces before which I wrote a little about in this post and, from unpicking that code, I know that the HoloToolkit-Unity has a component named TransformSynchronizer which does exactly this work and that it works with the SyncTransform class to do this;

image

Here, the TransformSynchronizer does the monitoring of the data held by the underlying SyncTransform which the code refers to as ‘the data model’. As such, it derives from SyncObject which lives within the whole SyncModel part of the toolkit;

image

At the time of writing, I’m not sure that I entirely understand how all of this ‘synchronized data model’ part of the sharing toolkit works to the ultimate level of detail but I think that I now know that the SharingStage component which is used to group up the networking functionality has a root object;

image

and that’s of type SyncRoot and it’s the data (broken down into SyncObject derived types) which is to be sync’d across devices with change notification support as you’d expect.

I noticed that the Prefab Spawn Manager has some data in this SyncRoot which it uses to synchronize the list of objects that have been spawned across devices via its various routines;

image

In this particular project, I’m not using the Prefab Spawn Manager because I am not dynamically adding objects into the scene once they are loaded from the web server.

However, I figure that I can use the same mechanism that the Prefab Spawn Manager uses in order to try and keep the transformations on my model in sync and so I added a property here;

#define MIKET_CHANGE
//
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License. See LICENSE in the project root for license information.
//

using HoloToolkit.Sharing.Spawning;
using HoloToolkit.Sharing.SyncModel;

namespace HoloToolkit.Sharing
{
  /// <summary>
  /// Root of the synchronization data model used by this application.
  /// </summary>
  public class SyncRoot : SyncObject
  {
#if MIKET_CHANGE
    [SyncData]
    public SyncSpawnedObject ModelObject;
#endif

    /// <summary>
    /// Children of the root.
    /// </summary>
    [SyncData]
    public SyncArray<SyncSpawnedObject> InstantiatedPrefabs;

    /// <summary>
    /// Constructor.
    /// </summary>
    /// <param name="rootElement">Root Element from Sharing Stage</param>
    public SyncRoot(ObjectElement rootElement)
    {
      Element = rootElement;
      FieldName = Element.GetName().GetString();
      InitializeSyncSettings();
      InitializeDataModel();
    }

    private void InitializeSyncSettings()
    {
      SyncSettings.Instance.Initialize();
    }

    /// <summary>
    /// Initializes any data models that need to have a local state.
    /// </summary>
    private void InitializeDataModel()
    {
      InstantiatedPrefabs.InitializeLocal(Element);

#if MIKET_CHANGE
      this.ModelObject.InitializeLocal(Element);
#endif
    }
  }
}

and I wanted to see if I could then make use of this to synchronize the model after it had already been created.

I modified the code in my Coordinator script so as to make sure that this was set up appropriately at the point when the world anchor’d object was either exported or imported from/into the current device. That changed the existing method that dealt with this to;

 void OnImportOrExportCompleted(bool succeeded)
  {
    StatusTextDisplay.Instance.SetStatusText("room in sync");

    // Allow the child to be moved now that it is positioned in
    // the right place.
    this.model.AddComponent<UserMoveable>();

    var dataModel = SharingStage.Instance.Root.ModelObject;
    dataModel.GameObject = this.model.gameObject;
    dataModel.Initialize(this.model.gameObject.name, this.model.transform.GetFullPath());
    dataModel.Transform.Position.Value = this.model.transform.localPosition;
    dataModel.Transform.Rotation.Value = this.model.transform.localRotation;
    dataModel.Transform.Scale.Value = this.model.transform.localScale;

    var synchronizer = this.model.EnsureComponent<TransformSynchronizer>();
    synchronizer.TransformDataModel = dataModel.Transform;

    // Switch on the remote head management.
    this.modelParent.GetComponent<RemoteHeadManager>().enabled = true;
  }

and that seemed to enable the experience that I was looking for as demonstrated in this little test video below;

and so that works reasonably well but it led me to think that there might be another option here where a user might want to move the four cubes of the model as a single group or, alternatively, as four separate objects.

Moving the Parent or the Children

In the solution at this point, I have a model (ultimately served from a web server) and that model is treated as a single thing. There’s a box collider wrapped around it so that it can be ‘hit tested’ and all the cubes behave as a group.

image

But there might be an occasion where a user wanted to treat each individual cube separately as they also have colliders on them. That could be taken further by allowing the user to do this recursively and let them choose levels of objects to manipulate but that might be going “a little far” for this blog post and so I’ve chosen just to think about the direct children of the model and I’ve made an assumption that the model designer will be kind and will have put colliders onto them for my code to hit against.

I don’t necessarily want to lose the original ‘grouped’ mode of operation though and so I figured it might be nice to have that be the default and then add some voice command (‘split’) which switches into the child-focused mode. At the time of writing, I haven’t attempted to implement the idea of reversing this choice as it’s a bit more tricky.

With that in mind, I decided that the way that I would approach this would be to make the model and its direct children all capable of being moved in a trackable way that’s synchronized across devices. If I assume this then the act of switching from ‘group mode’ to ‘child mode’ simply involves taking the collider off the model object in order to let the child colliders ‘come through’.

To make these changes, I first went and changed my SyncRoot again such that I had a SyncArray rather than a single SyncSpawnedObject instance;

#define MIKET_CHANGE
//
// Copyright (c) Microsoft Corporation. All rights reserved.
// Licensed under the MIT License. See LICENSE in the project root for license information.
//

using HoloToolkit.Sharing.Spawning;
using HoloToolkit.Sharing.SyncModel;

namespace HoloToolkit.Sharing
{
  /// <summary>
  /// Root of the synchronization data model used by this application.
  /// </summary>
  public class SyncRoot : SyncObject
  {
#if MIKET_CHANGE
    [SyncData]
    public SyncArray<SyncSpawnedObject> ModelObjects;
#endif

    /// <summary>
    /// Children of the root.
    /// </summary>
    [SyncData]
    public SyncArray<SyncSpawnedObject> InstantiatedPrefabs;

    /// <summary>
    /// Constructor.
    /// </summary>
    /// <param name="rootElement">Root Element from Sharing Stage</param>
    public SyncRoot(ObjectElement rootElement)
    {
      Element = rootElement;
      FieldName = Element.GetName().GetString();
      InitializeSyncSettings();
      InitializeDataModel();
    }

    private void InitializeSyncSettings()
    {
      SyncSettings.Instance.Initialize();
    }

    /// <summary>
    /// Initializes any data models that need to have a local state.
    /// </summary>
    private void InitializeDataModel()
    {
      InstantiatedPrefabs.InitializeLocal(Element);

#if MIKET_CHANGE
      this.ModelObjects.InitializeLocal(Element);
#endif
    }
  }
}

and then I modified the code which runs when the world anchor is either imported or exported to populate this array in a very similar way to the single-object case. Note that I made the slightly dubious decision to store both the parent model and all of its children in the same array which would likely lead some future maintainer (i.e. me) to make ‘off by 1’ type errors here;

  void OnImportOrExportCompleted(bool succeeded)
  {
    StatusTextDisplay.Instance.SetStatusText(
      string.Format("room import/export {0}", succeeded ? "succeeded" : "failed"));

    if (succeeded)
    {
      var isExporter = this.currentStatus == CurrentStatus.WaitingForWorldAnchorExport;

      StatusTextDisplay.Instance.SetStatusText("room is now in sync");

      // First, make sure the model itself is set up to be moveable in a trackable way.
      // NB: We use index 0 for the model itself.
      this.MakeModelPartMoveableAndTrackable(this.model, 0, isExporter);

      // And all of its children are also moveable.
      var childCount = this.model.transform.childCount;

      // NB: 1 to N because the have the model in slot 0.
      for (int i = 1; i <= childCount; i++)
      {
        // NB: We add 1 because the model itself is in slot 1.
        var child = this.model.transform.GetChild(i - 1);
        this.MakeModelPartMoveableAndTrackable(child.gameObject, i, isExporter);
      }
      // Switch on the remote head management.
      this.modelParent.GetComponent<RemoteHeadManager>().enabled = true;

      // Switch on the keyword recognizer listening for 'join' and 'split'
      this.gameObject.GetComponent<KeywordManager>().StartKeywordRecognizer();
    }
  }
  SyncSpawnedObject MakeModelPartMoveableAndTrackable(
    GameObject objectInstance, int indexIntoRootSyncStore, bool isExporter)
  {
    SyncSpawnedObject dataModel = null;

    if (isExporter)
    {
      dataModel = new SyncSpawnedObject();
      dataModel.GameObject = objectInstance; ;
      dataModel.Initialize(objectInstance.name, objectInstance.transform.GetFullPath());
      dataModel.Transform.Position.Value = objectInstance.transform.localPosition;
      dataModel.Transform.Rotation.Value = objectInstance.transform.localRotation;
      dataModel.Transform.Scale.Value = objectInstance.transform.localScale;

      SharingStage.Instance.Root.ModelObjects.AddObject(dataModel);
    }
    else
    {
      dataModel = 
        SharingStage.Instance.Root.ModelObjects.GetDataArray()[indexIntoRootSyncStore];
    }
    objectInstance.EnsureComponent<UserMoveable>();
    var synchronizer = objectInstance.EnsureComponent<TransformSynchronizer>();
    synchronizer.TransformDataModel = dataModel.Transform;

    return (dataModel);
  }

and so now my code is tracking all of the N objects that live at level 0 (the model) and level 1 (its direct children) in the scene which is loaded from the web server.

With that in place, I just need to add a new KeywordManager component to listen for the word “split”

image

and then I can write the code on my Coordinator script to ensure this takes away any top collider on the model;

  public void OnSplit()
  {
    this.GetComponent<KeywordManager>().enabled = false;
    this.model.GetComponent<Collider>().enabled = false;
  }

and that’s pretty much it for this post.

Testing and Wrapping Up

To test this last piece of ‘splitting up’ code, I had to be a little crafty and make use of HoloLens+Emulator as two separate users as I only had the one HoloLens. I’ll try it with multiple devices at the first opportunity but, for now, here’s a capture of it running across my device and my emulator and it seems to work reasonably well;

What I’d like to do next is to capture how this now looks when I have multiple users in a single room and also multiple users split across rooms represented by the T-shirt.

I’ll come back to that in a follow on post…

The code for all of this is here on github, keep in mind that you’ll need to set up IP addresses or host names to make it work.

Using OCR to Read an IP Address in a Holographic UWP App

NB: The usual blog disclaimer for this site applies to posts around HoloLens. I am not on the HoloLens team. I have no details on HoloLens other than what is on the public web and so what I post here is just from my own experience experimenting with pieces that are publicly available and you should always check out the official developer site for the product documentation.

Just a short post. I’ve been playing with a situation recently where I needed to get an IP address into a holographic app. There’s lots of different ways that you might do this depending on how frequently the IP address might change;

  • Hard-code it into the app.
  • Put it into a config file (perhaps loaded from some other server).
  • Type it into the app’s UI although I think it’s fair to say that typing IP addresses on HoloLens isn’t so much fun.
  • Magically download it via some sort of bluetooth service
  • Speak it to the application through the microphone and UWP speech recognition
  • Put the IP address into a QR code and have the app scan it through the camera

and I’m sure there’s many more but today I thought I’d experiment with what seemed like a fairly natural idea – when I want to give another person an IP address, I usually write it onto a piece of paper and hand it to them or pin it up on the wall.

So, why not with the device?

Because I already have some code which runs on HoloLens and scans for QR codes (see this blog post and accompanying github) it felt like it would be very little effort to change the QR code scanning that happens in that blog post into some “IP Address” OCR recognition and see how well or badly that works out.

Here’s an example of how well/badly it worked out – I think it works quite well but it’s fair to say that I haven’t managed to get it to work with handwriting although that doesn’t surprise me as I’ve never found the OCR engine in UWP to process handwriting and maybe it’s not meant to so I’m not being critical of it when I say that.

Note – it’s “a bit tricky” to record an app on the HoloLens that is actively taking over the webcam so I’m having to show this with static images;

20170328_144256_HoloLens

20170328_144320_HoloLens

but it feels like a scan of an IP address like that can be done in about 1-2s once the camera has a view of it although, naturally, it’s possible for the camera to get a partial view such as 2.168.0.1 for the above address so a re-scan might be necessary in some circumstances.

In terms of the code here, I simply did some minimal changes to what I already had around QR code processing in that I removed the ZXing library from my project and then changed the public interface so that it looks as it does in the code below and clearly I should do something about renaming the namespace here but this call says “Hey, find me an IP address from the first camera on the system or timeout after 30s”;

MediaFrameQrProcessing.Wrappers.IPAddressScanner.ScanFirstCameraForIPAddress(
        result =>
        {
          UnityEngine.WSA.Application.InvokeOnAppThread(() =>
          {
            // result here is a System.Net.IPAddress...
            this.textMesh.text = result?.ToString() ?? "not found";
          }, 
          false);
        },
        TimeSpan.FromSeconds(30));

and it gives me back an IPAddress instance. In my underlying library, I replaced the QR code frame scanner with a (basic!) IP address frame scanner;

namespace MediaFrameQrProcessing.Processors
{
  using MediaFrameQrProcessing.VideoDeviceFinders;
  using System.Runtime.InteropServices.WindowsRuntime;
  using System.Threading.Tasks;
  using Windows.Devices.Enumeration;
  using Windows.Media.Capture;
  using Windows.Media.Capture.Frames;
  using Windows.Media.Ocr;
  using System;
  using System.Text.RegularExpressions;
  using System.Net;

  public class IPAddressFrameProcessor : MediaCaptureFrameProcessor
  {
    public IPAddress Result { get; private set; }

    public IPAddressFrameProcessor(
      MediaFrameSourceFinder mediaFrameSourceFinder, 
      DeviceInformation videoDeviceInformation, 
      string mediaEncodingSubtype, 
      MediaCaptureMemoryPreference memoryPreference = MediaCaptureMemoryPreference.Cpu) 

      : base(
          mediaFrameSourceFinder, 
          videoDeviceInformation, 
          mediaEncodingSubtype, 
          memoryPreference)
    {
    }
    protected override async Task<bool> ProcessFrameAsync(MediaFrameReference frameReference)
    {
      bool done = false;

      // doc here https://msdn.microsoft.com/en-us/library/windows/apps/xaml/windows.media.capture.frames.videomediaframe.aspx
      // says to dispose this softwarebitmap if you access it.
      using (var bitmap = frameReference.VideoMediaFrame.SoftwareBitmap)
      {
        try
        {
          if (this.ocrEngine == null)
          {
            this.ocrEngine = OcrEngine.TryCreateFromUserProfileLanguages();
            this.regex = new Regex(IP_ADDRESS_PATTERN);
          }
          var results = await this.ocrEngine.RecognizeAsync(bitmap);

          if (results != null)
          {
            var matchingResults = this.regex.Matches(results.Text);

            for (int i = 0; !done && (i < matchingResults.Count); i++)
            {
              IPAddress parsedAddress;

              done = IPAddress.TryParse(matchingResults[i].Value, out parsedAddress);

              if (done)
              {
                this.Result = parsedAddress;
              }
            }
          }
        }
        catch
        {
        }
      }
      return (done);
    }
    Regex regex;
    OcrEngine ocrEngine;

    // Taken from S.O. http://stackoverflow.com/questions/106179/regular-expression-to-match-dns-hostname-or-ip-address
    const string IP_ADDRESS_PATTERN =
      @"(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)(\.(25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)){3}";
  }
}

and that all seems to work reasonably well.

The code for this (the library and the 2D XAML test app) is here if anyone wants to play with it.