NB: The usual blog disclaimer for this site applies to posts around HoloLens. I am not on the HoloLens team. I have no details on HoloLens other than what is on the public web and so what I post here is just from my own experience experimenting with pieces that are publicly available and you should always check out the official developer site for the product documentation.

To date in my posts around the HoloToolkit, I’ve generally taken one of two approaches;

• had some holograms in my scene at design time in Unity and then interacted with them in some fashion from code.
• created holograms from my code as some type of interaction is performed in the scene (e.g. an air tap gesture)

In both cases, I’ve used the world co-ordinate system to position my holograms relative to the origin of the scene and I’ve not worried about;

1. persisting those holograms across instances of my app.
2. linking the position of those holograms to something in the real world such that a hologram placed (e.g.) above a chair continues to be placed above the chair.

There are quite a few subtleties to (2) above.

The first would be that an object placed at (e.g.) the co-ordinate (3,3,3) when an application starts up may well appear in a different place in the real world each time the app is used because the origin and forward direction is relative to the position of the user’s gaze in a room and the co-ordinate system re-orientates itself around the user. So, the position 3,3,3 can be a completely different place in the same room depending on where the user is standing and looking when they run the app.

The article over here;

Coordinate Systems

explains this and calls it a “Stationary Frame of Reference” and it also explains another subtlety to point (2) above as;

“Over time, as the system learns more about the user’s environment it may determine that distances between various points in the real-world are shorter or longer than the system previously believed. Therefore, the system adjusts as the user walks around a larger area resulting in holograms that you’ve placed in a stationary coordinate system may be seen to drift off their original position.”

If I want my holograms then to appear at the same place (in the real world) and not to drift as the device learns more about its environment then I need to make sure that I make use of spatial anchors which are also explained in that same (Coordinate Systems) article which goes on to explain how I can also code for point (1) above;

“Spatial anchors can also allow your app to remember an important location even after your app suspends or the device is shut down.

You can save to disk the spatial anchors your app creates, and then load them back again later, by persisting them to your app’s spatial anchor store. When saving or loading an anchor, you provide a string key that is meaningful to your app, in order to identify the anchor later.”

It’s a double-win in that I can get the persistence that I want on my Holograms and I also get the ability to anchor them to something in the real world in a stable manner. As an aside, the article also explains that I can use this capability to share anchors between HoloLens devices such that more than one device can reason about the same location and, while I’ve used this and it’s very cool I’m not going to do anything more with it in this post.

In the frameworks, spatial anchors seem to be called ‘world anchors’ and there’s a documentation page here;

World Anchor in Unity

that talks about how to use them and I also found this discussion on the forums helpful;

How to use Unity world anchors?

as was this discussion which explains that trying to use a world anchor on a component in Unity which is a rigid body isn’t going to work;

Creating a WorldAnchor always comes back with a null?

as I banged my head a little against that for a few minutes so it was great to find that forum post which suggested “I was not alone”.

With that set of documentation references out of the way, I thought I’d put together an example of this using two different mechanisms to position objects. The first mechanism is to use the “tap to place” mechanism offered by the spatial mapping parts of the HoloToolkit whereas in the second mechanism I’ll try and make use of the “hand draggable” mechansim that the toolkit offers.

I set up a new Unity project and brought in a couple of Unity packages with the first being the HoloToolkit-Unity and the second being this nice little package of Xmas themed assets which provide me with a Xmas present model to play with.

I used the HoloToolkit’s menu options to ensure that my project, scene were configured for Holographic development and I also made sure that my UWP capabilities were set to allow access to the internet and spatial data;

and;

I then tinkered a little bit with the Xmas present prefab such as to change the scale of the asset a little as it was a bit large for my purposes;

and I also added a box collider to it;

so that I’ll be able to know when it crashes into things (specifically, the mesh of the room) at a later point.

Method 1 – Tap to Place

My first attempt at getting this up and running was to use the Tap to Place script from the HoloToolkit which already does some work on your behalf to set up spatial anchors for holograms. That script essentially adds a behaviour to a hologram such that on an initial tap it will follow your gaze, hugging the outline of the spatial mesh until the second tap when it will park itself in the location where you have positioned it.

Step 1 – Creating a Present On Air-Tap

I wanted to create a Xmas present in front of the user at the point where they did an air-tap and to do so is a pretty simple thing to do. I added a blank game object called ‘Placeholder’ to my scene and then added a number of Holotoolkit scripts to it;

with those scripts being;

and then the last one here called ‘Placeholder’ is my own script;

and that becomes very simple to write in that I just have this code;

using HoloToolkit.Unity.InputModule;
using UnityEngine;

public class Placeholder : MonoBehaviour, IInputClickHandler
{
  public Transform prefab;

  private void Start()
  {
    InputManager.Instance.PushFallbackInputHandler(
      this.gameObject);
  }
  public void OnInputClicked(InputEventData eventData)
  {
    var instance = Instantiate(prefab);

    instance.gameObject.transform.position =
      GazeManager.Instance.GazeOrigin +
      GazeManager.Instance.GazeNormal * 1.5f;
  }
}

and so the idea here is that the script is a ‘fallback handler’ for when the user air-taps (or clicks) and there isn’t an object in focus which handles that click. The handler then just creates an instance of the prefab at 1.5 metres in front of the user and I’ve configured the prefab in the editor to be my Xmas present;

and so with that, I’ve got a bit of code that will allow me to create Xmas presents in front of the user wherever they tap but I also wanted the user to be able to position them within the real world and so I need to add some ability to do that.

Step 2 – Moving and Placing the Xmas Present

In order to place the Xmas present in the real world, my app needs the picture of the real world provided by spatial mapping and so I dragged out the ‘Spatial Mapping’ prefab from the Holotoolkit onto my scene.

I altered the use of the prefab such that it did not draw the wireframe mesh of the map;

and I then altered my script which creates the Xmas presents in a couple of ways;

using HoloToolkit.Unity;
using HoloToolkit.Unity.InputModule;
using UnityEngine;

public class Placeholder : MonoBehaviour, IInputClickHandler
{
  public Transform prefab;

  private void Start()
  {
    InputManager.Instance.PushFallbackInputHandler(
      this.gameObject);
  }
  private void Update()
  {
    if (!this.loaded && (WorldAnchorManager.Instance.AnchorStore != null))
    {
      var ids = WorldAnchorManager.Instance.AnchorStore.GetAllIds();

      // NB: I'm assuming that the ordering here is either preserved or
      // maybe doesn't matter.
      foreach (var id in ids)
      {
        var instance = Instantiate(this.prefab);
        WorldAnchorManager.Instance.AttachAnchor(instance.gameObject, id);
      }
      this.loaded = true;
      this.count = ids.Length;
    }
  }
  public void OnInputClicked(InputEventData eventData)
  {
    var instance = Instantiate(this.prefab);

    instance.gameObject.transform.position =
      GazeManager.Instance.GazeOrigin +
      GazeManager.Instance.GazeNormal * 1.5f;

    var tapToPlace = instance.gameObject.AddComponent<TapToPlace>();
    tapToPlace.SavedAnchorFriendlyName = (++this.count).ToString();
  }
  bool loaded;
  int count;
}

The OnInputClicked handler has been updated to add a TapToPlace script to the Xmas present being created and to give it a ‘friendly name’ which is simply an incrementing counter and this name will be used by the TapToPlace script in order to save a spatial anchor which is associated with that game object.

I’ve also added the Update handler which attempts to wait until the WorldAnchorManager has loaded up its store of anchors at the start of the application and it then uses any IDs stored in there to recreate the objects that they represent at the place where they were previously positioned.

Surprisingly, because of the work that’s already going on inside of the TapToPlace script and the WorldAnchorManager in the toolkit, that’s all I need in order to position holograms with spatial anchors and have them persist across sessions.

Step 3 – Trying it Out

Trying it out to position some presents in a limited floor space looks something like this;

If you spot bits of mesh in that video then that’s because the Tap to Place script turns on the visible mesh while objects are being positioned and then turns it off against once they are in place. Also, keep in mind that the video is not a true reflection of the quality of the experience on the device.

Method 2 – Hand Draggable

I must admit that the previous method was a bit ‘magical’ to me in that I didn’t feel very involved in knowing where the spatial anchors were coming from or how they were being used.

Also, I wanted to have a method where I could manipulate the Xmas presents with my hands rather than have them follow my gaze and so I changed things a little to replace my use of the “Tap to Place” script with the “Hand Draggable” script.

To try and put this together, I changed the code in my Placeholder script to take out the piece which added the TapToPlace component;

using HoloToolkit.Unity;
using HoloToolkit.Unity.InputModule;
using UnityEngine;

public class Placeholder : MonoBehaviour, IInputClickHandler
{
  public Transform prefab;

  private void Start()
  {
    InputManager.Instance.PushFallbackInputHandler(
      this.gameObject);
  }
  private void Update()
  {
    if (!this.loaded && (WorldAnchorManager.Instance.AnchorStore != null))
    {
      var ids = WorldAnchorManager.Instance.AnchorStore.GetAllIds();

      // NB: I'm assuming that the ordering here is either preserved or
      // maybe doesn't matter.
      foreach (var id in ids)
      {
        var instance = Instantiate(this.prefab);
        WorldAnchorManager.Instance.AttachAnchor(instance.gameObject, id);
      }
      this.loaded = true;
    }
  }
  public void OnInputClicked(InputEventData eventData)
  {
    var instance = Instantiate(this.prefab);

    instance.gameObject.transform.position =
      GazeManager.Instance.GazeOrigin +
      GazeManager.Instance.GazeNormal * 1.5f;
  }
  bool loaded;
}

and I then modified my Xmas present prefab so as to have both the HandDraggable script on it and a new script that I called Present;

and that Present script is as below;

using HoloToolkit.Unity;
using HoloToolkit.Unity.InputModule;
using UnityEngine;

public class Present : MonoBehaviour
{
  static int sphereCount;
  int sphereNumber;

  void Start()
  {
    this.gameObject.GetComponent<HandDraggable>().StartedDragging += OnStartedDragging;
    this.gameObject.GetComponent<HandDraggable>().StoppedDragging += OnStoppedDragging;

    this.sphereNumber = ++sphereCount;
  }
  void OnStartedDragging()
  {
    // When we start dragging we clear out any existing world anchor for this
    // Xmas present.
    WorldAnchorManager.Instance.RemoveAnchor(this.gameObject);

    // We also add the RigidBody component so that (e.g.) you can't push
    // the present through the floor.
    var rigidBody = this.gameObject.AddComponent<Rigidbody>();

    // But we don't want it to fall on the floor on its own.
    rigidBody.useGravity = false;
  }
  void OnStoppedDragging()
  {
    // We take away the RigidBody because it doesn't play well with
    // Rigidbody.
    Destroy(this.gameObject.GetComponent<Rigidbody>());

    // Now remember where the present was.
    WorldAnchorManager.Instance.AttachAnchor(this.gameObject, this.sphereNumber.ToString());
  }
  void OnCollisionEnter(Collision collision)
  {
    // If we do get a collision then we (a bit rudley) just stop the
    // drag operation.
    this.gameObject.GetComponent<HandDraggable>().StopDragging();
  }
}

In an ideal world, I’d have just added the RigidBody component to the Xmas present as well but it’s not compatible with world anchors (see the forum post I referred to earlier) and so there’s a little code above to add/remove it dynamically when I need it in order to stop my Xmas presents being dragged through walls and floors and so on. Clearly, the collision detection that I’m doing there is quite basic and not very friendly to the user but it’s enough for me to be working with.

With that in place, I can now run this app a couple of times in the video capture below and see that it also remembers the location of my Xmas presents;

Taking that Further

It turns out to be relatively simple (subject to all the mistakes I’ve probably made ) to anchor my Holograms into the real world using the toolkit but it had me thinking about quite a few ideas that I’ll add to follow-on posts…