NB: The usual blog disclaimer for this site applies to posts around HoloLens. I am not on the HoloLens team. I have no details on HoloLens other than what is on the public web and so what I post here is just from my own experience experimenting with pieces that are publicly available and you should always check out the official developer site for the product documentation.

I must admit that two words which can sometimes strike fear into my heart are the words;

Case Study

I’m partially kidding but I’m not a huge fan of case studies which can sometimes be fairly dry write-ups of the form;

“Company C took technology T and solved problem P in time T and saved D dollars”

Of course, that sort of stuff is really important and it is always going to depend on the write-up but I don’t generally put reading case studies to the top of my to-do list.

Against that backdrop, I’ve been very pleasantly surprised by the really interesting HoloLens developer case studies that are published on this site;

HoloLens Developer Case Studies

and I’ve been working my way through them because they are much more at the level of;

“developer wanted to achieve X, this is how they went about it and the challenges they came across in doing it”

and one of the entries that I read quite a long time ago was this one about how to make holes in your walls, ceilings and floors;

“Case study – Looking through holes in your reality”

and it’s brilliant because this idea of being able to “look through surfaces” in HoloLens apps is one of the things that I’ve found to be truly magical across apps like Fragments, RoboRaid and others and the technique isn’t really a complicated one so it’s great to see some of the magic revealed here.

If you haven’t seen aliens coming through your walls in an app like RoboRaid then there’s a video here which shows it in action and also talks a little about what the device and the app developer are doing to pull off the illusion;

Inside of the HoloToolkit, there are some pieces that can help with experimenting with this type of effect and so I went off and got the toolkit (as per this video) and I brought the sections Build, Input, UI, Utilities and SpatialMapping into my project and set it up (as per this video).

Within the Utilities section of the toolkit there is a pre-baked scene called WindowOcclusion;

and that has a camera, 4 quads and 5 cubes set up to provide the ‘looking through a window’ effect that the Case Study talks about;

and those 4 quads are kind of ‘interesting’ in that they aren’t immediately visible here;

but they are being shaded so as to occlude the content behind them;

and so the user gets the illusion of looking through the window as it’s essentially the hole left by the 4 quads surrounding it which occlude everything else.

For me, that illusion works best if that window is positioned on a wall whereas this pre-baked scene places the window approx 1.7m in front of wherever the user was looking when the app started. It might be relatively simple though to use spatial mapping and the ‘tap to place’ behaviour to change that and have the user position the window onto a surface.

I thought I’d give that a spin…

Adding Tap to Place

I took the two components from that scene, collected them into an empty game object and then made a prefab from that in Unity called WindowAndContent.

and then I add a simple blank placeholder (at the origin) and a quad into my scene 2m in front of the origin. The idea of the quad is to give me something that I can tap and place onto a wall in order to position where I want my window (and the content beyond the window) to appear;

and so the intended process is going to be something like;

• Create quad 2m in front of the user.
• Allow the user to tap on the quad.
• Have the quad follow the user’s gaze around their walls (this is the tap to place behaviour).
• Allow the user to tap.
• Remove the quad and replace it with the WindowAndContent prefab positioned at the same place and oriented the same way.

To be able to position the quad and the window onto a wall involves spatial mapping and so I made sure that I had spatial perception switched on as a UWP capability and I added the SpatialMapping prefab from the HoloToolkity as you can see in the screenshot above.

I then gave my placeholder object a bunch of behaviours in order to facilitate using the Tap to Place script on my quad;

and then I added the Tap to Place script to that quad;

but I hacked that script ever so slightly in order to change two things;

1. By default, the script makes the spatial mapping mesh visible when the object has been tapped and is following the user’s gaze but I didn’t want this so I took it out.
2. I added a line or two of code such that when the object is placed, the script would fire a Placed event.

I wanted that Placed event so that I could add a script to my quad as shown below;

and that script handles the Placed event on my modified TapToPlace component in order to try and get rid of the quad and replace it with the WindowAndContent prefab so that the window would appear where the quad had been positioned. There’s probably a better way of doing this but here’s the script that I used;

using HoloToolkit.Unity;
using UnityEngine;

public class QuadScript : MonoBehaviour
{
  public Transform prefab;

  void Start()
  {
    this.tapToPlace = this.GetComponent<TapToPlace>();
    this.tapToPlace.Placed += this.OnPlaced;
  }

  void OnPlaced(object sender, System.EventArgs e)
  {
    // We're done now.
    this.tapToPlace.Placed -= this.OnPlaced;
    this.tapToPlace = null;

    var windowAndContent = Instantiate(prefab);
    windowAndContent.transform.parent = this.transform.parent;
    windowAndContent.transform.localPosition = this.transform.localPosition;
    windowAndContent.transform.forward = this.transform.forward;
    this.GetComponent<MeshRenderer>().enabled = false;
  }
  TapToPlace tapToPlace;
}

and, sure enough, I can now display a quad and then tap to position it on a wall in my environment and when I tap again it gets replaced with a window that I can look through into a ‘virtual world’ on the other side of that wall.

But that ‘virtual world’ is just a cube, I need something more interesting when I look through my window…

Making the View from the Window more Interesting

I figured that I’d make the window a bit larger and then would look at seeing if I could put some more interesting content on the other side of it.

I went out to the Unity Asset Store and found this set of town models and materials;

and it came with a nice scene or two demonstrating lots of the models and so I chopped one of those down in terms of the size of the scene and positioned it on the other side of my window.

Below is a screenshot of what I ended up with – you can see the scene and the relative position of my ‘window’ positioned such that everything in the scene is ‘in front’ of the window;

and this screenshot shows the reverse view of the quads shaded to occlude the buildings from the viewer on the other side of the window so that the window provides the only view;

and I baked all of this into my WindowAndContent prefab (replacing the existing cube) such that when I placed my quad on a wall this set of buildings would be instantiated on the other side of my window.

Trying this out, it all works surprisingly well.

What doesn’t work quite so well is showing how this looks in captured screenshots but hopefully the pictures below give an idea of two views through the same window.

It’s a very convincing effect when you’re actually using it – here’s a view taken from the right of the window. Note that you can see that I’ve left a little gap to the upper left of the window frame which needs closing in the Unity designer – i.e. that’s an artefact that I can fix rather than something from the device;

and here’s a view when I’m standing more to the centre of the window;

This is such a clever effect – I’m going to experiment some more but I’m also going to work way my through some more of those case studies, there’s a lot to learn…