Windows 10, UWP, HoloLens & A Simple Two-Way Socket Library

NB: The usual blog disclaimer for this site applies to posts around HoloLens. I am not on the HoloLens team. I have no details on HoloLens other than what is on the public web and so what I post here is just from my own experience experimenting with pieces that are publicly available and you should always check out the official developer site for the product documentation.

This post is around some code that I wrote to enable a specific, simple demo scenario where I would have 2 Windows 10 devices (including HoloLens devices) find and talk to each other on a local WiFi network with fairly low minimal infrastructure/setup.

There are many other ways of achieving what I did here so this is just my own implementation of something fairly common to suit my scenario rather than some radical new invention and it’s far from production ready because it doesn’t deal with the suspend/resume cycle of UWP properly and attempt to do the right work to keep sockets running when the application is suspended (as per these docs).

I’m really just putting it together for a demo where I want to connect two HoloLens devices so that I can experiment with ‘sharing holograms’…

Sharing Holograms – Backdrop

One of the really interesting/amazing things that you can do in a HoloLens application is to use the built in connectivity options of the device (including Bluetooth and more specifically WiFi) to connect multiple devices such that a number of users can share an experience across devices being used in the same space.

There are many examples of this out there on the web including in videos like this keynote video below which shows a couple working on their kitchen design;

Sharing Holograms – HoloToolkit Support

This notion of ‘shared holograms’ is one that is common enough to have been built into the HoloToolkit with its sharing support which also features in this Holographic Academy lesson;

The toolkit’s method for sharing holograms is to set up a network configuration with a PC acting as a server in order to distribute holographic data between participating HoloLens devices and keep them in sync. This is a great solution and one that you should look at if you’re interested in sharing holograms in this or a similar way.

I’ve used this solution before and it works very well but I wanted to go with my own solution here for a couple of reasons;

I wanted to make sure I understood the implementation by (essentially) writing small pieces of it myself from the ground up.
I want to simplify the solution as I was only thinking about two-way comms.

A Simple Socket Library for Two-Way Comms

I had a simpler scenario in mind where I didn’t want to have to rely on a PC to act as a server and I only needed to connect 2 devices rather than many. I also didn’t want to have to enter host names, IP addresses or port numbers into some UI in order to get connectivity up and running.

Towards that end, I built out a simple library that I plan to write a little about in the rest of this post and then I’ll follow up with a subsequent post where that library gets used in some kind of basic demo that shares holograms across devices.

In order to do that, I first looked to the PeerFinder API in the Universal Windows Platform which I thought would provide the perfect solution in that those APIs are specifically about two devices running the same app on the same network discovering each other for further communication.

However, as I found in this blog post, those APIs don’t look like they function on HoloLens.

I might have also looked at something like UWP App Services with Project Rome but that didn’t suit my scenario because that’s specifically about communicating between apps on devices that belong to the same user whereas I wanted to communicate between the same app but on (potentially) any user’s device.

I wrote some code to try and enable the scenario that I wanted to hit with two devices operating in a peer manner such that;

The user of a device selects whether they want to create/connect to a connection.
1. This choice can be made before or after the companion device comes along.
The creating device creates a TCP socket and advertises its details over Bluetooth LE (like a beacon).
The connecting device looks for advertisements over Bluetooth LE.
Once the devices find each other, they connect up the socket and stop advertising/listening over Bluetooth LE.
Messages are interchanged over the TCP socket. At the time of writing, these can be;
1. Strings.
2. Byte arrays.
3. Serialized objects (via JSON.NET).
As/when the socket drops, the process can be repeated from step 1 again.

Note that this is just a set of choices that I made and that I could have gone in other directions. For example;

Rather than use Bluetooth LE to advertise a TCP socket, I could have used a UDP multicast group.
Rather than use a TCP socket, I could have used UDP sockets but I felt that my comms were best suited to a connected, streamed socket between 2 parties here.

Implementation – A library for both XAML UI and Unity UI

The implementation that I spent an hour or two on sits on UWP APIs and I wanted to try and ensure that it would work on any UWP device including the HoloLens and also that I could make use of it from regular UWP code as I might write in a XAML based 2D UI but also from 3D Unity applications for HoloLens as my primary purpose here is to share holograms from one instance of my app on one device to another instance on another device.

In writing this code, I made some choices around sticking with the coding pattern that I would usually use for UWP applications which ends up relying quite heavily on the async/await capabilities of C# that marry well with the async APIs that you see across the UWP although I debated quite a lot about whether I should go down this route because it isn’t perhaps the standard coding approach that you might take inside of the Unity environment.

Because I’m only targeting UWP, I feel that this is a reasonable trade-off although it does lead to a bit of;

#if UNITY_UWP

inside of my code to keep the Unity editor happy. If you’ve not bumped up against these limitations then the essence is that the Unity editor can’t cope with a script that looks like this;

using System;
using UnityEngine;

public class Placeholder : MonoBehaviour
{
  private async void Start()
  {
    await Task.Delay(TimeSpan.FromSeconds(1));
  }
}

because the C# language in Unity is V4 and so async/await etc. aren’t available there and the version of .NET doesn’t have Task.

However, because of the way that the build process works with Unity, if I accept that I’m only building for UWP and none of Unity’s other platforms I can write conditional code so that the Unity editor doesn’t know what I’m up to and I can still execute pretty much what I want at runtime. For example (and, yes, I know this example is ugly);

public class Placeholder : MonoBehaviour
{
  void Start()
  {
    this.InternalStartAsync();
  }
#if UNITY_UWP
  async
#endif
  void InternalStartAsync()
  {
#if UNITY_UWP

    // Thread before?

    await Task.Delay(TimeSpan.FromSeconds(1));
 
    // Thread after?
#endif
  }
}

That should satisfy both the editor and build out to do what I want at runtime. There’s more written about this elsewhere on the web but to my mind it also comes with two other implications;

The implication of having async methods that no caller ever awaits – as in the code above where the Start() method does not await the InternalStartAsync() method.
The implication of the (possible) thread switching between the 2 comments marked ‘Thread before?’ and ‘Thread after?’.

This is unlike a UWP XAML scenario where, by default, using await on a UI thread would cause the continuation of the async function to be dispatched back onto the UI thread because the framework sets up a SynchronizationContext to make that happen.

With some of that in mind, I set off building something fairly simple that I called the AutoConnectMessagePipe …

Connecting 2 Machines with the AutoConnectMessagePipe

I wanted the consumption of this API to be simple and I managed to boil down my API surface to offer an experience something like the code below in terms of getting a pipe connected between two devices;

    async Task<bool> ConnectPipeAsync()
    {
      // True if we want to advertise the pipe, false otherwise
      AutoConnectMessagePipe pipe = new AutoConnectMessagePipe(true);

      // We wait for the pipe to be connected providing a timeout if
      // we like.
      await pipe.WaitForConnectionAsync(TimeSpan.FromMilliseconds(-1));

      return (pipe.IsConnected);
    }

In order to function, the app is going to need the Bluetooth UWP capability and either/both of the ‘Private networks’ or ‘Internet Client/Server’ capabilities to be switched on.

Once the pipe is connected, the API makes it fairly simple to send a string or a byte array;

      await pipe.SendStringAsync("Hello");
      await pipe.SendBytesAsync(new byte[] { 1, 2, 3 });

and I made a somewhat arbitrary decision around sending strongly typed objects to the other end of the pipe in that I defined a simple (intended to be abstract) base class;

  public class MessageBase
  {
  }

and so sending a message would involve deriving it from this base class;

public class MyMessage : MessageBase
  {
    public string MyProperty { get; set; }
  }

and sending an instance over the network;

await pipe.SendObjectAsync(
        new MyMessage()
        {
          MyProperty = "Foo"
        }
      );

Note that it’s absolutely my intention here that the same process is on both ends of the wire so I don’t have to worry about the data types in question not being available for de-serialization, this library is really for an app to talk to another instance of itself on another machine.

Note also that I really would have liked the BinaryFormatter because it doesn’t, as far as I remember, need up-front knowledge of the types that it’s dealing with but it doesn’t exist in the UWP. I glanced at one or two of the available serialization libraries out there and I also tried to work with the DataContractSerializer for a while but I don’t think that works on HoloLens right now and so I fell back to using JSON.NET (not that this is a bad thing Smile ).

In terms of reading messages, the pipe class offers the opportunity to read a single message or to sit in an async loop dispatching messages until the socket connection goes away and the code to do that looks something like;

      // We will not 'return' from this until the socket is closed, remembering what
      // 'return' means in an async setting!
      await pipe.ReadAndDispatchMessageLoopAsync(
        (messageType, body) =>
        {
          switch (messageType)
          {
            case MessageType.Buffer:
              byte[] bits = (byte[])body;
              break;
            case MessageType.String:
              string text = (string)body;
              break;
            case MessageType.SerializedObject:
              MyMessage typedObject = body as MyMessage;
              break;
            default:
              break;
          }
        }

and that’s pretty much the interface into my library.

I wrote a few classes that underpin this which probably need a bit of refinement and a lot more testing but they connect together in the manner below;

The source for this library is on github here .

With this in place, how would this look and operate inside of a 2D UWP app?

Usage – Inside 2D UWP App

I put together a simple, blank UWP app and added a user control with a UI which easily allows me to select the connect/advertise option;

and once the application is connected, it displays 3 coloured buttons;

and pressing the button of a particular colour changes the background colour of the app that’s on the other end of the connection;

That’s a fairly simple piece of XAML defining the UI;

<UserControl
    x:Class="XamlTestApp.MainControl"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:local="using:XamlTestApp"
    xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
    xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
    mc:Ignorable="d"
    d:DesignHeight="300"
    d:DesignWidth="400">
    <UserControl.Resources>
        <local:InverseVisibilityConverter
            x:Name="negate" />
    </UserControl.Resources>
    <Grid>
        <Grid.Resources>
            <Style
                TargetType="Rectangle">
                <Setter
                    Property="Width"
                    Value="24" />
                <Setter
                    Property="Height"
                    Value="24" />
                <Setter
                    Property="Stroke"
                    Value="Black" />
            </Style>
            <Style
                TargetType="Button">
                <Setter
                    Property="Margin"
                    Value="2" />
                <Setter
                    Property="BorderBrush"
                    Value="Black" />
                <Setter
                    Property="BorderThickness"
                    Value="1" />
                <Setter
                    Property="HorizontalAlignment"
                    Value="Center" />
            </Style>
        </Grid.Resources>
        <Grid.Background>
            <SolidColorBrush
                Color="{x:Bind BackgroundColour,Mode=OneWay,FallbackValue=White}" />
        </Grid.Background>
        <Viewbox
            Margin="96">
            <StackPanel
                HorizontalAlignment="Center"
                VerticalAlignment="Center">
                <StackPanel
                    HorizontalAlignment="Center"
                    VerticalAlignment="Center"
                    Orientation="Horizontal"
                    Visibility="{x:Bind Path=IsConnected, Mode=OneWay,Converter ={StaticResource ResourceKey=negate}}">
                    <StackPanel Margin="8">
                        <Button
                            Content=""
                            FontFamily="Segoe MDL2 Assets"
                            Click="{x:Bind OnAdvertise}"/>
                        <TextBlock
                            Text="advertise"
                            TextAlignment="Center" />
                    </StackPanel>
                    <StackPanel Margin="8">
                        <Button
                            Content=""
                            FontFamily="Segoe MDL2 Assets"
                            Click="{x:Bind OnConnect}" />
                        <TextBlock
                            Text="connect"
                            TextAlignment="Center" />
                    </StackPanel>
                </StackPanel>
                <StackPanel
                    Orientation="Horizontal"
                    Visibility="{x:Bind IsConnected,Mode=OneWay}"
                    Margin="0,8,0,0"
                    HorizontalAlignment="Center">
                    <Button
                        Template="{x:Null}"
                        Click="{x:Bind OnRed}">
                        <Rectangle
                            Fill="Red" />
                    </Button>
                    <Button
                        Template="{x:Null}"
                        Click="{x:Bind OnGreen}">
                        <Rectangle
                            Fill="Green" />
                    </Button>
                    <Button
                        Template="{x:Null}"
                        Click="{x:Bind OnBlue}">
                        <Rectangle
                            Fill="Blue" />
                    </Button>
                </StackPanel>
            </StackPanel>
        </Viewbox>
    </Grid>
</UserControl>

accompanied by (mostly) a single code-behind file which is pretty simple in its implementation;

//#define USE_OBJECTS
namespace XamlTestApp
{
  using SimpleUwpTwoWayComms;
  using System;
  using System.ComponentModel;
  using System.Runtime.CompilerServices;
  using System.Threading.Tasks;
  using Windows.UI;
  using Windows.UI.Xaml.Controls;

#if USE_OBJECTS
  public class ColourMessage : MessageBase
  {
    public byte Red { get; set; }
    public byte Green { get; set; }
    public byte Blue { get; set; }
  }
#endif

  public sealed partial class MainControl : UserControl, INotifyPropertyChanged
  {
    public event PropertyChangedEventHandler PropertyChanged;

    public MainControl()
    {
      this.InitializeComponent();
      this.Disconnect();
    }
    public Color BackgroundColour
    {
      get {
        return (this.backgroundColour);
      }
      set
      {
        if (this.backgroundColour != value)
        {
          this.backgroundColour = value;
          this.FirePropertyChanged();
        }
      }
    }
    Color backgroundColour;

    public bool IsConnected
    {
      get
      {
        return (this.isConnected);
      }
      set
      {
        if (this.isConnected != value)
        {
          this.isConnected = value;
          this.FirePropertyChanged();
        }
      }
    }
    bool isConnected;

    public void OnAdvertise()
    {
      this.OnInitialise();
    }
    public void OnConnect()
    {
      this.OnInitialise(false);
    }
    public async void OnInitialise(bool advertise = true)
    {
      this.pipe = new AutoConnectMessagePipe(advertise);

      await this.pipe.WaitForConnectionAsync(TimeSpan.FromMilliseconds(-1));

      this.IsConnected = this.pipe.IsConnected;

      if (this.IsConnected)
      {
        await this.pipe.ReadAndDispatchMessageLoopAsync(this.MessageHandler);
      }
    }
    void Disconnect()
    {
      this.IsConnected = false;
      this.BackgroundColour = Colors.White;
    }
    public async void OnRed()
    {
      await this.OnColourAsync(Colors.Red);
    }
    public async void OnGreen()
    {
      await this.OnColourAsync(Colors.Green);
    }
    public async void OnBlue()
    {
      await this.OnColourAsync(Colors.Blue);
    }
    async Task OnColourAsync(Color colour)
    {
#if USE_OBJECTS
      ColourMessage message = new ColourMessage()
      {
        Red = colour.R,
        Green = colour.G,
        Blue = colour.B
      };
      await this.pipe.SendObjectAsync(message);
#else
      await this.pipe.SendBytesAsync(
        new byte[] { colour.R, colour.G, colour.B });
#endif
    }
    void MessageHandler(MessageType messageType, object messageBody)
    {
#if USE_OBJECTS
      if (messageType == MessageType.SerializedObject)
      {
        var msg = messageBody as ColourMessage;

        if (msg != null)
        {
          this.BackgroundColour = Color.FromArgb(0xFF, msg.Red, msg.Green, msg.Blue);
        }
      }
#else
      // We just handle byte arrays here.
      if (messageType == MessageType.Buffer)
      {
        var bits = (byte[])messageBody;
        this.BackgroundColour = Color.FromArgb(0xFF, bits[0], bits[1], bits[2]);
      }
#endif
    }
    void FirePropertyChanged([CallerMemberName] string propertyName = null)
    {
      this.PropertyChanged?.Invoke(this,
        new PropertyChangedEventArgs(propertyName));
    }
    AutoConnectMessagePipe pipe;
  }
}

and I checked that project into the github repository as well and this 2D app works fine on HoloLens.

That code can be compiled in two different ways (depending on the USE_OBJECTS symbol) so that it either sends byte arrays over the network or a small object serialized with JSON.NET.

Usage – Inside 3D HoloLens Unity App

I made a blank 3D Unity project and set it up for HoloLens development pretty much as I do at the start of this video so as to configure the project and the scene for HoloLens development.

I also made sure that I had the UWP capabilities for Bluetooth and Private networks and I added my new simple comms library as an assembly to my project’s assets as I wrote about in this post and I took the same approach to Newtonsoft.Json;

With that set up, I made a basic UI as below;

and essentially what I’ve got here are;

An empty Placeholder GameObject.
A Basic Cursor (from the HoloToolkit – more or less as I do in the ‘Adding Gaze’ section of this post).
A Canvas holding two panels

One panel has advertise/connect buttons.
One panel has red/green/blue buttons.

A Cube that I can use to display colours.

and then I’ve added some scripts to the Placeholder here – 4 from the HoloToolkit to give me a tag-along behaviour and the next 3 scripts are dependencies for the Basic Cursor;

The last script listed above is my own Placeholder.cs script which is the interesting piece here and you might have noticed from the screenshot above that I have put public properties onto that script to make the two panels and the cube accessible to the code. Here’s that script;

using System;
using System.Collections;
using System.Collections.Generic;
using UnityEngine;
#if UNITY_UWP && !UNITY_EDITOR
using SimpleUwpTwoWayComms;
using System.Threading.Tasks;
#endif

public class Placeholder : MonoBehaviour
{
  public GameObject panelConnection;
  public GameObject panelColours;
  public GameObject cube;

  public void OnAdvertise()
  {
#if UNITY_UWP && !UNITY_EDITOR
    this.OnInitialiseAsync();
#endif
  }
  public void OnConnect()
  {
#if UNITY_UWP && !UNITY_EDITOR
    this.OnInitialiseAsync(false);
#endif
  }
  public void OnRed()
  {
    this.OnColour(Color.red);
  }
  public void OnGreen()
  {
    this.OnColour(Color.green);
  }
  public void OnBlue()
  {
    this.OnColour(Color.blue);
  }
  public void OnColour(Color colour)
  {
    // Convert 0 to 1 values into bytes so that we can be compatible with the 2D XAML
    // app.
    var message = new byte[]
    {
      (byte)(colour.r * 255.0f),
      (byte)(colour.g * 255.0f),
      (byte)(colour.b * 255.0f)
    };

#if UNITY_UWP && !UNITY_EDITOR
    this.OnColourAsync(message);
#endif
  }
  void Dispatch(Action action)
  {
    UnityEngine.WSA.Application.InvokeOnAppThread(() =>
    {
      action();
    },
    false);
  }
#if UNITY_UWP && !UNITY_EDITOR
  async Task OnColourAsync(byte[] bits)
  {
    await this.pipe.SendBytesAsync(bits);
  }

  async Task OnInitialiseAsync(bool advertise = true)
  {
    if (this.pipe == null)
    {
      this.pipe = new AutoConnectMessagePipe(advertise);
    }

    await this.pipe.WaitForConnectionAsync(TimeSpan.FromMilliseconds(-1));

    if (pipe.IsConnected)
    {
      this.TogglePanels(false);
      await this.pipe.ReadAndDispatchMessageLoopAsync(this.MessageHandler);
      this.TogglePanels(true);
    }
  }
  void TogglePanels(bool connectionPanel)
  {
    this.Dispatch(
      () =>
      {
        this.panelConnection.SetActive(connectionPanel);
        this.panelColours.SetActive(!connectionPanel);
      }
    );
  }
  void MessageHandler(MessageType messageType, object messageBody)
  {
    // We just handle byte arrays here.
    if (messageType == MessageType.Buffer)
    {
      var bits = (byte[])messageBody;

      if (bits != null)
      {
        this.Dispatch(() =>
          {
            this.cube.GetComponent<Renderer>().material.color =
              new Color(
                (float)(bits[0]) / 255.0f,
                (float)(bits[1]) / 255.0f,
                (float)(bits[2] / 255.0f));
          }
        );
      }
    }
  }
  AutoConnectMessagePipe pipe;
#endif
}

and I’ve committed that project to the github repository as well.

Wrapping Up

In as far as it goes, this code all seems to work reasonably well and here’s a quick test video of me using it to communicate from the 2D XAML app on my phone to the 3D Unity app on my HoloLens but the 2D app would also work on HoloLens as it does on PC and (presumably) other devices that I haven’t yet tried.

I’ll come back with a follow on post which is more about using this library to share holograms from one device to another but this post is already way too long so I’ll stop here Smile

Mike Taulty

I do some developer stuff on Microsoft technologies in the UK

Windows 10, UWP, HoloLens & A Simple Two-Way Socket Library

Sharing Holograms – Backdrop

Sharing Holograms – HoloToolkit Support

A Simple Socket Library for Two-Way Comms

Implementation – A library for both XAML UI and Unity UI

Connecting 2 Machines with the AutoConnectMessagePipe

Usage – Inside 2D UWP App

Usage – Inside 3D HoloLens Unity App

Wrapping Up

3 thoughts on “Windows 10, UWP, HoloLens & A Simple Two-Way Socket Library”

Sharing Holograms – Backdrop

Sharing Holograms – HoloToolkit Support

A Simple Socket Library for Two-Way Comms

Implementation – A library for both XAML UI and Unity UI

Connecting 2 Machines with the AutoConnectMessagePipe

Usage – Inside 2D UWP App

Usage – Inside 3D HoloLens Unity App

Wrapping Up

Share this:

Like this:

Related

3 thoughts on “Windows 10, UWP, HoloLens & A Simple Two-Way Socket Library”