Windows 10, UWP, QR Code Scanning with ZXing and HoloLens

NB: The usual blog disclaimer for this site applies to posts around HoloLens. I am not on the HoloLens team. I have no details on HoloLens other than what is on the public web and so what I post here is just from my own experience experimenting with pieces that are publicly available and you should always check out the official developer site for the product documentation.

I wrote this app for Windows 8.0 over 4 years ago and ported it to Windows 10 over a few blog posts around a year ago. The app sits on top of the ZXing library for barcode/QRcode scanning and as part of the work that I did here I moved the code to sit on top of the Nuget package ZXing.NET.

At the time, I wanted to extract out the QR code scanning code that I wrote but I never quite ‘got around to it’. I think I did once extract some pieces and send them to someone in an email as they’d asked me directly if they could borrow it but I suspect that the code at the time was coupled into visually previewing video frames in a XAML based app using CaptureElement.

Just the other week, I saw a post on the HoloLens developer forums;

Barcode/QR code detection

which was looking to do some kind of QR code scanning on HoloLens and it prompted me to have another try at extracting this little bit of code so as to produce some code which could process preview video frames from a camera without necessarily having to display those preview frames anywhere.

In writing this post, I’ve managed to boil the QR scanning code down to the point where it might be as simple as a piece of code like this one taken from the 2D XAML test app that I wrote to try things out;

      ZXingQrCodeScanner.ScanFirstCameraForQrCode(
        result =>
        {
          // TODO: do something with the QR code result.
        },
        TimeSpan.FromSeconds(30));

and so it’s a simple static call passing a callback function which handles any result produced and a timespan after which to timeout if no QR code is found.

There’s an assumption that the first camera found on the system should be used for QR code scanning but the classes that underpin this would allow for taking a more flexible approach and that ScanFirstCameraForQrCode function expands out into the following steps below;

  public static class ZXingQrCodeScanner
  {
    public static async void ScanFirstCameraForQrCode(
      Action<Result> resultCallback, 
      TimeSpan timeout)
    {
      Result result = null;

      var mediaFrameSourceFinder = new MediaFrameSourceFinder();      

      // We want a source of media frame groups which contains a color video
      // preview (and we'll take the first one).
      var populated = await mediaFrameSourceFinder.PopulateAsync(
        MediaFrameSourceFinder.ColorVideoPreviewFilter,
        MediaFrameSourceFinder.FirstOrDefault);

      if (populated)
      {
        // We'll take the first video capture device.
        var videoCaptureDevice =
          await VideoCaptureDeviceFinder.FindFirstOrDefaultAsync();

        if (videoCaptureDevice != null)
        {
          // Make a processor which will pull frames from the camera and run
          // ZXing over them to look for QR codes.
          var frameProcessor = new QrCaptureFrameProcessor(
            mediaFrameSourceFinder,
            videoCaptureDevice,
            MediaEncodingSubtypes.Bgra8);

          // Remember to ask for auto-focus on the video capture device.
          frameProcessor.SetVideoDeviceControllerInitialiser(
            vd => vd.Focus.TrySetAuto(true));

          // Process frames for up to 30 seconds to see if we get any QR codes...
          await frameProcessor.ProcessFramesAsync(timeout);

          // See what result we got.
          result = frameProcessor.QrZxingResult;
        }
      }
      // Call back with whatever result we got.
      resultCallback(result);
    }
  }

where we proceed to;

  1. Find details of a MediaFrameSourceGroup which can deliver color video frames in preview form
  2. Find the first DeviceInformation for a video capture device.
  3. Create a MediaCapture and use it to create a MediaFrameReader such that we can pull preview frames from the device.
  4. Process those frames for up to 30s, feeding them into the ZXing BarcodeReader class to see if we can find QR codes.
  5. Return any result.

All of that code and the supporting classes are in this github repository;

Source Code

and there’s a simple 2D test application for XAML in that repo as well. The classes in that repo aren’t ‘comprehensive’ in the sense that they could be expanded to offer lots more options but I just focused on re-structuring a few pieces that I already largely had to hand.

Trying this out on the PC

Here’s a quick screen capture of trying to make use of this test app on a PC in order to scan a QR code that’s being displayed on my phone using the forward facing camera on a Surface Book. There’s not a lot to ‘see’ here but I include it for completeness!

Trying this out on HoloLens in 2D

The same code runs on the HoloLens as a 2D app but because it needs to take over the camera in order to scan for QR codes I think it’s bit “tricky” to capture that to a video that I can include here to show you that it works just like it does on the PC Smile

Trying this out on HoloLens in 3D in Unity

To try this out in Unity, I made a blank HoloLens project in Unity 5.5 and imported pieces of the HoloToolkit (Build, Input, UI, Utilities) and then set up my project for HoloLens development as I do at the start of this video so as to configure the project and the scene for HoloLens development. I also ensured that from a UWP capabilities perspective my app had access to the webcam and microphone.

From there, I followed my own blog post and I added my new QR code class library (the Debug x86 build) and the ZXing winmd file into the assets for my Unity project;

Sketch2 

and then I tried to make sure that the settings on these 2 files were such that I might be able to build out my code at least for x86;

Sketch3

Sketch4

and then used the 3DTextPrefab from the HoloToolkit to make a very cheap and cheerful form of ‘tag along’ heads-up-display;

image

and a blank game object acting as a Placeholder with the script below attached to it;

using System;
using UnityEngine;

public class Placeholder : MonoBehaviour
{
  public Transform textMeshObject;

  private void Start()
  {
    this.textMesh = this.textMeshObject.GetComponent<TextMesh>();
    this.OnReset();
  }
  public void OnScan()
  {
    this.textMesh.text = "scanning for 30s";

#if !UNITY_EDITOR
    MediaFrameQrProcessing.Wrappers.ZXingQrCodeScanner.ScanFirstCameraForQrCode(
        result =>
        {
          UnityEngine.WSA.Application.InvokeOnAppThread(() =>
          {
            this.textMesh.text = result?.Text ?? "not found";
          }, 
          false);
        },
        TimeSpan.FromSeconds(30));
#endif
  }
  public void OnReset()
  {
    this.textMesh.text = "say scan to start";
  }
  TextMesh textMesh;
}

and I used the designer to make sure that the textMeshObject was set to point to the 3DTextPrefab and I also wired up a HoloToolkit KeywordManager in order to direct the spoken commands “scan” and “reset” into the respective methods;

Sketch5

and that seems to work reasonably well in the sense that I can wander around a room, say “scan” and then scan a QR code from a reasonable distance away although I’ve not done any testing to see what sort of limits I might be able to get down to in terms of how small those QR codes might be and whether that makes them usable in a real-world scenario.

I tried to stitch together a video of how this looks below working around the limitation that both my app and the system can’t realistically record from the camera at the same time;

Whether this is what the folks in the HoloLens forums thread were looking for, I’m not sure – they may have been wanting to find QR codes in the 3D world rather than activate the webcam in order to scan QR codes that way but at least it caused me to put that code somewhere where it’s easier to re-use in the future and also to have a think about using this sort of regular UWP code in a 3D Unity environment on HoloLens which was interesting in itself for me.