Windows 10 1607, UWP and Project Rome–Transferring Browser Tabs Across Devices

I was inspired by this article that I saw on the Windows Blog in the past week or so;

Cross-device experiences with Project Rome

to revisit ‘Project Rome’ that I’ve had a look at in these previous posts;

Project Rome posts.

The article is a great read and, having it read it once,  I later went back to it to see if I could get hold of the ‘Contoso Music’ app that it talks about but then I realised that ‘Contoso Music’ is being used in the article to talk about concepts rather than being a code sample in itself and so I set off to build my own sample.

For me, the Rome APIs in Windows 10 1607 today enable;

  1. A developer’s code to discover the user’s device graph that they’ve registered under the same Microsoft Account.
  2. A developer’s code to do a remote launch of a (custom/standard) URI on a remote system.
  3. A developer’s code to invoke a remote app service on a remote system.

Naturally, all under the control of the user and under the control of the remote app being invoked.

I wanted to try this out for a scenario that I find I’m often doing where I’ve opened up a set of tabs in my browser and I want to transfer that set of tabs to another device whether that be a PC or a phone or whatever.

I wrote a basic browser in order to try that out and I demo it in the video below;

There’s a few things that I found interesting in build that and so I thought I’d share those here.

The Device Discovery Code is Tiny

Here’s the code that I wrote in order to figure out which remote systems I have available to me. I like how short and neat it can be;

    async Task StartRemoteSystemDetectionAsync()
    {
      var result = await RemoteSystem.RequestAccessAsync();

      if (result == RemoteSystemAccessStatus.Allowed)
      {
        this.RemoteSystems = new ObservableCollection<RemoteSystem>();
        this.remoteWatcher = RemoteSystem.CreateWatcher();
        this.remoteWatcher.RemoteSystemAdded += (s, e) =>
        {
          this.Dispatch(() =>
          {
            this.AddRemoteSystem(e.RemoteSystem);
          });
        };
        this.remoteWatcher.Start();
      }
    }

I ultimately data-bind the DisplayName, Kind and IsAvailableByProximity properties of the RemoteSystem object into a data template within the ComboBox that sits on the UI and it’s as simple as that.

The Remote App Service Invocation Code is Also Tiny

When it comes time to invoke the remote app service, that’s pretty easy to do as well, mostly relying on AppServiceConnection.OpenRemoteAsync() to do the work for me. In my app, the code looks like this;

    internal async void OnTransfer()
    {
      var errorString = string.Empty;
      this.IsTransferring = true;
      this.TransferStatus = "connecting...";

      using (var connection = new AppServiceConnection())
      {
        connection.PackageFamilyName = App.PackageFamilyName;
        connection.AppServiceName = App.AppServiceName;

        var remoteRequest = new RemoteSystemConnectionRequest(this.SelectedRemoteSystem);

        var result = await connection.OpenRemoteAsync(remoteRequest);

        if (result == AppServiceConnectionStatus.Success)
        {
          this.TransferStatus = "Connected, calling...";

          var message = new ValueSet();
          var content = this.Serialize();
          message[App.AppServiceParameterKey] = content;

          var response = await connection.SendMessageAsync(message);

          if (response.Status != AppServiceResponseStatus.Success)
          {
            this.TransferStatus = $"Failed to call - status was [{response.Status}]";
          }
          this.TransferStatus = "Succeeded";
        }
        else
        {
          this.TransferStatus = $"Failed to open connection with status {result}";
        }
        this.transferStatus = "Closing";
      }
      // time for the display of errors etc. to be seen.
      await Task.Delay(2000);

      this.IsTransferring = false;
    }
  }

Some of that might seem a little opaque without seeing the code for the whole app but it’s essentially;

  1. Create an AppServiceConnection, filling in the details of the PackageFamlyName and AppServiceName.
  2. Create a RemoteSystemConnectionRequest to the RemoteSystem that is selected in the UI.
  3. Make a call to OpenRemoteAsync()
  4. Create a ValueSet with the serialized list of web page URLs in it.
  5. Send it over the connection.
  6. Add a few pieces to update the UI and delay a little so that I can visually see whether it thinks it’s working or not.

and that’s pretty much it although it’s important to flag that the app manifest for the app has to advertise the right app service in the right way via;

      <Extensions>
        <uap:Extension Category="windows.appService">
          <uap3:AppService Name="OpenTabsService" SupportsRemoteSystems="true"/>
        </uap:Extension>
      </Extensions>

to flag that it’s available for remoting.

Using LeavingBackground and EnteringBackground in my App Class

In this app, I tried the technique of creating my UI when the app goes through its LeavingBackground event and destroying that UI when it hits its EnteringBackground event. I also saved state for the app when it hits EnteringBackground. This was the first time I’d tried this and it worked fine for me but I wasn’t sure whether my choice to destroy the UI at the point where the app moved into the background was a great idea because it might mean that a user who frequently moved the app foreground->background got frustrated with the UI keep rebuilding itself so perhaps I’m being too enthusiastic there.

As an example, my App.EnteringBackground handler does;

  async void OnEnteringBackground(object sender, EnteredBackgroundEventArgs e)
    {
      // NB: getting rid of the UI here means that if the user opens up
      // N tabs then when they come back to the app those tabs will still
      // be there but will all start loading fresh again. That might not
      // be the best user experience as it loses their position on the
      // page.
      if (BrowserViewModel.ForegroundInstance != null)
      {
        var deferral = e.GetDeferral();

        try
        {
          var serialized = BrowserViewModel.ForegroundInstance.Serialize();
          await Storage.StoreAsync(serialized);
        }
        finally
        {
          deferral.Complete();
        }
      }

      Window.Current.Content = null;
      this.background = true;
    }

and so it serializes any state (i.e. the URLs of the web pages) and then it clears the contents of the Window entirely and sets a flag to note that we are now in the background. When I come back from the background I have code that does the reverse;

    void OnLeavingBackground(object sender, LeavingBackgroundEventArgs e)
    {
      this.EnsureUI();
      this.background = false;
    }

and EnsureUI is pretty much the boilerplate code from a blank UWP template in that it creates a Frame, puts it into the Window and navigates it to a MainPage.

Using the Single Process Execution Model for Background Tasks

This app also uses the single process execution model for its background task implementation, override App.OnBackgroundActivated;

protected override void OnBackgroundActivated(BackgroundActivatedEventArgs args)
{
}

what I found interesting in there is that I can now check the value of my background flag to decide what to do with an incoming app service request as below;

 async Task ProcessAppServiceRequestAsync(AppServiceRequestReceivedEventArgs args)
    {
      var webPages = args.Request.Message[AppServiceParameterKey] as string;

      if (!string.IsNullOrEmpty(webPages))
      {
        // What we do now depends on whether we are running in the foreground
        // or not.
        if (!this.background)
        {
          BrowserViewModel.ForegroundInstance.Dispatch(() =>
          {
            BrowserViewModel.ForegroundInstance.Deserialize(webPages);
          });
        }
        else
        {
          // Store the pending web pages into our state file.
          await Storage.StoreAsync(webPages);

          // Flag that the app should read that state next time it launches
          // regardless of whether it was terminated or not.
          Storage.FlagRemoteRequestForNextLaunch();
        }
      }
    }

where the code is essentially extracting the list of web pages from the incoming App Service message and then it’s deciding based on the background flag whether to simple store that data into a file for the next time the app runs or whether to feed it into a ‘live’ ViewModel such that the UI will actively update.

Using x : Bind

The other thing that I played with quite a bit in this (admittedly simple) bit of code was to use x : Bind for everything. I still find myself forgetting to say Mode=OneWay from time to time but I’m increasingly getting used to it and a few things that I did here were quick/easy wins.

One of those is the simple binding to a boolean property as a visibility as you can see in the code below with the IsClosable property;

image

in that same snippet you can see that I’m also binding the Click event to a function on the ViewModel called OnClose and I really like that too as it (arguably) avoids me having to create an ICommand implementation. I also did that here on my Pivot control with the SelectionChanged event;

image

Where this fell down for me slightly was that I wanted to run code at the point where my WebView has loaded and has updated its DocumentTitle property. I ended up bringing in the UWP behaviors/interactivity pieces for this as below;

            <WebView
              x:Name="webView"
              Grid.Row="1"
              Source="{x:Bind BrowserUrl,Mode=OneWay}"
              xmlns:Interactivity="using:Microsoft.Xaml.Interactivity"
              xmlns:Core="using:Microsoft.Xaml.Interactions.Core">
              <Interactivity:Interaction.Behaviors>
                <Core:EventTriggerBehavior
                  EventName="LoadCompleted">
                  <Core:ChangePropertyAction
                    PropertyName="Label"
                    TargetObject="{x:Bind}"
                    Value="{Binding ElementName=webView,Path=DocumentTitle}" />
                </Core:EventTriggerBehavior>
              </Interactivity:Interaction.Behaviors>
            </WebView>

It’s certainly descriptive! Smile but what I really wanted to do was something a little bit like this;

    <WebView
              x:Name="webView"
              Grid.Row="1"
              Source="{x:Bind BrowserUrl,Mode=OneWay}"
	      LoadCompleted="{x:Bind OnLoaded, Parameter={Binding DocumentTitle}"/>

which, as far as I know, isn’t quite achievable with the current xBind implementation because the docs state that the function that is bound to an event must;

For events, the target method must not be overloaded and must also:

  • Match the signature of the event.
  • OR have no parameters.
  • OR have the same number of parameters of types that are assignable from the types of the event parameters.

and so I don’t think it’s possible for me to bind this event to a method which takes a string parameter and then somehow pass the DocumentTitle into that method but it’s what I’d have liked to do.

The last thing I found with x : Bind is that there were places where I used converters as below;

  <ComboBox.ItemTemplate>
          <DataTemplate
            x:DataType="rem:RemoteSystem">
            <Grid>
              <Grid.ColumnDefinitions>
                <ColumnDefinition />
                <ColumnDefinition
                  Width="Auto" />
              </Grid.ColumnDefinitions>
              <TextBlock
                VerticalAlignment="Center"
                Margin="0,0,2,0">
                <Run Text="{x:Bind DisplayName}" />
                <Run
                  Text=" " /> <!-- Iknow, I know -->
                <Run Text="{x:Bind IsAvailableByProximity,Converter={StaticResource cxnTypeConverter}}"/>
              </TextBlock>
              <Image
                Height="24"
                Grid.Column="1"
                Source="{x:Bind Kind,Converter={StaticResource deviceImageConverter}}" />
            </Grid>
          </DataTemplate>
        </ComboBox.ItemTemplate>

In that snippet, I’m using converters to convert the Kind and IsAvailableByProximity properties into different types of objects for display. I couldn’t use a binding to a function on the ViewModel here because I’d been lazy and simply bound a collection of RemoteSystem objects (which I don’t own) as the ItemsSource of the ComboBox. I can’t just add a method to a RemoteSystem class to do my conversions.

I could have;

  • Wrapped a ViewModel around the RemoteSystem, enabling me to bind to a function on that ViewModel to do this conversion
  • Maybe bound to a static function which did the conversion, passing it the property to convert.
  • Gone with a converter.

I went with the 3rd option but it was perhaps just laziness and I should have done the first as I usually would but I did get a little side-tracked thinking about the middle option.

Wrapping Up

I spent maybe 1-2 hours putting this together. I’ve shared the source here if you want to play with it but it’s not intended for anything beyond just experimenting with Rome APIs.

Windows 10, 1607 and UWP –Returning to Rome for an Experiment with Remote App Services

I wanted to return to the experiment that I did with ‘Project Rome’ in this post;

Windows 10 Anniversary Update (1607) and UWP Apps – Connected Apps and Devices via Rome

where I managed to experiment with the new APIs in Windows 10 1607 which allow you to interact with your graph of devices.

If you’ve not seen ‘Rome’ and the Windows.Systems.RemoteSystems classes then there’s a good overview here;

Connected Apps and Devices

In that previous post, I’d managed to use the RemoteSystemWatcher class to determine which remote devices I had and then to use its friends the RemoteSystemConnectionRequest and the RemoteLauncher class to have code on one of my devices launch an application (Maps) on another one of my devices. That post was really my own experimentation around the document here;

Launch an app on a remote device

I wanted to take that further though and see if I could use another capability of ‘Rome’ which is the ability for an app on one device to invoke an app service that is available on another device. That’s what this post is about and it’s really my own experimentation around the document here;

Communicate with a remote app service

In order to do that, I needed to come up with a scenario and I made up an idea that runs as follows;

  • There’s some workflow which involves redacting faces from images
  • The images to be redacted are stored in some blob container within Azure acting as a ‘queue’ of images to be worked on
  • The redacted images are to be stored in some other blob container within Azure
  • The process of downloading images, redacting them and then uploading the new images might be something that you’d want to run either locally on the device you’re working on or, sometimes, you might choose to do it remotely on another device which perhaps was less busy or had a faster/cheaper network connection.

Getting Started

Clearly, this is a fairly ‘contrived’ scenario but I wandered off into one of my Azure Storage accounts with the ‘Azure Storage Explorer’ and I made two containers named processed and unprocessed respectively;

Capture

and here’s the empty processed container;

Capture1

I then wrote some a fairly clunky class based on top of the Nuget package WindowsAzure.Storage which would do a few things for me;

  • Get me lists of the URIs of the blobs in the two containers.
  • Download a blob and present it back as a decoded bitmap in the form of a SoftwareBitmap
  • Upload a StorageFile to the processed container given the file and a name for the new blob
  • Delete a blob given its URI

i.e. it’s pretty much just the subset of the CRUD operations that I need for what my app needs to do.

That class ended up looking like this and, if you take a look at it, then note that it’s hard-wired to expect JPEG images;

namespace App26
{
  using Microsoft.WindowsAzure.Storage;
  using Microsoft.WindowsAzure.Storage.Auth;
  using Microsoft.WindowsAzure.Storage.Blob;
  using System;
  using System.Collections.Generic;
  using System.IO;
  using System.Linq;
  using System.Threading.Tasks;
  using Windows.Graphics.Imaging;
  using Windows.Storage;

  public class AzurePhotoStorageManager
  {
    public AzurePhotoStorageManager(
      string azureStorageAccountName,
      string azureStorageAccountKey,
      string unprocessedContainerName = "unprocessed",
      string processedContainerName = "processed")
    {
      this.azureStorageAccountName = azureStorageAccountName;
      this.azureStorageAccountKey = azureStorageAccountKey;
      this.unprocessedContainerName = unprocessedContainerName;
      this.processedContainerName = processedContainerName;
      this.InitialiseBlobClient();
    }
    void InitialiseBlobClient()
    {
      if (this.blobClient == null)
      {
        this.storageAccount = new CloudStorageAccount(
          new StorageCredentials(this.azureStorageAccountName, this.azureStorageAccountKey),
          true);

        this.blobClient = this.storageAccount.CreateCloudBlobClient();
      }
    }
    public async Task<IEnumerable<Uri>> GetProcessedPhotoUrisAsync()
    {
      var entries = await this.GetPhotoUrisAsync(this.processedContainerName);
      return (entries);
    }
    public async Task<IEnumerable<Uri>> GetUnprocessedPhotoUrisAsync()
    {
      var entries = await this.GetPhotoUrisAsync(this.unprocessedContainerName);
      return (entries);
    }
    public async Task<SoftwareBitmap> GetSoftwareBitmapForPhotoBlobAsync(Uri storageUri)
    {
      // This may not quite be the most efficient function ever known to man 🙂
      var reference = await this.blobClient.GetBlobReferenceFromServerAsync(storageUri);
      await reference.FetchAttributesAsync();

      SoftwareBitmap bitmap = null;

      using (var memoryStream = new MemoryStream())
      {
        await reference.DownloadToStreamAsync(memoryStream);

        var decoder = await BitmapDecoder.CreateAsync(
          BitmapDecoder.JpegDecoderId,
          memoryStream.AsRandomAccessStream());

        // Going for BGRA8 and premultiplied here saves me a lot of pain later on
        // when using SoftwareBitmapSource or using CanvasBitmap from Win2D.
        bitmap = await decoder.GetSoftwareBitmapAsync(
          BitmapPixelFormat.Bgra8, BitmapAlphaMode.Premultiplied);
      }
      return (bitmap);
    }
    public async Task PutFileForProcessedPhotoBlobAsync(
      string photoName,
      StorageFile file)
    {
      var container = this.blobClient.GetContainerReference(this.processedContainerName);

      var reference = container.GetBlockBlobReference(photoName);
      
      await reference.UploadFromFileAsync(file);
    }
    public async Task<bool> DeletePhotoBlobAsync(Uri storageUri)
    {
      var container = await this.blobClient.GetBlobReferenceFromServerAsync(storageUri);
      var result = await container.DeleteIfExistsAsync();
      return (result);
    }
    async Task<IEnumerable<Uri>> GetPhotoUrisAsync(string containerName)
    {
      var uris = new List<Uri>();
      var container = this.blobClient.GetContainerReference(containerName);

      BlobContinuationToken continuationToken = null;

      do
      {
        var results = await container.ListBlobsSegmentedAsync(continuationToken);

        if (results.Results?.Count() > 0)
        {
          uris.AddRange(results.Results.Select(r => r.Uri));
        }
        continuationToken = results.ContinuationToken;

      } while (continuationToken != null);

      return (uris);
    }
    CloudStorageAccount storageAccount;
    CloudBlobClient blobClient;
    string azureStorageAccountName;
    string azureStorageAccountKey;
    string unprocessedContainerName;
    string processedContainerName;
  }
}

and is probably nothing much to write home about Smile I also wrote another little class which attempts to take a SoftwareBitmap, to use the FaceDetector (UWP) API to find faces within that SoftwareBitmap and then to use Win2D.uwp to replace any faces that the FaceDetector finds with black rectangles.

For my own ease, I had the class then store the resultant bitmap into a temporary StorageFile. That class ended up looking like this;

namespace App26
{
  using Microsoft.Graphics.Canvas;
  using System;
  using System.Collections.Generic;
  using System.Linq;
  using System.Runtime.InteropServices.WindowsRuntime;
  using System.Threading.Tasks;
  using Windows.Foundation;
  using Windows.Graphics.Imaging;
  using Windows.Media.FaceAnalysis;
  using Windows.Storage;
  using Windows.UI;

  public class PhotoFaceRedactor
  {
    public async Task<StorageFile> RedactFacesToTempFileAsync(SoftwareBitmap incomingBitmap)
    {
      StorageFile tempFile = null;

      await this.CreateFaceDetectorAsync();

      // We assume our incoming bitmap format won't be supported by the face detector. 
      // We can check at runtime but I think it's unlikely.
      IList<DetectedFace> faces = null;
      var pixelFormat = FaceDetector.GetSupportedBitmapPixelFormats().First();

      using (var faceBitmap = SoftwareBitmap.Convert(incomingBitmap, pixelFormat))
      {
        faces = await this.faceDetector.DetectFacesAsync(faceBitmap);
      }
      if (faces?.Count > 0)
      {
        // We assume that our bitmap is in decent shape to be used by CanvasBitmap
        // as it should already be BGRA8 and Premultiplied alpha.
        var device = CanvasDevice.GetSharedDevice();

        using (var target = new CanvasRenderTarget(
          CanvasDevice.GetSharedDevice(),
          incomingBitmap.PixelWidth,
          incomingBitmap.PixelHeight,
          96.0f))
        {
          using (var canvasBitmap = CanvasBitmap.CreateFromSoftwareBitmap(device, incomingBitmap))
          {
            using (var session = target.CreateDrawingSession())
            {
              session.DrawImage(canvasBitmap,
                new Rect(0, 0, incomingBitmap.PixelWidth, incomingBitmap.PixelHeight));

              foreach (var face in faces)
              {
                session.FillRectangle(
                  new Rect(
                    face.FaceBox.X,
                    face.FaceBox.Y,
                    face.FaceBox.Width,
                    face.FaceBox.Height),
                  Colors.Black);
              }
            }
          }
          var fileName = $"{Guid.NewGuid()}.jpg";

          tempFile = await ApplicationData.Current.TemporaryFolder.CreateFileAsync(
            fileName, CreationCollisionOption.GenerateUniqueName);

          using (var fileStream = await tempFile.OpenAsync(FileAccessMode.ReadWrite))
          {
            await target.SaveAsync(fileStream, CanvasBitmapFileFormat.Jpeg);
          }
        }
      }
      return (tempFile);
    }
    async Task CreateFaceDetectorAsync()
    {
      if (this.faceDetector == null)
      {
        this.faceDetector = await FaceDetector.CreateAsync();
      }
    }
    FaceDetector faceDetector;
  }
}

I also wrote a static method that co-ordinated these two classes to perform the whole process of getting hold of a photo, taking out the faces in it and uploading it back to blob storage and that ended up looking like this;

namespace App26
{
  using System;
  using System.Threading.Tasks;

  static class RedactionController
  {
    public static async Task RedactPhotoAsync(Uri photoBlobUri, string newName)
    {
      var storageManager = new AzurePhotoStorageManager(
        Constants.AZURE_STORAGE_ACCOUNT_NAME,
        Constants.AZURE_STORAGE_KEY);

      var photoRedactor = new PhotoFaceRedactor();

      using (var bitmap = await storageManager.GetSoftwareBitmapForPhotoBlobAsync(photoBlobUri))
      {
        var tempFile = await photoRedactor.RedactFacesToTempFileAsync(bitmap);

        await storageManager.PutFileForProcessedPhotoBlobAsync(newName, tempFile);

        await storageManager.DeletePhotoBlobAsync(photoBlobUri);
      }
    }
  }
}

Adding in Some UI

I added in a few basic ‘ViewModels’ which surfaced this information into a UI and made something that seemed to essentially work. The UI is as below;

Capture2

and you can see the 2 lists of processed/unprocessed photos and if I click on one of the View buttons then the UI displays that photo;

Capture3

and then tapping on that photo takes it away again. If I click on one of the ‘Process’ buttons then there’s a little bit of a progress ring followed by an update to the UI which I’m quite lazy about in the sense that I simply requery all the data from Azure again. Here’s the UI after I’ve processed that particular image;

Capture4

and if I click on that bottom View button then I see;

Capture5

As an aside, the Image that is displaying things here has its Stretch set which is perhaps why the images look a bit odd Smile

Without listing all the XAML and all the view model code, that got me to the point where I had my basic bit of functionality working.

What I wanted to add to this then was a little bit from ‘Project Rome’ to see if I could set this up such that this functionality could be offered as an ‘app service’ and what especially interested me about this idea was whether the app could become a client of itself in the sense that this app could choose to let the user either do this photo ‘redaction’ locally on the device they were on or remotely on another one of their devices.

Making an App Service

Making a (basic) app service is pretty easy. I simply edited my manifest to say that I was making an App Service but I thought that I’d highlight that it’s necessary (as per the official docs) to make sure that my service called PhotoRedactionService has marked itself as being available to remote systems as below;

Capture6

and then I wrote the basics of a background task and an app service using the new mechanism that’s present in 1607 which is to override the OnBackgroundActivated method on the App class and do the background work inside of there rather than having to go off and write a completely separate WinRT component. Here’s that snippet of code;

    protected override void OnBackgroundActivated(BackgroundActivatedEventArgs args)
    {
      this.taskDeferral = args.TaskInstance.GetDeferral();
      args.TaskInstance.Canceled += OnBackgroundTaskCancelled;

      var details = args.TaskInstance.TriggerDetails as AppServiceTriggerDetails;

      if ((details != null) && (details.Name == Constants.APP_SERVICE_NAME))
      {
        this.appServiceConnection = details.AppServiceConnection;
        this.appServiceConnection.RequestReceived += OnRequestReceived;        
      }
    }
    void OnBackgroundTaskCancelled(IBackgroundTaskInstance sender, BackgroundTaskCancellationReason reason)
    {
      this.appServiceConnection.Dispose();
      this.appServiceConnection = null;
      this.taskDeferral?.Complete();
    }
    async void OnRequestReceived(AppServiceConnection sender, AppServiceRequestReceivedEventArgs args)
    {
      var deferral = args.GetDeferral();

      var incomingUri = args.Request.Message[Constants.APP_SERVICE_URI_PARAM_NAME] as string;

      var uri = new Uri(incomingUri);

      // TODO: Move this function off the viewmodel into some utiliy class.
      await RedactionController.RedactPhotoAsync(uri, MainPageViewModel.UriToFileName(uri));

      deferral.Complete();
    }
    AppServiceConnection appServiceConnection;
    BackgroundTaskDeferral taskDeferral;

In that code fragment – you’ll see that all that’s happening is;

  1. We receive a background activation.
  2. We check to see if its an ‘app service’ type of activation and, if so, whether the name of the activation matches my service (Constants.APP_SERVICE_NAME = “PhotoRedactionService”)
  3. We handle the RequestReceived event
    1. We look for a URI parameter to be passed to us (the URI of the photo to be redacted)
    2. We call into our code to do the redaction

and that’s pretty much it. I now how have an app service that does ‘photo redaction’ for me and I’ve got no security or checks around it whatsoever (which isn’t perhaps the best idea!).

Adding in some ‘Rome’

In that earlier screenshot of my ‘UI’ you’d have noticed that I have a Checkbox which says whether to perform ‘Remote Processing’ or not;

Capture7

this Checkbox is simply bound to a property on a ViewModel and the ComboBox next to it is bound to an ObservableCollection<RemoteSystem> in this way;

        <ComboBox
          Margin="4"
          MinWidth="192"
          HorizontalAlignment="Center"
          ItemsSource="{x:Bind ViewModel.RemoteSystems, Mode=OneWay}"
          SelectedValue="{x:Bind ViewModel.SelectedRemoteSystem, Mode=TwoWay}">
          <ComboBox.ItemTemplate>
            <DataTemplate x:DataType="rem:RemoteSystem">
              <TextBlock
                Text="{x:Bind DisplayName}" />
            </DataTemplate>
          </ComboBox.ItemTemplate>
        </ComboBox>

The population of that list of ViewModel.RemoteSystems is pretty easy and it was something that I learned in my previous post. I simply have some code which bootstraps the process;

      var result = await RemoteSystem.RequestAccessAsync();

      if (result == RemoteSystemAccessStatus.Allowed)
      {
        this.RemoteSystems = new ObservableCollection<RemoteSystem>();
        this.remoteWatcher = RemoteSystem.CreateWatcher();
        this.remoteWatcher.RemoteSystemAdded += OnRemoteSystemAdded;
        this.remoteWatcher.Start();
      }

and then when a new RemoteSystem is added I make sure it goes into my collection;

    void OnRemoteSystemAdded(RemoteSystemWatcher sender, RemoteSystemAddedEventArgs args)
    {
      this.Dispatch(
        () =>
        {
          this.remoteSystems.Add(args.RemoteSystem);

          if (this.SelectedRemoteSystem == null)
          {
            this.SelectedRemoteSystem = args.RemoteSystem;
          }
        }
      );
    }

and so now I’ve got a list of remote systems that might be able to process an image for me.

Invoking the Remote App Service

The last step is to invoke the app service remotely and I have a method which does that for me that is invoked with the URI of the blob of the photo to be processed;

    async Task RemoteRedactPhotoAsync(Uri uri)
    {
      var request = new RemoteSystemConnectionRequest(this.selectedRemoteSystem);
      using (var connection = new AppServiceConnection())
      {
        connection.AppServiceName = Constants.APP_SERVICE_NAME;

        // Strangely enough, we're trying to talk to ourselves but on another
        // machine.
        connection.PackageFamilyName = Package.Current.Id.FamilyName;
        var remoteConnection = await connection.OpenRemoteAsync(request);

        if (remoteConnection == AppServiceConnectionStatus.Success)
        {
          var valueSet = new ValueSet();
          valueSet[Constants.APP_SERVICE_URI_PARAM_NAME] = uri.ToString();
          var response = await connection.SendMessageAsync(valueSet);

          if (response.Status != AppServiceResponseStatus.Success)
          {
            // Bit naughty throwing a UI dialog from this view model
            await this.DisplayErrorAsync($"Received a response of {response.Status}");
          }
        }
        else
        {
          await this.DisplayErrorAsync($"Received a status of {remoteConnection}");
        }
      }
    }

For me, the main things of interest here would be that this code looks pretty much like any invocation to an app service except that we have the extra step here of constructing the RemoteSystemConnectionRequest based on the RemoteSystem that the ComboBox has selected and then on the AppServiceConnection class I use the OpenRemoteAsync() method rather than the usual OpenAsync() method.

The other thing which I think is unusual in my scenario here is that the PackageFamilyName that I set for the remote app is actually the same as the calling app because I’ve conjured up this weird scenario where my app talks to its own app service on another device.

It’s worth noting that I don’t need to have the app running on another device to invoke it, it just has to be installed.

Wrapping Up

As is often the case, my code here is sketchy and quite rough-and-ready but I quite enjoyed putting this little experiment together because I wasn’t sure whether the ‘Rome’ APIs would;

  1. Allow an app to invoke another instance of ‘itself’ on one of the user’s other devices
  2. Make (1) difficult if it even allowed it

and I was pleasantly surprised to find that the APIs actually made it pretty easy and it’s just like invoking a regular App Service.

I need to have a longer think about what sort of scenarios this enables but I found it interesting here to toy with the idea that I can run this app on my phone, get a list of work items to be processed and then I can elect to process those work items (using the exact same app) on one of my other devices which might have better/cheaper bandwidth and/or more CPU power.

I need to think on that. In the meantime, the code is here on github if you want to play with it. Be aware that to make it run you’d need;

  1. To edit the Constants file to provide storage account name and key.
  2. To make sure that you’d created blob containers called processed and unprocessed within your storage account.

Enjoy.

Windows 10 Anniversary Update (1607) and UWP Apps – Connected Apps and Devices via Rome

I tried something out for the first time today that really caught my interest and so I thought I’d share.

I first heard reference to ‘Project Rome’ back at //build/ 2016 but I must admit that I hadn’t found time to watch the session about it properly until recently and that’s partly just due to ‘never having time’ and also partly because I had started to watch it and it didn’t seem to immediately leap into the topic and so I’d put it to one side.

I should have had more faith Smile because it’s a really good session and it’s absolutely worth watching – click the image below to jump into it (it’s short at around 30mins).

image

There’s two discrete parts to that session.

The first part about ‘Web to App Linking’. The core idea there is that an app can register itself (under certain circumstances and user control) to be a handler for what I’d call a ‘portion of the HTTP/HTTPS namespace’ such that if the device comes across a URI like “http://www.myAmazingApp.com” then it’s possible that when the system launches that URI the result ends up in the app from ‘My Amazing App’ rather than the standard system action which would be to launch the browser.

That’s written up really well here;

Support web-to-app linking with app URI handlers

But that //build/ 2016 session then switches at the 15m mark to talk about ‘Project Rome’ which feels like it is doing some of the legwork on the idea of an experience which spans devices rather than being confined to a single device.

There’s been lots of talk at Microsoft around ‘the mobility of the experience’ and it feels to me that ‘Project Rome’ has a part to play in that idea by providing some pieces of a framework which allows for asking questions about a user’s set of devices and then taking higher level actions across that set of devices.

I wanted to try it out for the first time so having watched the video above I went and got my developer laptop running 1607 and I started to see if the docs for classes like RemoteSystem that I spotted in the video were online (they are).

I then paused for a while as I hadn’t really considered whether I had multiple devices running Windows 10 1607 to test on until I remembered that I had a home all-in-one that I’d upgraded to 1607 which is sitting in another room on standby with some active logon sessions.

It’s worth saying that the set of devices that is being worked with here is the set of devices that you see when you go to;

http://account.microsoft.com

and then have a look at your list of Devices there;

Untitled

and I have this home office PC which shows up there.

Capture

As is often the case, I’d started to write a little bit of code when I found that there’s an article which brings some of these classes together;

Launch an app on a remote device

and that led me to the higher level documentation;

Connected apps and devices (Project “Rome”)

and my ‘Hello World’ is pretty much a duplicate of what happens in that former article in that I made simple XAML page which presents a data-bound ListView and a Button;

  <Grid
    Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">
    <Grid.RowDefinitions>
      <RowDefinition />
      <RowDefinition
        Height="Auto" />
    </Grid.RowDefinitions>
    <ListView
      Margin="8"
      Header="Remote Systems"
      ItemsSource="{x:Bind RemoteSystems}"
      SelectionMode="Single"
      SelectedItem="{x:Bind SelectedSystem,Mode=TwoWay}">
      <ListView.ItemContainerStyle>
        <Style
          TargetType="ListViewItem">
          <Setter
            Property="HorizontalContentAlignment"
            Value="Stretch" />
        </Style>
      </ListView.ItemContainerStyle>
      <ListView.ItemTemplate>
        <DataTemplate
          x:DataType="remote:RemoteSystem">
          <Grid>
            <Grid.ColumnDefinitions>
              <ColumnDefinition />
              <ColumnDefinition />
              <ColumnDefinition />
              <ColumnDefinition />
              <ColumnDefinition />
            </Grid.ColumnDefinitions>
            <TextBlock
              Text="{x:Bind DisplayName}" />
            <TextBlock
              Text="{x:Bind Id}"
              Grid.Column="1" />
            <TextBlock
              Text="{x:Bind Kind}"
              Grid.Column="2" />
            <TextBlock
              Text="{x:Bind Status}"
              Grid.Column="3" />
            <TextBlock
              Text="{x:Bind IsAvailableByProximity}"
              Grid.Column="4" />
          </Grid>
        </DataTemplate>
      </ListView.ItemTemplate>
    </ListView>
    <Button
      Margin="8"
      Grid.Row="1"
      HorizontalContentAlignment="Center"
      HorizontalAlignment="Stretch"
      VerticalAlignment="Stretch"
      Click="{x:Bind OnLaunchUri}">
      <Button.Content>
        <StackPanel Orientation="Horizontal"
                    HorizontalAlignment="Center">
          <SymbolIcon
            Symbol="Map"/>
          <TextBlock
            Margin="4,0,0,0"
            Text="launch map of New York" />
        </StackPanel>
      </Button.Content>
    </Button>
  </Grid>

and then added some code-behind to see if I could get the basics of this to work;

namespace App24
{
  using System;
  using System.Collections.ObjectModel;
  using System.ComponentModel;
  using System.Linq;
  using System.Runtime.CompilerServices;
  using System.Threading.Tasks;
  using Windows.System;
  using Windows.System.RemoteSystems;
  using Windows.UI.Core;
  using Windows.UI.Xaml;
  using Windows.UI.Xaml.Controls;

  public sealed partial class MainPage : Page, INotifyPropertyChanged
  {
    public event PropertyChangedEventHandler PropertyChanged;

    public MainPage()
    {
      this.InitializeComponent();
      this.Loaded += OnLoaded;
      this.RemoteSystems = new ObservableCollection<RemoteSystem>();
    }
    public ObservableCollection<RemoteSystem> RemoteSystems
    {
      get; private set;
    }
    public RemoteSystem SelectedSystem
    {
      get
      {
        return (this.selectedSystem);
      }
      set
      {
        if (this.selectedSystem != value)
        {
          this.selectedSystem = value;
          this.FirePropertyChanged();
        }
      }
    }
    // Note, this is bound to from an x:Bind, just trying that out for the
    // first time.
    public async void OnLaunchUri()
    {
      if (this.SelectedSystem != null)
      {
        var request = new RemoteSystemConnectionRequest(this.SelectedSystem);

        await RemoteLauncher.LaunchUriAsync(
          request, 
          new Uri("bingmaps:?cp=40.726966~-74.006076&lvl=10&where=New%20York "));
      }
    }
    void FirePropertyChanged([CallerMemberName] string prop = "")
    {
      this.PropertyChanged?.Invoke(
        this, new PropertyChangedEventArgs(prop));
    }
    async void OnLoaded(object sender, RoutedEventArgs e)
    {
      var status = await RemoteSystem.RequestAccessAsync();

      if (status == RemoteSystemAccessStatus.Allowed)
      {
        this.watcher = RemoteSystem.CreateWatcher();
        this.watcher.RemoteSystemAdded += OnAdded;
        this.watcher.RemoteSystemRemoved += OnRemoved;
        this.watcher.RemoteSystemUpdated += OnUpdated;
        this.watcher.Start();
      }
    }
    async void OnUpdated(RemoteSystemWatcher sender, RemoteSystemUpdatedEventArgs args)
    {
      await this.DispatchAsync(
        () =>
        {
          // Laziness on my part not wanting to create a view model for a remote system
          // with property change notification etc.
          this.Remove(args.RemoteSystem.Id);
          this.Add(args.RemoteSystem);
        }
      );
    }
    async void OnRemoved(RemoteSystemWatcher sender, RemoteSystemRemovedEventArgs args)
    {
      await this.DispatchAsync(
        () =>
        {
          this.Remove(args.RemoteSystemId);
        }
      );
    }
    async void OnAdded(RemoteSystemWatcher sender, RemoteSystemAddedEventArgs args)
    {
      await this.DispatchAsync(
        () =>
        {
          this.Add(args.RemoteSystem);

        }
      );
    }
    void Add(RemoteSystem remoteSystem)
    {
      this.RemoteSystems.Add(remoteSystem);
    }
    async Task DispatchAsync(DispatchedHandler action)
    {
      await this.Dispatcher.RunAsync(CoreDispatcherPriority.Normal, action);
    }
    void Remove(string remoteSystemId)
    {
      var entry = this.RemoteSystems.SingleOrDefault(system => system.Id == remoteSystemId);

      if (entry != null)
      {
        this.RemoteSystems.Remove(entry);
      }
    }
    RemoteSystem selectedSystem;
    RemoteSystemWatcher watcher;
  }
}

and so most of this code is just using the RemoteSystemWatcher class and, having asked for permission, this is going to deliver RemoteSystem information to me via events for Added/Removed/Updated. I use that to build up an ObservableCollection which is bound into the ListView on the screen and I also data-bind the SelectedSystem to the selected item in that ListView.

I liked that the RemoteSystemWatcher follows what feels like a pattern that’s consistent with (e.g.) looking for AllJoyn devices or looking for Bluetooth LE devices – there are quite a few ‘watcher’ objects within the UWP APIs and this one feels natural to use.

I hooked the button press to then construct a RemoteSystemConnectionRequest instance and to do a LaunchUriAsync on that remote system and I’m letting the default options stand here but it’s possible to specify the list of apps that I’d prefer handle the request remotely and also any launch parameters to be passed to the remote app.

I must admit that the first time that I ran this code, I was not expecting success. It seemed complex enough to be unlikely to work and I figured that there’d likely be some configuring and messing around to do.

I got success Smile 

The little UI listed the all-in-one office machine that had been sitting in another room on standby the whole time I’d been writing this code;

Capture

and I tapped on the button, went off to visit that machine and (sure enough) there was the Maps app displaying a map of New York.

I like this a lot Winking smile

Being an awkward type, I logged myself out of that machine to see if the status would go ‘Unavailable’ and it didn’t seem to and so I tried to launch the Maps app remotely again without an active logon session on that machine and this caused me to add a bit of code to display the return value from the LaunchUriAsync method;

        var response = await RemoteLauncher.LaunchUriAsync(
          request, 
          new Uri("bingmaps:?cp=40.726966~-74.006076&lvl=10&where=New%20York "));

        var messageDialog = new MessageDialog(response.ToString(), "return status");

        await messageDialog.ShowAsync();

and, sure enough, this now returned a RemoteSystemUnavailable status even though the RemoteSystem itself was still claiming availability so I’m not sure quite how that ‘availability’ status works and when it updates.

I also noticed a very interesting thing in that when I logged back in to that system the Maps app popped up with a map of New York – it seems like my request has been queued somewhere and delivered at a later point in time but I’d need to dig into that more to understand whether that’s by design or not.

Being an awkward type, I took the machine that is acting as a client here off the network that it shares with the ‘mthome’ PC and moved it to a 4G network to see what might happen there and it worked just the same way as you might expect.

I wondered how I might get these 2 PCs to be ‘proximally’ connected without a cloud connection and so I played with a few options there but have yet to figure out quite how that works for 2 PCs, maybe I’d have more luck with a phone for that scenario but it needs returning to.

Being able to launch an app on a remote machine with parameters opens up a tonne of scenarios which involve transferring a task from one device to another and especially when you think that those parameters might be some identifier to a piece of shared state in the cloud but there’s another part of this ‘Rome’ framework that interests me here too and that’s the idea of;

Communicate with a remote app service

and that gives us the possibility of an app on one device making use of app services from an app on another device.

That’s a topic for another post but it seems really quite powerful in that it has the potential to open up an opportunity to run the constituent pieces of an application on the machines that I think are best suited to the task at the time. I could see examples where I might choose to run some processing on a remote device because it has more processing power or a better (or cheaper, or more reliable) connection to the internet.

I want to dig into that but, again, that’s for another post…