Windows 10, UWP and Experimenting with Inking onto a Map Control (Updated)

Just a quick update to this earlier post;

Windows 10, UWP and Experimenting with Inking onto a Map Control

I got some feedback on that post from a few different places and people pointed out that I could have made more of ‘option 3’ in that post where I’d tried to overlay an InkCanvas on top of a MapControl.

Specifically;

    <Grid
        Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">

        <Maps:MapControl 
            x:Name="map"
            MapTapped="OnMapTapped"
            xmlns:Maps="using:Windows.UI.Xaml.Controls.Maps" />

        <InkCanvas
            x:Name="inkCanvas"
            Visibility="Collapsed">
        </InkCanvas>

    </Grid>

Now, I got some feedback on Twitter and elsewhere that I could make that InkCanvas non hit-testable and it’s true that this does work for me (thanks Dave!);

image

but, like Dave, I notice two things if (on OS 10586) I put a non-hit-testable InkCanvas over the top of my MapControl;

  1. I don’t get a MapTapped event as Dave says above.
  2. I notice a reasonable amount of lag when I do a pinch gesture which is going to the map control ‘through’ the InkCanvas control. It feels a little like I’m driving the map control through treacle and it’s not nearly as responsive as it would usually be.

so I veered away from having the InkCanvas initially Visible and using IsHitTestVisible=”False” and moved more towards having the InkCanvas be initially Collapsed as you can see in the XAML above and then dynamically making it visible as/when I wanted it.

That then ties up with this code behind where you’ll notice that I’ve now got 2 different ways of trying to turn an ink stroke into a set of waypoints and I suspect that neither of them are ‘perfect’ by any means and maybe you have a better one;

#define INK_POINTS

namespace Update
{
  using System;
  using System.Collections.Generic;
  using System.Linq;
  using Windows.Devices.Geolocation;
  using Windows.Foundation;
  using Windows.Services.Maps;
  using Windows.UI;
  using Windows.UI.Input.Inking;
  using Windows.UI.Xaml;
  using Windows.UI.Xaml.Controls;
  using Windows.UI.Xaml.Controls.Maps;

  public sealed partial class MainPage : Page
  {
    public MainPage()
    {
      this.InitializeComponent();

      this.tapPositions = new MapInputEventArgs[POI_COUNT];

      this.Loaded += OnLoaded;
    }
    void OnLoaded(object sender, RoutedEventArgs e)
    {
      this.inkCanvas.InkPresenter.StrokesCollected += OnStrokesCollected;

      var drawingAttr = this.inkCanvas.InkPresenter.CopyDefaultDrawingAttributes();
      drawingAttr.PenTip = PenTipShape.Rectangle;
      drawingAttr.Size = new Size(4, 4);
      drawingAttr.IgnorePressure = true;
      drawingAttr.Color = Colors.Orange;
      this.inkCanvas.InkPresenter.UpdateDefaultDrawingAttributes(drawingAttr);
    }
    Geopoint GeopointFromPoint(Point point)
    {
      Geopoint geoPoint = null;

      this.map.GetLocationFromOffset(point, out geoPoint);

      return (geoPoint);
    }
    async void OnStrokesCollected(InkPresenter sender, InkStrokesCollectedEventArgs args)
    {
      // We're going to take the first stroke, we're not going to try and
      // somehow stitch together many strokes :-S
      var firstStroke = args.Strokes.FirstOrDefault();

      if (firstStroke != null)
      {
        // How to split this down into some waypoints?
        var geoPoints = new List<Geopoint>();

        // add the initial point.
        geoPoints.Add(GeopointFromPoint(this.tapPositions[0].Position));


        // Now try to add some intermediate waypoints to try and 'guide'
        // the route so that it follows the line that's been drawn.

        // I'm not sure how much this next section of code is going to cause
        // duplication by (e.g.) adding the same or similar points along the
        // route due to the ink points potentially being very close together
        // etc.
#if INK_POINTS
        // Add some sprinkling of the ink points...I've chosen 20,40, etc.
        var inkPoints = firstStroke.GetInkPoints();

        const int SAMPLE_RATE = 20;

        for (int i = SAMPLE_RATE; i < inkPoints.Count; i += SAMPLE_RATE)
        {
          geoPoints.Add(GeopointFromPoint(inkPoints[i].Position));
        }
#else
        // Add the positions of the segments that make up the ink stroke.
        foreach (var segment in firstStroke.GetRenderingSegments())
        {
          geoPoints.Add(GeopointFromPoint(segment.Position));
        }
#endif

        // add the follow on point.
        geoPoints.Add(GeopointFromPoint(this.tapPositions[1].Position));

        var routeResult =
          await MapRouteFinder.GetDrivingRouteFromWaypointsAsync(geoPoints);

        // We should do something about failures too
        if (routeResult.Status == MapRouteFinderStatus.Success)
        {
          var mapPolyline = new MapPolyline();
          mapPolyline.Path = routeResult.Route.Path;
          mapPolyline.StrokeThickness = 4;
          mapPolyline.StrokeColor = Colors.Orange;
          mapPolyline.Visible = true;
          this.map.MapElements.Add(mapPolyline);
        }
      }
      this.inkCanvas.Visibility = Visibility.Collapsed;

      this.inkCanvas.InkPresenter.StrokeContainer.Clear();
    }

    void OnMapTapped(MapControl sender, MapInputEventArgs args)
    {
      var mapElementCount = this.map.MapElements.Count;

      if (mapElementCount < POI_COUNT)
      {
        this.tapPositions[mapElementCount] = args;

        var mapIcon = new MapIcon()
        {
          Location = args.Location,
          NormalizedAnchorPoint = new Point(0.5, 0.5)
        };
        this.map.MapElements.Add(mapIcon);

        // When you add a 2nd point of interest, we put an InkCanvas over the entire map
        // waiting to see what you do next. If your next move is to use a pen then 
        // we'll wait for an ink stroke. If your next move is to use something
        // other than a pen then we'll get rid of the InkCanvas and get out of your
        // way.
        mapElementCount = this.map.MapElements.Count;

        if (mapElementCount == POI_COUNT)
        {
          // Switch the ink canvas on and leave it on until the user completes
          // a stroke.
          this.inkCanvas.Visibility = Visibility.Visible;
        }
      }
    }
    static readonly int POI_COUNT = 2;
    MapInputEventArgs[] tapPositions;
  }
}

And that works pretty well for me compared to my original post and it’s quite a bit less code (especially if you removed the conditional compilation).

If I take away the requirement to handle MapTapped event then I could make the InkCanvas initially visible and non hit-testable as in;

    <Grid
        Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">

        <Maps:MapControl 
            x:Name="map"
            xmlns:Maps="using:Windows.UI.Xaml.Controls.Maps" />

        <InkCanvas
            x:Name="inkCanvas"
            IsHitTestVisible="False">
        </InkCanvas>

    </Grid>

and then my code behind does a little less as it does not wait for 2 taps to add 2 points of interest but, instead, just waits for a single line to be drawn and uses it to try and sketch out a route;

  using System;
  using System.Collections.Generic;
  using System.Linq;
  using Windows.Devices.Geolocation;
  using Windows.Foundation;
  using Windows.Services.Maps;
  using Windows.UI;
  using Windows.UI.Input.Inking;
  using Windows.UI.Xaml;
  using Windows.UI.Xaml.Controls;
  using Windows.UI.Xaml.Controls.Maps;

  public sealed partial class MainPage : Page
  {
    public MainPage()
    {
      this.InitializeComponent();

      this.Loaded += OnLoaded;
    }
    void OnLoaded(object sender, RoutedEventArgs e)
    {
      this.inkCanvas.InkPresenter.StrokesCollected += OnStrokesCollected;

      var drawingAttr = this.inkCanvas.InkPresenter.CopyDefaultDrawingAttributes();
      drawingAttr.PenTip = PenTipShape.Rectangle;
      drawingAttr.Size = new Size(4, 4);
      drawingAttr.IgnorePressure = true;
      drawingAttr.Color = Colors.Orange;
      this.inkCanvas.InkPresenter.UpdateDefaultDrawingAttributes(drawingAttr);
    }
    Geopoint GeopointFromPoint(Point point)
    {
      Geopoint geoPoint = null;

      this.map.GetLocationFromOffset(point, out geoPoint);

      return (geoPoint);
    }
    async void OnStrokesCollected(InkPresenter sender, InkStrokesCollectedEventArgs args)
    {
      // We're going to take the first stroke, we're not going to try and
      // somehow stitch together many strokes :-S
      var firstStroke = args.Strokes.FirstOrDefault();

      if (firstStroke != null)
      {
        // How to split this down into some waypoints?
        var geoPoints = new List<Geopoint>();

        // Now try to add some intermediate waypoints to try and 'guide'
        // the route so that it follows the line that's been drawn.

        // I'm not sure how much this next section of code is going to cause
        // duplication by (e.g.) adding the same or similar points along the
        // route due to the ink points potentially being very close together
        // etc.
#if INK_POINTS
        // Add some sprinkling of the ink points...I've chosen 20,40, etc.
        var inkPoints = firstStroke.GetInkPoints();

        const int SAMPLE_RATE = 20;

        for (int i = SAMPLE_RATE; i < inkPoints.Count; i += SAMPLE_RATE)
        {
          geoPoints.Add(GeopointFromPoint(inkPoints[i].Position));
        }
#else
        // Add the positions of the segments that make up the ink stroke.
        foreach (var segment in firstStroke.GetRenderingSegments())
        {
          geoPoints.Add(GeopointFromPoint(segment.Position));
        }
#endif

        var routeResult =
          await MapRouteFinder.GetDrivingRouteFromWaypointsAsync(geoPoints);

        // We should do something about failures too
        if (routeResult.Status == MapRouteFinderStatus.Success)
        {
          var mapPolyline = new MapPolyline();
          mapPolyline.Path = routeResult.Route.Path;
          mapPolyline.StrokeThickness = 4;
          mapPolyline.StrokeColor = Colors.Orange;
          mapPolyline.Visible = true;
          this.map.MapElements.Add(mapPolyline);
        }
      }
      this.inkCanvas.InkPresenter.StrokeContainer.Clear();
    }
  }

and that works fine albeit with that slight lag that gets introduced (on OS 10586 at least) by having the InkCanvas over the top of the MapControl.

Sketchy Thoughts on Bots/Agents/Conversations

It seems that ‘bots’ are ‘hot’ in the sense that the topic’s really attracting a lot of attention and if I had to pick out one great article that I read in this area then I’d say that it was this one on “Conversational Commerce” by Uber’s Chris Messina which really brought it home to me – I think that’s a really good read although I was late in coming to it Smile

You can read about some of Microsoft’s plans in the area of bots here [Bot Framework] and here [Cortana] and there were quite a few sessions at //build that relate to this area too.

The rest of this post does not relate to anything that Microsoft does or makes, it’s more my own brain dump from trying to think through some of the pieces that might be part of a platform that provides an ability to have these types of conversations.

I’ve been thinking on this topic of ‘bots’ for a little while and I wanted to;

  • start to try and get my thoughts in order and write them up so that I can come back to them and refine them
  • have a framework that I can use to look at particular platforms for ‘bots’ and try to evaluate whether that platform covers some of the areas that I’d expect a ‘bot’ platform to address.

Beyond here, I’m going to use the term ‘Agent’ rather than ‘Bot’ to avoid getting tied up in any particular implementation or anything like that.

Once again, it’s just a brain dump and a pretty sketchy one but you have to start somewhere Smile

Conversations

We’ve been having conversations of different forms with software Agents for the longest time. You could argue that when I do something like;

image

then I’m having a ‘conversation’ with the command prompt.

I “say” something and it “says” something back. It’s a short conversation. It’s not a very natural conversation but, nonetheless, it’s a form of conversation.

It also doesn’t offer much in the way of choice around the input/output mechanisms here. As far as I know, I can’t speak to the command prompt and it doesn’t speak back although it may have accessibility capabilities that I’m unaware of.

At a more advanced level, I can have a conversation with an Agent on one of my devices today and I can actually say something like;

    • “Agent, can you play the song [SONG TITLE] for me?”
    • “Did you mean this one or that one?”
    • “The second one”
    • “Ok, playing it”

This one definitely feels a bit more “conversational” and an Agent that accepts speech like this usually accepts at least typing as another possible input mechanism and displaying on a screen as another possible output mechanism.

Implicit in there is the idea that the Agent that I’m speaking to knows of some kind of app or service that can actually find music and get it played for me and it’s debatable as to whether that app/service does or doesn’t display a graphical interface as maybe sometimes it should and sometimes it shouldn’t depending on the context.

What’s interesting though would be that if I then continued the conversation with something like;

    • “Agent, remember that song you played for me just before lunch? Play it again”

then I don’t know whether there are any platforms out there today that can handle even a simple conversation like that and the notion that conversations might last a while and might have related history.

The context has been lost at that point and we have to start again and it feels like even the simplest elements of human conversations are going to need quite a lot of work if they’re going to be brought to a ‘conversation platform’ and, naturally, this will be done incrementally with ever growing value along the way.

With that in mind, I was trying to think of some of the pieces that might make up that kind of platform and mostly so that I can come back to them at a later point. Some of these pieces I’ve seen referred to in other articles, videos, etc. and others I’m perhaps conjuring out of the air.

Conversational Pieces

I scratched my head for quite a while and this list dropped out of some of the pieces that might be involved in a conversational platform when thinking of conversations in a broad sense;

  • The Conversational Host or Canvas
  • The Agent
  • Discoverability
  • Inputs
  • Outputs
  • Initiation
  • Identity
  • Dialogs
  • Language Processing
  • User Knowledge
  • Trust
  • Context
  • Termination
  • Services
  • Decisions
  • Telemetry
  • History

That list isn’t ordered in any fashion.

I did a quick sketch below and you’ll soon realise from the number of arrows on the diagram that I haven’t reached any kind of clarity on this just yet and am still fumbling a bit and ‘making it up’ Smile but, again, it’s a starting point that can be refined.

foo

The Conversational Host or Canvas

This feels like a very broad term but it seems that there’s a need for “something” to host the conversation and it might be something like an app that hosts a voice call or an SMS conversation. It might be a chat app or an email client. It might be a part of an operating system “shell”.

It’s the “host” of the conversation and, naturally, I might want to move from one host to another and have a conversation follow me which almost certainly comes with a set of technical challenges.

Some conversational hosts might serve a specific purpose. For example, a device on a kitchen table that is hard-wired to play music.

Others might broker between many agents – for example a chat application that can both book train tickets and return flight information.

It seems to me that it’s likely that the Canvas will control the modes of input and output, perhaps offering some subset of those available on the device that it’s running on and it also seems to me that it’s unlikely that developers will want to build for every Canvas out there and so, over time, perhaps some canvases will be specifically targeted whereas others might somehow be treated generically.

The Agent

The Agent is the piece of software that the user is having the conversation with through the Canvas in question. The Canvas and the Agent might sometimes be one and the same thing and/or might be produced by the same company but I guess that in the general case the Canvas (e.g. an IM chat window) could be separate from the Agent (e.g. a travel Agent) which might itself rely on separate Services (e.g. a weather service, a train booking service, a plane booking service) in order to get work done.

Discoverability

How does the user discover that a particular (complex) Canvas has an Agent available and, beyond that, how do they discover what that Agent can do?

It’s the age-old ‘Clippy’ style problem. A Canvas (e.g. a chat app) can broker conversations with N Agents but the user doesn’t know that and we see this today with personal assistants offering menus via “Tell me what I can say” type commands.

It seems to me that there’s a general need for discovery and it might involve things like;

  1. Reading the instructions that came with the Canvas
  2. Asking the Canvas before…
  3. Asking the Agent.
  4. Looking up services in a directory.
  5. Being prompted by the Canvas (hopefully with some level of intelligence) when an appropriate moment seems to arrive – e.g. “did you know that I can book tickets for you?”.

and no doubt more but you need to know that you can start a conversation before you start a conversation Smile

Inputs/Outputs

There’s lots of ways to converse. We can do it by voice, by typing, by SMS. We might even stretch to include things like pointing with gamepads or waving our arms to dismiss dialogs but maybe that’s pushing things a bit far.

Equally, there’s many ways to display outputs and a lot of this is going to depend on the Canvas and device in question.

For example, if I have an Agent that knows how to find photos. I might input;

“Agent, go get me some high quality, large size images of people enjoying breakfast”

What should the output be? Maybe a set of hyperlinks? Maybe open an imaging app and display the photos themselves ready for copy/paste? Maybe offer to send me an email with all the details in so I can read it later?

I’d argue that it depends on the Canvas, the device and what I’m currently doing. If I’m walking down the street then the email option might be a good one. If I’m sitting at my PC then maybe open up an app and show me the results.

I suspect that this might get complex over time but I/O options seem to be a big part of trying to have a conversation.

Initiation

How to start a conversation?

At what point does a conversation with an Agent begin in the sense that the Agent tracks the flow of interactions back and forth such that it can build up Context and start to offer some kind of useful function?

Most Agents support some kind of “Hey, how are you?” type interaction but that’s not really the conversation opener, it perhaps comes more at the point where someone says “I need a train” or “I need a ticket” or similar.

Conversations are stateful and could potentially span across many devices and Canvases and so there’s going to need to be some kind of conversational identifier that can be (re)presented to the agent at a later point in time. The analogy in the human world would be something like;

Remember when we were talking about that holiday in Spain last week?

and, no doubt, if we’re to make conversations work in the virtual world then there is likely to be an equivalent.

Identity

An identifier for a conversation is one thing but it’s pretty much useless without a notion of the user who was involved in the conversation.

You’d imagine that this is perhaps one of the things that a Canvas can do for a user – e.g. an IM Canvas has presumably already authenticated the user so it might be able to provide some kind of token representing that identify to an Agent such that the Agent can know the differences between conversations with Mike and conversations with Michelle.

If a conversation then moves from one Canvas to another then the Agent has to be able to understand whatever identity token might come from the second Canvas as well.

I suspect that this is a roundabout way of saying that it feels to me like identity is going to be an important piece in a platform that does conversations.

Context

I’m obsessed with context Smile and I guess that a conversation with an Agent is, in some ways, about building up the context to the point where some ‘transaction’ can be completed.

That context needs to be associated with the conversation and with the identity of the user and perhaps needs to have some kind of associated lifetime such that it doesn’t stay around for ever in a situation where a conversation starts but never completes.

There’s then the question of whether that content can be;

  • pre-loaded with some of the knowledge that either the Agent or the Canvas has about the user.
  • used to add to the knowledge that either the agent or the Canvas keeps about the user after the conversation.

For example – if a user has a conversation with an Agent about a train journey then part of the context might be the start/end locations.

If one of those locations turns out to be the user’s home then that might become part of the future knowledge that an Agent or a Canvas has about the user such that in the future it can be smarter. Naturally, that needs to remain within the user’s control in terms of the consent around where it might be used and/or shared.

User Knowledge

I’m unsure whether knowledge about a user lives with an Agent, with a Canvas, with a Service or with all of them and I suspect it’s perhaps the latter – i.e. all of them.

No doubt, this is related to Identity, Context and Trust in the sense that if I use some Canvas on a regular basis (like a favourite chat app) and if it comes from a vendor that I trust then I might be prepared to share more personal data with that Canvas than I do perhaps with a an Agent which does (e.g.) concert-ticket bookings and which I only use once every 1-2 years.

The sort of knowledge that I’m thinking of here stems from personal information like age, date-of-birth, gender, height, weight, etc. through into locations like home, work and so on and then perhaps also spans into things like friends/family.

You can imagine scenarios like;

“Hey, can you ask the train ticketing service to get me a ticket from home to my brother’s place early in the morning a week on Saturday and drop him an SMS to tell him I’m coming?”

and a Canvas (or Agent) that can use knowledge about the user to plug in all the gaps around the terms ‘home’ and ‘brother’ in order to work out the locations and phone numbers is a useful thing Smile

Now, whether it’s the Canvas that turns these terms into hard data or whether it’s the Agent that does it, I’m not sure.

Trust

Trust is key here. As a user, I don’t want to have a conversation with an Agent that is then keeping or sharing data that I didn’t consent to but, equally, conversations that are constantly interrupted by privacy settings aren’t likely to progress well.

In a conversation between User<->Canvas<->Agent<->Service it’s not always going to be easy to know where the trust boundaries are being drawn or stretched and perhaps it becomes the responsibility of the Canvas/Agent to let the user know what’s going on as knowledge is disseminated? For example, in a simple scenario of;

“Hey, can you buy me train tickets from home to work next Tuesday?”

there’s a question around whether the Agent needs to prompt if it’s not aware of what ‘home’ and ‘work’ might mean and doesn’t have a means to purchase the ticket.

Also, does the Canvas attempt to help in fleshing out those definitions of ‘home’ and ‘work’ and does it do with/without the user’s explicit consent?

Dialogs

It feels like a conversational platform needs to have the ability to define dialogs for how a conversation is supposed to flow between the user and the agent.

I suspect that it probably shouldn’t be a developer who defines what the structure and the content of these dialogs should be.

I also suspect that they shouldn’t really be hard-wired into some piece of code somewhere but should, somehow, be open to being created and revised by someone who has a deep understanding of the business domain and who can then tweak the dialogs based on usage.

That would imply a need for some sort of telemetry to be captured which lets that Agent be tweaked in response to how users are actually interacting with it.

Part of defining dialogs might tie in with inputs and outputs in that you might define different sets of dialogs depending on the input/output capabilities of the Canvas that’s hosting the conversation with the Agent. It’s common to use different techniques when using speech input/output versus (say) textual input/output and so dialogs would presumably need to cater for those types of differences.

Another part of defining dialogs might be to plug in decision logic around how dialogs flow based on input from the user and responses from services.

Language Understanding and Intent

One of the challenges of defining those dialogs is trying to cater for all the possible language variations in the ways in which a user might frame or phrase a particular question or statement. There are so many ways of achieving the same result that it’s practically impossible to define dialogs that cater for everything. For example;

  • “I want to book a taxi”
  • “I need a lift to catch my plane”
  • “Can I get a car to the airport”

are simple variations of possibly the very same thing and so it feels to me like there’s a very definite need here for a service which can take all of these variations and turn them into more of a canonical representation which can report the intent that’s common across all three of them.

Without that, I think all developers are going to be building their own variant of that particular wheel and it’s not an easy wheel to build.

Termination

Just for completeness, if there is an “initiation” step to a conversation with an Agent then I guess there should be a point at which a conversation is “over” whether that be due to a time-out or whether it be that the user explicitly states that they are done.

I can see a future scenario analogous to clearing out your cookies in the browser today where you want to make sure that whatever you’ve been conversing about with some Agent via some Canvas has truly gone away.

Services

An Agent is a representative for some collection of Services. These might be a bunch of RESTful services or similar and it’s easy to think of some kind of travel Agent that provides a more natural and interactive front end to a set of existing back-end services for booking hotels, flights, trains, ferries, etc. and looking up timetables.

A platform for conversations would probably want to make it as easy as possible to call services, bring back their data and surface it into decision-logic.

Decisions

Sticking with decisions – there’s likely to be a need for making decisions in all but the simplest of conversations and those decisions might well steer the conversational flow in terms of the ‘dialogs’ that are presented to the user.

Those decisions might be based on the user’s input, the responses that come back from services invoked by the Agent or might be based on User Knowledge or some ambient context like the current date and time, weather, traffic or similar.

Some of that decision making might be influenced by use of the Agent itself – e.g. if the Agent uses telemetry to figure out that 95% of all users go with the default options across step 1 to step 5 of a conversation then maybe it can start to adapt and offer the user shortcuts based on that knowledge?

Telemetry

I’d expect an Agent to be gathering telemetry as it was progressing such that aggregate data was available across areas like;

  • Agent usage – i.e. where traffic is coming from, how long conversations last, etc.
  • Dialog flow – which paths through the Agent’s capabilities are ‘hot’ in the sense of followed by most users.
  • Dialog blockers – points where conversations are consistently abandoned.
  • Choices – the sorts of options that users are choosing as they navigate the Agent.
  • Failures – places where an agent isn’t understanding the user’s intent.

I’m sure that there’s probably a lot more telemetry that an Agent would gather – it’s definitely an important part of the picture.

History

It’s common to refer to a previous conversation from a current one and I think that over time a conversation platform needs to think about this as it’s pretty common in the real world to refer to conversations that happened at some earlier point in time including perhaps reaching back months or years.

That needs to fit with Trust but I think it would add a lot of value to an Agent to be able to cope with the idea of something like;

“I need to re-order the same lightbulbs that I ordered from you six months ago”

or similar. Whether that needs to be done by the Agent “remembering” the conversation or whether it needs to be done by one of its supporting Services taking on that responsibility, I’m not sure.

Done

That’s my initial ramble over. I need to go away and think about this some more. Please feel free to feedback as these are just my rough notes but there’s a few things in here that I think are probably going to stick with me as I think more on this topic of conversations…

Windows 10 Anniversary Update Preview, Visual Layer–Mocking Up the Lock Screen

I wanted something relatively simple to experiment with using some of the things that I’d picked up about the Visual Layer when writing these posts;

Visual Layer Posts

and from Rob’s posts;

Rob’s Posts

and, specifically, I wanted to try and do a little bit more with interactions that I’d started playing with;

Windows 10, UWP and Composition– Experimenting with Interactions in the Visual Layer

and so I thought I’d make a stab at a cheap reproduction of what I see with the Windows 10 lock-screen’s behaviour which (purely from staring at it) seems to;

  1. Slide up with the user’s finger.
  2. Fade out the text it displays as it slides
  3. On completion of “enough” of a slide, hides the text and appears to both zoom and darken the lock screen image before displaying the logon box.

Here’s a screen capture of my attempt to date;

and you’ll probably notice that it’s far from perfect but (I hope) it captures a little of what the lock-screen does.

In experimenting with this, I used a Blank UWP app on SDK preview 14388 with Win2d.uwp referenced and I had a simple piece of XAML as my UI;

  &lt;Grid
    Background=&quot;Red&quot;
    PointerPressed=&quot;OnPointerPressed&quot;
    x:Name=&quot;xamlRootGrid&quot;&gt;
    &lt;Image
      x:Name=&quot;xamlImage&quot;
      Source=&quot;ms-appx:///Assets/lockImage.jpg&quot;
      HorizontalAlignment=&quot;Left&quot;
      VerticalAlignment=&quot;Top&quot;
      Stretch=&quot;UniformToFill&quot; /&gt;
    &lt;!-- this grid is here to provide an easy place to add a blur to the image behind it --&gt;
    &lt;Grid
      x:Name=&quot;xamlBlurPlaceHolder&quot; /&gt;
    &lt;Grid
      x:Name=&quot;xamlContentPanel&quot;
      HorizontalAlignment=&quot;Stretch&quot;
      VerticalAlignment=&quot;Stretch&quot;&gt;
      &lt;StackPanel
        HorizontalAlignment=&quot;Left&quot;
        VerticalAlignment=&quot;Bottom&quot;
        Margin=&quot;48,0,0,48&quot;&gt;
        &lt;TextBlock
          Text=&quot;09:00&quot;
          FontFamily=&quot;Segoe UI Light&quot;
          Foreground=&quot;White&quot;
          FontSize=&quot;124&quot; /&gt;
        &lt;TextBlock
          Margin=&quot;0,-30,0,0&quot;
          Text=&quot;Thursday, 14th July&quot;
          FontFamily=&quot;Segoe UI&quot;
          Foreground=&quot;White&quot;
          FontSize=&quot;48&quot; /&gt;
        &lt;TextBlock
          Text=&quot;Jim's Birthday&quot;
          Margin=&quot;0,48,0,0&quot;
          FontFamily=&quot;Segoe UI Semibold&quot;
          Foreground=&quot;White&quot;
          FontSize=&quot;24&quot; /&gt;
        &lt;TextBlock
          Text=&quot;Friday All Day&quot;
          FontFamily=&quot;Segoe UI Semibold&quot;
          Foreground=&quot;White&quot;
          FontSize=&quot;24&quot; /&gt;
      &lt;/StackPanel&gt;
    &lt;/Grid&gt;
  &lt;/Grid&gt;

and you’ll probably notice that I don’t quite have the fonts or spacing quite right but it’s an approximation and then I wrote some code behind to try and achieve what I wanted;

namespace App12
{
  using Microsoft.Graphics.Canvas.Effects;
  using System;
  using System.Numerics;
  using Windows.UI;
  using Windows.UI.Composition;
  using Windows.UI.Composition.Interactions;
  using Windows.UI.Xaml;
  using Windows.UI.Xaml.Controls;
  using Windows.UI.Xaml.Hosting;
  using Windows.UI.Xaml.Input;

  public static class VisualExtensions
  {
    public static Visual GetVisual(this UIElement element)
    {
      return (ElementCompositionPreview.GetElementVisual(element));
    }
  }
  public sealed partial class MainPage : Page, IInteractionTrackerOwner
  {
    public MainPage()
    {
      this.InitializeComponent();
      this.Loaded += OnLoaded;
    }
    void OnLoaded(object sender, Windows.UI.Xaml.RoutedEventArgs e)
    {
      // The visual for our root grid
      this.rootGridVisual = this.xamlRootGrid.GetVisual();

      // Keep hold of our compositor.
      this.compositor = this.rootGridVisual.Compositor;

      // The visual for the grid which contains our text content.
      this.contentPanelVisual = this.xamlContentPanel.GetVisual();

      // And for the image
      this.imageVisual = this.xamlImage.GetVisual();

      // Set up the centre point for scaling the image 
      // TODO: need to alter this on resize?
      this.imageVisual.CenterPoint = new Vector3(
        (float)this.xamlRootGrid.ActualWidth / 2.0f,
        (float)this.xamlRootGrid.ActualHeight / 2.0f,
        0);

      // Get the visual for the grid which sits in front of the image that I can use to blur the image
      this.blurPlaceholderVisual = this.xamlBlurPlaceHolder.GetVisual();

      // Create the pieces needed to blur the image at a later point.
      this.CreateDarkenedVisualAndAnimation();

      this.CreateInteractionTrackerAndSource();

      // NB: Creating our animations here before the layout pass has gone by would seem
      // to be a bad idea so we defer it. That was the big learning of this blog post.

    }
    void CreateInteractionTrackerAndSource()
    {
      // Create an interaction tracker with an owner (this object) so that we get
      // callbacks when interesting things happen, this was a major learning for
      // me in this piece of code.
      this.interactionTracker = InteractionTracker.CreateWithOwner(this.compositor, this);

      // We're using the root grid as the source of our interactions.
      this.interactionSource = VisualInteractionSource.Create(this.rootGridVisual);

      // We only want to be able to move in the Y direction.
      this.interactionSource.PositionYSourceMode = InteractionSourceMode.EnabledWithoutInertia;

      // From 0 to the height of the root grid (TODO: recreate on resize)
      this.interactionTracker.MaxPosition = new Vector3(0, (float)this.xamlRootGrid.ActualHeight, 0);
      this.interactionTracker.MinPosition = new Vector3(0, 0, 0);

      // How far do you have to drag before you unlock? Let's say half way.
      this.dragThreshold = this.xamlRootGrid.ActualHeight / 2.0d;

      // Connect the source to the tracker.
      this.interactionTracker.InteractionSources.Add(this.interactionSource);
    }

    void CreateDarkenedVisualAndAnimation()
    {
      var darkenedSprite = this.compositor.CreateSpriteVisual();
      var backdropBrush = this.compositor.CreateBackdropBrush();

      // TODO: resize?
      darkenedSprite.Size = new Vector2(
        (float)this.xamlRootGrid.ActualWidth,
        (float)this.xamlRootGrid.ActualHeight);

      // I borrowed this effect definition from a Windows UI sample and
      // then tweaked it.
      using (var graphicsEffect = new ArithmeticCompositeEffect()
      {
        Name = &quot;myEffect&quot;,
        Source1Amount = 0.0f,
        Source2Amount = 1.0f,
        Source1 = new ColorSourceEffect()
        {
          Name = &quot;Base&quot;,
          Color = Color.FromArgb(255, 0, 0, 0),
        },
        Source2 = new CompositionEffectSourceParameter(&quot;backdrop&quot;)
      })
      {
        this.darkenImageAnimation = this.compositor.CreateScalarKeyFrameAnimation();
        this.darkenImageAnimation.InsertKeyFrame(0.0f, 1.0f);
        this.darkenImageAnimation.InsertKeyFrame(0.0f, 0.6f);
        this.darkenImageAnimation.Duration = TimeSpan.FromMilliseconds(250);

        using (var factory = this.compositor.CreateEffectFactory(graphicsEffect,
          new string[] { &quot;myEffect.Source2Amount&quot; }))
        {
          this.mixedDarkeningBrush = factory.CreateBrush();
          this.mixedDarkeningBrush.SetSourceParameter(&quot;backdrop&quot;, backdropBrush);
          darkenedSprite.Brush = this.mixedDarkeningBrush;
        }
      }
      ElementCompositionPreview.SetElementChildVisual(this.xamlBlurPlaceHolder, darkenedSprite);
    }

    void OnPointerPressed(object sender, PointerRoutedEventArgs e)
    {
      // First time around, create our animations.
      if (this.positionAnimation == null)
      {
        LazyCreateDeferredAnimations();
      }
      if (e.Pointer.PointerDeviceType == Windows.Devices.Input.PointerDeviceType.Touch)
      {
        // we send this to the interaction tracker.
        this.interactionSource.TryRedirectForManipulation(
          e.GetCurrentPoint(this.xamlRootGrid));
      }
    }
    void LazyCreateDeferredAnimations()
    {
      // opacity.
      this.opacityAnimation = this.compositor.CreateExpressionAnimation();

      this.opacityAnimation.Expression =
        &quot;1.0 - (tracker.Position.Y / (tracker.MaxPosition.Y - tracker.MinPosition.Y))&quot;;

      this.opacityAnimation.SetReferenceParameter(&quot;tracker&quot;, this.interactionTracker);

      this.contentPanelVisual.StartAnimation(&quot;Opacity&quot;, this.opacityAnimation);

      // position.
      this.positionAnimation = this.compositor.CreateExpressionAnimation();
      this.positionAnimation.Expression = &quot;-tracker.Position&quot;;
      this.positionAnimation.SetReferenceParameter(&quot;tracker&quot;, this.interactionTracker);
      this.contentPanelVisual.StartAnimation(&quot;Offset&quot;, this.positionAnimation);

      // scale for the background image when we &quot;unlock&quot;
      CubicBezierEasingFunction easing = this.compositor.CreateCubicBezierEasingFunction(
        new Vector2(0.5f, 0.0f),
        new Vector2(1.0f, 1.0f));

      // this animation and its easing don't 'feel' right at all, needs some tweaking
      this.scaleAnimation = this.compositor.CreateVector3KeyFrameAnimation();
      this.scaleAnimation.InsertKeyFrame(0.0f, new Vector3(1.0f, 1.0f, 1.0f), easing);
      this.scaleAnimation.InsertKeyFrame(0.2f, new Vector3(1.075f, 1.075f, 1.0f), easing);
      this.scaleAnimation.InsertKeyFrame(1.0f, new Vector3(1.1f, 1.1f, 1.1f), easing);
      this.scaleAnimation.Duration = TimeSpan.FromMilliseconds(500);
    }

    // From hereon in, these methods are the implementation of IInteractionTrackerOwner.
    public void CustomAnimationStateEntered(
      InteractionTracker sender, 
      InteractionTrackerCustomAnimationStateEnteredArgs args)
    {
    }
    public void IdleStateEntered(
      InteractionTracker sender, 
      InteractionTrackerIdleStateEnteredArgs args)
    {
      if (this.unlock)
      {
        // We make sure that the text disappears
        this.contentPanelVisual.Opacity = 0.0f;

        // We try and zoom the image a little.
        this.imageVisual.StartAnimation(&quot;Scale&quot;, this.scaleAnimation);

        // And darken it a little.
        this.mixedDarkeningBrush.StartAnimation(&quot;myEffect.Source2Amount&quot;, this.darkenImageAnimation);
      }
      else
      {
        sender.TryUpdatePosition(Vector3.Zero);
      }
    }
    public void InertiaStateEntered(
      InteractionTracker sender, 
      InteractionTrackerInertiaStateEnteredArgs args)
    {
    }  
    public void InteractingStateEntered(
      InteractionTracker sender, 
      InteractionTrackerInteractingStateEnteredArgs args)
    {
      this.unlock = false;
    }
    public void RequestIgnored(
      InteractionTracker sender, 
      InteractionTrackerRequestIgnoredArgs args)
    {
    }
    public void ValuesChanged(
      InteractionTracker sender, 
      InteractionTrackerValuesChangedArgs args)
    {
      if (!this.unlock &amp;&amp; (args.Position.Y &gt; this.dragThreshold))
      {
        this.unlock = true;
      }
    }
    bool unlock;
    double dragThreshold;
    InteractionTracker interactionTracker;
    VisualInteractionSource interactionSource;
    Visual rootGridVisual;
    Visual contentPanelVisual;
    Visual blurPlaceholderVisual;
    Compositor compositor;
    ExpressionAnimation positionAnimation;
    ExpressionAnimation opacityAnimation;
    ScalarKeyFrameAnimation darkenImageAnimation;
    CompositionEffectBrush mixedDarkeningBrush;
    Vector3KeyFrameAnimation scaleAnimation;
    Visual imageVisual;
  }
}

What’s that code doing?

  1. At start-up
    1. getting hold of a bunch of Visuals for the various XAML UI elements.
    2. creating a Visual (darkenedSprite) which lives in the Grid named xamlBlurPlaceHolder and which will effectively paint itself with a mixed combination of the colour black and the image which sits under it in the Z-order.
    3. creating an animation (darkenImageAnimation) which will change the balance between black/image when necessary.
    4. creating an interaction tracker and an interaction source to track the Y movement of the touch pointer up the screen within some limits.
  2. On pointer-pressed
    1. Creating an animation which will cause the text content to slide up the screen wired to the interaction tracker
    2. Creating an animation which will cause the text content to fade out wired to the interaction tracker
    3. Creating an animation which will later be used to scale the image as the lock-screen is dismissed (this could, perhaps, be done earlier)
    4. Passing the pointer event (if it’s touch) across to the interaction tracker

In building that out, I learned 2 main things. One was that things have changed since build 10586 and I need to read the Wiki site more carefully as talked about in this post.

The other was around how to trigger the ‘dismissal’ of my lock-screen at the point where the user’s touch point has travelled far enough up the screen.

I was puzzled by that for quite a while. I couldn’t figure out how I was meant to know what the interaction tracker was doing and I kept looking for events without finding any.

Equally, I couldn’t figure out how to debug what the interaction tracker was doing when my code didn’t work.

That changed when I came across IInteractionTrackerOwner and the InteractionTracker.CreateWithOwner() method. Whether I have this right or not, it let me plug code (and diagnostics) into the InteractionTracker and I used the ValueChanged method to try and work out when the user’s touch point has gone 50% of the way up the screen so that I can then dismiss the lock-screen.

I don’t dismiss it immediately though. Instead, I wait for the IdleStateEntered callback and in that code I try to take steps to;

  1. Set the opacity of the text panel to 0 so that it disappears.
  2. Begin the animation on the image Visual so as to zoom it a little
  3. Begin the animation on the composite brush that I have so as to darken the image by mixing it with Black.

The other thing that I learned was that I don’t understand the lighting features of the Visual Layer well enough yet and that I need to explore them some more in isolation to try and work that out.

But, the main thing for me here was to learn about IInteractionTrackerOwner and hence sharing that here (even in this rough, experimental form).

Windows 8.1, WPF, Kinect for Windows V2 and (Not Quite) Skeletal Fingerprints

This post is just to share some code that I wrote quite a while ago for fun.

At Microsoft’s UK campus there are some corridors that have gates which allow people to freely walk in one direction but which only allow people with the right ID cards to go in the other.

This set me wondering whether it might be possible to develop code with the Kinect for Windows V2 which monitored people walking towards the camera and;

  1. Took skeletal measurements as they approached the camera to build up average ‘limb lengths’ based on some set of limbs and measurement counts that can be configured.
  2. Grabbed a head and shoulders photo of each person based on knowledge of where their head and shoulders are within the camera’s view.
  3. Stored both of (1) and (2) in the cloud using some configurable table and blob storage.
  4. Scanned previously stored measurements in the cloud to determine whether a person of very similar ‘shape and size’ has been previously seen based on some configurable tolerance value.
  5. Ultimately, opened the door based on recognising the user’s skeleton.

Now, in truth, I’ve never really got this to work Smile I’m not sure whether the idea of a ‘skeletal fingerprint’ is a flawed idea in the first place but I got way too many false positives to consider asking Microsoft to use my new system for their security Winking smile but I had some fun in putting it together.

However, this was prior to the arrival of Microsoft’s Cognitive Services and I may revisit the code in coming weeks/months to see if I can make a better attempt at it in the light of those new services coming along as perhaps I can combine my not so successful idea of ‘skeletal fingerprint’ with an additional call to Cognitive Services to do facial identification and produce a better result than I previously did.

As an aside, this seems like a reasonable use of Cognitive Service’s facial identification service. I wouldn’t want a security camera constantly sending data to the cloud but, instead, I’d want to pair it with some smart camera which knew when new people entered/exited the video frames and could capture a photo of those people and send it off to the cloud for identification. That avoids lots of (costly) extraneous calls to the cloud.

That’s in the future. For now, I’m just sharing this code in case anyone wants to play with it and develop it.

Where’s the Code?

The code I’m talking about is in this GitHub repository;

https://github.com/mtaulty/KinectBodyMeasurement

What’s with the Weird Code Structure?

The structure of this code is a bit unusual in that I wanted to be able to build out both a WPF application and a Windows application from the same code. The Windows application is a Windows 8.1 app rather than a UWP app because there isn’t a Kinect for Windows V2 SDK for UWP and so this code targets Windows 8.1 but will, naturally, run on Windows 10.

The Kinect for Windows V2 SDK is structured such that, with one or two conditional compilation statements around namespaces, it’s possible to write the same code for WinRT and WPF and so that’s what I set out to do although it does mean that I haven’t (e.g.) used something like Win2D for some drawing because Win2D only exists in WinRT, not in WPF.

If you look at the code structure, you’ll see that there is a project called WpfApp and another called WinRTApp and neither one seems to contain much code because they both share a lot of code by referencing a shared project named SharedApp.

image

and most of the code for these 2 apps is in that project and is largely identical across WPF and WinRT;

image

Within that code you’ll find quite a bit of;

image

but, hopefully, not too much.

Each app project also references 3 library projects named BodyReaders, Measurements and Storage.

image

and so each of these libraries needs to be built both for WPF and for WinRT in order to pick up the different dependencies across those two platforms.

The same ‘shared project’ approach is taken again such that there are 3 shared library projects of code;

image

and then there’s a NetFramework folder which contains 3 library projects with each referencing the source from the corresponding shared project and there’s a WinRT folder which does the same.

image

and, in this way, we have a WinRT App and a WPF app sharing a lot of source and both taking dependencies on 3 libraries which also share common source that then needs to be built for WinRT and .NET Framework respectively.

How Is It Configured?

There are elements of configuration needed here whether running the WinRT app or the WPF app. The default configuration lives in 2 files;

image

The global.json files configures items as below;

{
    "CloudAccount": "your cloud storage account goes here",
    "CloudKey": "your cloud storage account key goes here",
    "CloudTable": "skeletalData",
    "MeasurementTolerance": "0.05",
    "LeastSquaresTolerance" :  "10.0",
    "MeasurementFrameCount": "50",
    "IrScaleRange": "200",
    "IrScaleLow": "30",
    "CloudBlobContainerName": "skeletalphotos",
    "CloudRowScanSize" :  "5"
}  

The idea of the values CloudAccount and CloudKey is that they will be fed into the constructor of a StorageCredentials  instance in order to talk to Azure storage.

The CloudTable value is the name of a table within the storage account which will be used to store key/value pairs relating to skeletal data captured.

The CloudBlobContainerName value is the name of a blob container within which the app will store photos captured.

The measurements.json file configures items as below;


    {
        "Name": "Spine",
        "Start": "SpineBase",
        "End": "SpineShoulder",
        "IsPartitionKey": "true"
    },
    {
        "Name": "Chest",
        "Start": "ShoulderLeft",
        "End" :  "ShoulderRight"
    },

This is defining the set of measurements that we want to capture for each user in terms of ‘limbs’. The names of the Start/End values need to correspond to a value from the JointType enumeration from the Kinect SDK.

The values of the Name properties will be used as keys within Azure table storage.

Each of these ‘limb’ measurements will be computed MeasurementFrameCount times before an average value is produced and used.

Note that one limb is defined as IsPartitionKey and the average value of that measurement will be used to partition data within the Azure table. However, the ‘raw’ average value of that measurement is not used. Instead, the value is multiplied by 10 and then its integral part is taken such that a person with a 58cm spine length and another with a 51cm spine length would both live in a partition keyed off the value ‘5’.

When a person is encountered and average measurements have been taken, the app attempts to ‘find’ them in Azure table storage by;

  1. Matching on the partition key.
  2. Retrieving up to CloudRowScanSize rows from table storage where the values for each limb measurement that has been computed are within MeasurementTolerance of the measurement captured. It may be that the cloud table storage has more/fewer limb measurements than that currently configured for the app but that shouldn’t matter too much.

Once the app has a candidate set of rows from Azure table storage which may or may not match the individual being matched, it then computes a “sum of least squares” difference across all the average measurements and selects the row with the lowest difference that is also less than the LeastSquaresTolerance value.

The IrScaleLow and IrScaleRange values are used to scale data from the Infra Red frames off the camera to be between the low value and the high value. Note that this only comes into play if the UseInfrared property of the VideoControl user control is set to true which it is not in the code base at the time of writing.

That’s all of the configuration.

What Does Running the App Do?

I can run either the WPF application or the WinRT application and (as long as I’ve got my Kinect for Windows V2 camera plugged in) I’ll get a video (or IR) display like the one below.

image

Once the app has recognised that a person has arrived in the frame, it will start to do calculations;

image

Once it has done calculations, it will display a floating panel showing the results;

image

and it will then visit the cloud to see if it can make a match against an existing table record. If not, the circle will fill orange;

image

and the person will be given a new GUID and an entry will be added to Azure table storage with the measurements captured as shown below;

image

and a new blob dropped into the configured photos table with the same GUID;

image

containing the photo;

image

and that’s pretty much it.

If the system does recognise the user based on the measurements then it will display a green-filled circle;

image

which indicates that it will not create a new GUID and a new entry in the Azure table and blob storage as the person’s been recognised.

Wrapping Up

I’ve had this code for a long time but have never written it up so I found a few spare cycles today to do that.

It was built 100% ‘just for fun’ but I thought I’d share in case anyone else was interested or wanted to take pieces of it and improve it Smile

What I’d perhaps like to do next is to extend this by adding in some use of facial identification via Cognitive Services to see if I could build a system that worked a little better by using a combination of skeletal measurements with facial identity such that both mechanisms were used to determine identify.

That would take a little bit of work – I’ll post an update as/when I get there.

Windows 10 Anniversary Update 14388, UWP, Visual Layer–Offsets on Preview SDK 14388

Just a small thing but I’m posting it here in case it helps anyone else. I spent a good few ‘minutes’ trying to figure out why this piece of XAML;

   <Grid Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">
        
        <Rectangle x:Name="xamlRectangle" Width="96" Height="96" Fill="Black" HorizontalAlignment="Left" VerticalAlignment="Bottom">
            
        </Rectangle>
    </Grid>

married up with this piece of code behind;

  public sealed partial class MainPage : Page
  {
    public MainPage()
    {
      this.InitializeComponent();
      this.Loaded += this.OnLoaded;
    }
    void OnLoaded(object sender, RoutedEventArgs e)
    {
      this.rectangleVisual = ElementCompositionPreview.GetElementVisual(
        this.xamlRectangle);

      this.rectangleVisual.Offset = new Vector3(100, -100, 0);
    }
    Visual rectangleVisual;
  }

Wasn’t giving me the result that I expected on Windows 10 Anniversary Update 14388 with SDK preview 14388.

What it gives me is a black rectangle which is not offset by the 100,-100 vector that I’ve specified.

1

However, if I simply retarget my project to target SDK build 10586 and run it on the same machine then I find that I get the results I expect.

2

I initially thought this was a bug but I learned that it was a change between 10586 and the builds post 14332 – the detail is here and I think I need to read it at least another 5 times before I understand it:-)

Update

I have now read that article more than 5 times and I’m still trying to understand it although I think I have slightly more of a handle on its implications even if I don’t quite understand the mechanics.

In terms of dealing with Offset, I think I understand that if XAML thinks that it has set some value for Visual.Offset then it will override whatever value I set for Visual.Offset when it comes to do a layout pass.

For example, in the code/XAML scenario that I have further up in this post I have a Rectangle which is set to be arranged in the Bottom/Left of its parent Grid.

I also have a Loaded event handler which attempts to set the Offset property to (100,-100,0) and on SDK 14388 nothing happens whereas on 10586 the offset gets applied.

It’s worth saying that the same thing appears to happen if I try and start an animation which targets “Offset” from the Loaded handler – it doesn’t do anything on 14388.

Why? Because after my code changes the Offset value to (100,-100,0) XAML is doing a layout pass and setting it back to whatever value is necessary to position the rectangle at the bottom left of the Grid.

If I change my Loaded handler to something like this;

    async void OnLoaded(object sender, RoutedEventArgs e)
    {
      this.rectangleVisual = ElementCompositionPreview.GetElementVisual(
        this.xamlRectangle);

      await Task.Delay(5000);

      this.rectangleVisual.Offset = new Vector3(100, -100, 0);
    }

then I notice different behavior in that after 5 seconds the rectangle disappears:-) Why? Presumably because my asynchronously waiting for 5 seconds gives the thread time to go back and process the layout work before completing this function which then applies an Offset of (100,-100,0) from the origin of the Window which makes the rectangle (at 96 pixels high) disappear.

If I then resize the Window to force another layout pass then the Rectangle re-appears back in the bottom left hand corner of the Window.

So, the XAML layer has had to take an active hand in the positioning of this Rectangle and it attempts to reassert its authority every time it does a layout.

However, if I change my XAML such that the VerticalAlignment of the Rectangle is set to “Top” then the XAML layer doesn’t explicitly set the Visual.Offset value on any of its layout passes and so there’s no need for the artificial delay that I added into the code – the Rectangle positions itself off screen from the start and stays that way regardless of resizing the Window because the XAML pieces have never been involved in explicitly setting the Offset property in the first place.

I must admit that I find this a little tricky to reason about and I guess that it would lead me towards not letting XAML position elements that I then wanted reposition using the Visual layer although that feels a bit restrictive in terms of not being then able to use a lot of ‘goodness’ that XAML’s layout system can give me like proportional sizing and so on. I perhaps need to experiment a little more with this to figure out more around how it works and how to best code with it rather than against it:)

Update 2

Rob has written a post here which digs into more detail around what I was seeing in this post and I think the 2 diagrams that he’s posted help massively in explaining what happens in the Anniversary Update versus the November Update.

Windows 10, UWP and Composition– Experimenting with Interactions in the Visual Layer

I wanted to experiment with how interactions work in the Visual Layer and so on the November update of Windows 10 (10586.x) I made a new, blank UWP app with a MainPage that presented a green rectangle as below;

<Page
  x:Class="App33.MainPage"
  xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
  xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
  xmlns:local="using:App33"
  xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
  xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
  mc:Ignorable="d">

  <Grid
    Background="{ThemeResource ApplicationPageBackgroundThemeBrush}"
    x:Name="gridParent">
    <Rectangle
      x:Name="rectangle"
      Stroke="Black"
      StrokeThickness="4"
      Width="388"
      Height="388"
      Fill="Lime"
      RenderTransformOrigin="0.5,0.5"
      Tapped="XamlRectangleTapped">
      <Rectangle.RenderTransform>
        <CompositeTransform
          x:Name="transform" />
      </Rectangle.RenderTransform>
    </Rectangle>
  </Grid>
</Page>

and I’ve tried to set this up so that I can use pinch/rotate gestures to manipulate the rectangle;

using Windows.UI.Xaml.Controls;
using Windows.UI.Xaml.Input;

namespace App33
{
  public sealed partial class MainPage : Page
  {
    public MainPage()
    {
      this.InitializeComponent();
      this.Loaded += OnLoaded;
    }
    void OnLoaded(object sender, Windows.UI.Xaml.RoutedEventArgs e)
    {
      this.rectangle.ManipulationMode =
        ManipulationModes.Rotate | ManipulationModes.Scale;

      this.rectangle.ManipulationDelta += OnManipulationDelta;
    }
    void OnManipulationDelta(object sender, ManipulationDeltaRoutedEventArgs e)
    {
      this.transform.Rotation += e.Delta.Rotation;
      this.transform.ScaleX *= e.Delta.Scale;
      this.transform.ScaleY *= e.Delta.Scale;
    }
    void XamlRectangleTapped(object sender, TappedRoutedEventArgs e)
    {
    }
  }
}

and that works nicely enough;

image

and I added a little code to add another, orange rectangle painted by the Visual or Composition layer and created when the first tap happens on the green rectangle. At that point, the class file looks like;

namespace App33
{
  using System.Numerics;
  using Windows.UI;
  using Windows.UI.Composition;
  using Windows.UI.Xaml.Controls;
  using Windows.UI.Xaml.Hosting;
  using Windows.UI.Xaml.Input;

  public sealed partial class MainPage : Page
  {
    public MainPage()
    {
      this.InitializeComponent();
      this.Loaded += OnLoaded;
    }
    void OnLoaded(object sender, Windows.UI.Xaml.RoutedEventArgs e)
    {
      this.rectangle.ManipulationMode =
        ManipulationModes.Rotate | ManipulationModes.Scale;

      this.rectangle.ManipulationDelta += OnManipulationDelta;

      this.compositor = ElementCompositionPreview.GetElementVisual(this.rectangle).Compositor;
    }
    void OnManipulationDelta(object sender, ManipulationDeltaRoutedEventArgs e)
    {
      this.transform.Rotation += e.Delta.Rotation;
      this.transform.ScaleX *= e.Delta.Scale;
      this.transform.ScaleY *= e.Delta.Scale;
    }
    void XamlRectangleTapped(object sender, TappedRoutedEventArgs e)
    {
      if (this.rectangleVisual == null)
      {
        this.rectangleVisual = this.compositor.CreateSpriteVisual();

        this.rectangleVisual.Brush = this.compositor.CreateColorBrush(Colors.Orange);

        this.rectangleVisual.Size = new Vector2(
          (float)this.gridParent.ActualWidth / 2.0f,
          (float)this.gridParent.ActualHeight / 2.0f);

        this.rectangleVisual.Offset = new Vector3(
          ((float)this.gridParent.ActualWidth / 2.0f) - this.rectangleVisual.Size.X / 2.0f,
          ((float)this.gridParent.ActualHeight / 2.0f) - this.rectangleVisual.Size.Y / 2.0f,
          0.0f);

        ElementCompositionPreview.SetElementChildVisual(this.gridParent, this.rectangleVisual);
      }
    }
    SpriteVisual rectangleVisual;
    Compositor compositor;
  }
}

and then I didn’t really know what to expect.

The orange ‘visual layer’ rectangle completely obscures the green ‘XAML’ and so I had a split expectation around whether the green rectangle would or would not still receive the touch events that let it be rotated and scaled.

That is – does the visual layer intercept those events? Here’s the answer;

image

So, from the point of view of the touch events travelling towards the green rectangle, the orange rectangle painted by the Visual layer seems to be invisible and the events pass through it.

That’s not entirely surprising and if the events had been impacted by the Visual layer then I’d have wondered how that might be possible but I wouldn’t have necessarily bet money on it prior to having tried it out Smile

With that trial out of the way, how would I actually process input on that orange rectangle? There aren’t any events visible on a Visual or its hierarchy;

image

and so maybe it’s time for a video that talks about interactions in the Visual Layer;

image

What did that tell me?

That some work was done in the November update such that some of the hidden inner-workings of a XAML ScrollViewer have been surfaced via a ManipulationPropertySet.

PropertySets are explained in that same article and I came across them in this post and they’re interesting in the way in which they can be used in expression based animations.

I’d like to come back and experiment with that ScrollViewer functionality but, for the moment, I went past it in the video and learned that in the Anniversary Update there’s a new “interaction tracker” component that it seems runs the algorithms to work out what type of interactions are going on and, following the diagram from the video, it looks like it can take input from different sources with one being a “Visual Interaction Source”;

image

Based on that initial description, in my mind I’m thinking of this a little bit like the XAML-based manipulation engine which I made use of to rotate and scale my green rectangle without ever having to write the explicit code to handle the pointer events and so on but it remains to be seen as to whether that’s a good or bad analogy.

Having watched the video, I felt that I mostly understood it from the point of view of a set of concepts but I didn’t feel that I’d actually be able to code it myself and so I thought that I should give that a whirl to get a feel for it and so I switched to my development machine running the Anniversary Update and tried to make a ‘Hello World’ on the latest preview SDK.

I made a completely blank UWP project (i.e. no XAML in there) with the single code file;

namespace App10
{
  using System.Numerics;
  using Windows.ApplicationModel.Core;
  using Windows.Devices.Input;
  using Windows.UI;
  using Windows.UI.Composition;
  using Windows.UI.Composition.Interactions;
  using Windows.UI.Core;

  class MainView : IFrameworkView, IFrameworkViewSource
  {
    // The first 2 functions in this source file are the 2 that really do anything
    // 'new' from the point of view of this blog post. It's where the 
    // VisualInteractionSource and InteractionTracker are set up and used.
    public void SetWindow(CoreWindow window)
    {
      this.window = window;
      this.CreateCompositorAndRootContainer();
      this.CreateSpriteVisualForRectangle();
      this.CreateInteractionTracker();

      this.window.PointerPressed += (s, e) =>
      {
        if (e.CurrentPoint.PointerDevice.PointerDeviceType == PointerDeviceType.Touch)
        {
          // TBD: I can get an UnauthorizedAccessException here if I pass 
          // a point that's beyond the bounds of my rectangle and I haven't
          // figured out why yet.
          this.interactionSource.TryRedirectForManipulation(e.CurrentPoint);
        }
      };
    }
    void CreateInteractionTracker()
    {
      this.tracker = InteractionTracker.Create(this.compositor);

      // TBD: I'm unsure as to whether I want my source here to be the visual itself
      // or the container it lives in but I've gone with the visual as I guess I
      // really want interactions on that rather than the whole window.
      this.interactionSource = VisualInteractionSource.Create(this.spriteVisual);

      this.interactionSource.ScaleSourceMode = InteractionSourceMode.EnabledWithoutInertia;
      this.tracker.InteractionSources.Add(this.interactionSource);
      this.tracker.MinScale = 1.0f;
      this.tracker.MaxScale = 2.0f;

      var scaleAnimation = this.compositor.CreateExpressionAnimation("Vector3(tracker.Scale, tracker.Scale, 1.0f)");
      scaleAnimation.SetReferenceParameter("tracker", this.tracker);

      this.spriteVisual.StartAnimation("Scale", scaleAnimation);
    }
    // From hereon in, everything in this code file has been talked about in previous
    // blog posts so becomes 'boilerplate' to some extent.
    public static void Main()
    {
      CoreApplication.Run(new MainView());
    }
    public MainView()
    {
    }
    public IFrameworkView CreateView()
    {
      return (this);
    }
    public void Run()
    {
      this.window.Activate();
      this.window.Dispatcher.ProcessEvents(CoreProcessEventsOption.ProcessUntilQuit);
    }
    void CreateSpriteVisualForRectangle()
    {
      this.spriteVisual = this.compositor.CreateSpriteVisual();

      this.spriteVisual.Size = new Vector2(
        (float)this.window.Bounds.Width / 4.0f,
        (float)this.window.Bounds.Height / 4.0f);

      this.spriteVisual.Offset = new Vector3(
        ((float)this.window.Bounds.Width / 2.0f) - ((float)this.spriteVisual.Size.X / 2.0f),
        ((float)this.window.Bounds.Height / 2.0f) - ((float)this.spriteVisual.Size.Y / 2.0f),
        0.0f);

      this.spriteVisual.CenterPoint = new Vector3(
        this.spriteVisual.Size.X / 2.0f,
        this.spriteVisual.Size.Y / 2.0f,
        0.0f);

      this.spriteVisual.Brush = this.compositor.CreateColorBrush(Colors.Orange);

      this.containerVisual.Children.InsertAtTop(this.spriteVisual);
    }

    void CreateCompositorAndRootContainer()
    {
      this.compositor = new Compositor();
      this.containerVisual = this.compositor.CreateContainerVisual();
      var target = this.compositor.CreateTargetForCurrentView();
      target.Root = this.containerVisual;
    }

    public void Initialize(CoreApplicationView applicationView)
    {
    }
    public void Uninitialize()
    {
    }
    public void Load(string entryPoint)
    {
    }
    VisualInteractionSource interactionSource;
    InteractionTracker tracker;
    SpriteVisual spriteVisual;
    CoreWindow window;
    ContainerVisual containerVisual;
    Compositor compositor;
  }
}

and it’s only the first 2 functions of this class (as listed above) that do anything that I haven’t mentioned in previous blog posts on this topic. It’s those first two functions that;

  1. Pass on pointer events from the window to the VisualInteractionSource.
  2. Set up the InteractionTracker in the first place along with the VisualInteractionSource which feeds it the events.

In setting up that InteractionTracker in the method CreateInteractionTracker, I try to limit it to scale between 1.0 and 2.0 and it’s worth saying that the UI allows me to go beyond those limits and then ‘snaps back’ to them when I complete a pinch gesture.

The other thing that happens within that method is the somewhat magical tie-up between the InteractionTracker and the visual which is done by the expression animation. That is – neither the Visual nor the animation need to know anything about the InteractionTracker because it’s just producing a bunch of properties (in a PropertySet) and the expression animation already knows how to deal with expressions that come from PropertySets.

I like that Smile

The code works, I can scale the rectangle as intended. One thing I’d flag is that I expected to be able to do rotation here and either it’s not in the bits that I have here or I’m missing how to do it as I can only see how to pick up changes to the scale and to the position. I don’t know about other gestures like rotate, flick, etc.

I’d like to try and do something a bit more “realistic” with this capability. Naturally, there are the demos in the video referenced above which look really nice but I’d like to create something simpler of my own to try this out on.

That’ll be a follow on post as will some sort of experiment around what it’s like to have interactions at both the XAML layer and the Visual layer.

Windows 10, UWP and Experimenting with Inking onto a Map Control

There’s a demo in this video (at the 7min mark, sorry for not being able to directly link to it here);

where Brian seems to tap on a map with his pen to mark one spot, tap again to mark another and then draw an ink line between the two points and the map responds by giving a measurement of the line that’s been drawn and can use that as the basis for getting directions.

That may or may not be a feature of the Maps app on the Windows 10 Anniversary Edition, we’ll have to wait and see what happens when the OS ships but it’s a cool feature if it’s there Smile

What interested me more though was how this might be achieved? That is – how do you combine the platform pieces which are primarily;

such that you can ink on a map in this way without losing the regular capabilities of the MapControl?

I thought about this for a while and tried a few approaches and it wasn’t as simple as I might have hoped it would be.

The Ideal Approach

In an ideal world, I would place an InkCanvas on top of a MapControl in the z-order and I’d then somehow tell the InkCanvas to pass through all non-pen related events to the MapControl underneath it while waiting for some kind of sequence of;

  • Tap to mark position 1
  • Tap to mark position 2
  • Line drawn between position 1 and 2

which it could then (perhaps) pass down to the MapControl to work out routes or similar.

The challenge with this is that the InkCanvas is very definitely not a transparent control in the sense that if you put it over some other content then pointer events will not reach that underlying content. That’s even the case if you tell the InkCanvas to set its inking mode to process only pen events (the default). This isn’t surprising in the XAML world, it’s just “how it is”, events don’t travel through controls.

It might be possible to capture all the pointer events on the InkCanvas and then somehow manually ‘pass them on’ to the underlying MapControl but I suspect that will always be a little bit ‘leaky’ in that I doubt that the MapControl surfaces APIs which can perfectly mimic the way in which it responds to gestures like pinch, rotate, etc.

I didn’t think that approach would be viable and so I set about trying a few other approaches which all centre around the assumption that mouse/touch events should drive the map interaction and pen events should (depending on context) be picked up by an InkCanvas that needs to be activated on demand.

That activation means being able to pick up pointer events at some level ‘over’ or ‘within’ the MapControl so as to be able to tell when a pen is being used versus when a mouse/touch are being used and I tried a few ways of doing this as below…

Approach One – Ask the Map.

The MapControl has events like MapTapped which feels like it might be useful but I got blocked in the early stages because the event abstracts away the details of the Pointer event which caused it to fire.

That is, if I handle MapTapped then I can’t tell whether the event was fired by a pen or by mouse/touch and so I don’t know whether to make some InkCanvas visible or not.

Initially, then, these sorts of events don’t really seem to help and so I went on to another approach.

Approach Two – Handle Pointer Events ‘Around’ the Map

My next attempt was to try and use lower-level pointer events around the MapControl in the sense that I’m trying to get hold of the pointer events either before the MapControl gets them or maybe after it gets them.

This turns out to be ‘a bit more difficult’ than I’d hoped for and it seems that this is because the MapControl is quite ‘hungry’ when it comes to pointer events.

Here’s a quick example. If I have this UI;

<Page
  xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
  xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
  xmlns:local="using:MapApp"
  xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
  xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
  xmlns:Maps="using:Windows.UI.Xaml.Controls.Maps"
  x:Class="MapApp.MainPage"
  mc:Ignorable="d">

  <Grid
    Background="{ThemeResource ApplicationPageBackgroundThemeBrush}"
    PointerEntered="OnGridPointerEntered"
    PointerExited="OnGridPointerExited"
    PointerPressed="OnGridPointerPressed">

    <Button
      HorizontalAlignment="Stretch"
      VerticalAlignment="Stretch"
      Content="Button" />
  </Grid>
</Page>

with this code-behind;

namespace MapApp
{
  using System.Diagnostics;
  using Windows.UI.Xaml.Controls;
  using Windows.UI.Xaml.Input;

  public sealed partial class MainPage : Page
  {
    public MainPage()
    {
      this.InitializeComponent();
    }
    private void OnGridPointerEntered(object sender, PointerRoutedEventArgs e)
    {
      Debug.WriteLine(nameof(OnGridPointerEntered));
    }

    private void OnGridPointerExited(object sender, PointerRoutedEventArgs e)
    {
      Debug.WriteLine(nameof(OnGridPointerExited));
    }

    private void OnGridPointerPressed(object sender, PointerRoutedEventArgs e)
    {
      Debug.WriteLine(nameof(OnGridPointerPressed));
    }
  }
}

Then I find that as I move my mouse into and out of this Window and click on the button then I see these traces in my output window;

image

Ok, so I get PointerEntered and PointerExited and I’m not entirely surprised that I don’t get PointerPressed because I have a Button taking all of the available space and Button turns raw pointer events into some higher order event “Click” so if I wanted to see those events as well I’d have to be a bit more sneaky;

namespace MapApp
{
  using System.Diagnostics;
  using Windows.UI.Xaml;
  using Windows.UI.Xaml.Controls;
  using Windows.UI.Xaml.Input;

  public sealed partial class MainPage : Page
  {
    public MainPage()
    {
      this.InitializeComponent();
      this.Loaded += OnLoaded;
    }

    void OnLoaded(object sender, RoutedEventArgs args)
    {
      this.AddHandler(
        UIElement.PointerPressedEvent,
        new PointerEventHandler(
          (s,e) =>
          {
            Debug.WriteLine("The real pressed handler");
          }
        ),
        true);          
    }

    void OnGridPointerEntered(object sender, PointerRoutedEventArgs e)
    {
      Debug.WriteLine(nameof(OnGridPointerEntered));
    }

    void OnGridPointerExited(object sender, PointerRoutedEventArgs e)
    {
      Debug.WriteLine(nameof(OnGridPointerExited));
    }

    void OnGridPointerPressed(object sender, PointerRoutedEventArgs e)
    {
      Debug.WriteLine(nameof(OnGridPointerPressed));
    }
  }
}

and then I get the output that I want;

image

and that’s all well understood and the event handler added by AddHandler fires before the Button’s Click event but let’s say that I now replace this Button with a MapControl;

  <Grid
    Background="{ThemeResource ApplicationPageBackgroundThemeBrush}"
    PointerEntered="OnGridPointerEntered"
    PointerExited="OnGridPointerExited"
    PointerPressed="OnGridPointerPressed">

    <Maps:MapControl
      HorizontalAlignment="Stretch"
      VerticalAlignment="Stretch"/>
  </Grid>

and I leave my code-behind alone then the results that I see are;

image

No, not a single event. No pointer entered, no pointer exited and no pointer pressed. It seems that the MapControl swallows all of those events and doesn’t give me much of a chance to get involved. Changing the UI to;

<Page
  xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
  xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
  xmlns:local="using:MapApp"
  xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
  xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
  xmlns:Maps="using:Windows.UI.Xaml.Controls.Maps"
  x:Class="MapApp.MainPage"
  mc:Ignorable="d">

  <Grid
    Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">

    <Maps:MapControl
      HorizontalAlignment="Stretch"
      VerticalAlignment="Stretch"/>
  </Grid>
</Page>

and the code behind to;

namespace MapApp
{
  using System.Diagnostics;
  using Windows.UI.Xaml;
  using Windows.UI.Xaml.Controls;
  using Windows.UI.Xaml.Input;

  public sealed partial class MainPage : Page
  {
    public MainPage()
    {
      this.InitializeComponent();
      this.Loaded += OnLoaded;
    }

    void OnLoaded(object sender, RoutedEventArgs args)
    {
      this.AddHandler(
       UIElement.PointerEnteredEvent,
       new PointerEventHandler(
         (s, e) =>
         {
           Debug.WriteLine("The real entered handler");
         }
       ),
       true);

      this.AddHandler(
       UIElement.PointerEnteredEvent,
       new PointerEventHandler(
         (s, e) =>
         {
           Debug.WriteLine("The real exited handler");
         }
       ),
       true);

      this.AddHandler(
        UIElement.PointerPressedEvent,
        new PointerEventHandler(
          (s,e) =>
          {
            Debug.WriteLine("The real pressed handler");
          }
        ),
        true);                
    }
  }
}

makes no difference, I still don’t see any events firing here.

I wondered if I could do something at the “Window level” here, that is;

    void OnLoaded(object sender, RoutedEventArgs args)
    {
      var window = CoreWindow.GetForCurrentThread();
      window.PointerEntered += (s, e) =>
      {
      };
      window.PointerExited += (s, e) =>
      {
      };     
    }

but those events don’t fire either once I’ve got a MapControl taking up the entirety of my UI’s space.

Approach Three – An InkCanvas In Front of the Map

I went back to explore the ‘ideal approach’ again, wondering whether I might be able to have the InkCanvas initially visible but in “passive” mode using some of the control’s capabilities around “unprocessed input”.

That is, I made my UI;

<Grid
    Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">

    <Maps:MapControl/>

    <InkCanvas
      x:Name="inkCanvas" />
  </Grid>

 

with some code;

namespace MapApp
{
  using Windows.Devices.Input;
  using Windows.UI.Core;
  using Windows.UI.Input.Inking;
  using Windows.UI.Xaml;
  using Windows.UI.Xaml.Controls;
  using Windows.UI.Xaml.Controls.Maps;

  public sealed partial class MainPage : Page
  {
    public MainPage()
    {
      this.InitializeComponent();
      this.Loaded += OnLoaded;
    }
    void OnLoaded(object sender, RoutedEventArgs args)
    {
      inkCanvas.InkPresenter.InputDeviceTypes = 
        CoreInputDeviceTypes.Pen |
        CoreInputDeviceTypes.Mouse |
        CoreInputDeviceTypes.Touch;

      inkCanvas.InkPresenter.InputProcessingConfiguration.Mode = InkInputProcessingMode.None;
      inkCanvas.InkPresenter.UnprocessedInput.PointerEntered += OnEntered;
    }
    void OnEntered(Windows.UI.Input.Inking.InkUnprocessedInput sender, PointerEventArgs args)
    {
      if (args.CurrentPoint.PointerDevice.PointerDeviceType != PointerDeviceType.Pen)
      {
        this.inkCanvas.Visibility = Visibility.Collapsed;
      }
      else
      {
        this.inkCanvas.InkPresenter.InputProcessingConfiguration.Mode = 
          InkInputProcessingMode.Inking;
      }
    }
  }
}

This “kind of works” in the sense that if I approach the UI with a pen then I can ink whereas if I approach the UI with a mouse or with touch I can manipulate the map.

I say “kind of” because it’s fair to say that if I approach the UI with touch then the very first touch event is used up in dismissing the InkCanvas and a user would notice that they always have to tap twice which isn’t perfect.

The bigger problem though is that this trick is essentially;

  • Have an InkCanvas dormant over the whole UI
  • Look at the first pointer event. If it is a pen then activate the InkCanvas. Otherwise, hide the InkCanvas so that further pointer events go to the map.
  • When the pointer that we’re dealing with exits, put the InkCanvas back, dormant, over the whole UI so that we can repeat the process we’ve just gone through.

The challenge is that I can’t do that last step without some kind of “pointer exiting” event from the MapControl and I don’t get one.

Approach Four – Put the InkCanvas In the Map

This approach is a little like the first approach but I remembered that the MapControl has an ability to host XAML elements via its Children property. Now, the documentation talks about;

“Display XAML user interface elements such as a Button, a HyperlinkButton, or a TextBlock by adding them as Children of the MapControl. You can also add them to the MapItemsControl, or bind the MapItemsControl to an item or a collection of items.”

but it doesn’t say that I can’t add an InkCanvas in there Smile I made sure that I was handling the MapTapped event on the MapControl;

  <Grid
    Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">

    <Maps:MapControl
      x:Name="map" 
      MapTapped="OnMapTapped"/>
  </Grid>

and put some code behind that;

namespace MapApp
{
  using Windows.Devices.Input;
  using Windows.Foundation;
  using Windows.UI;
  using Windows.UI.Core;
  using Windows.UI.Input.Inking;
  using Windows.UI.Xaml.Controls;
  using Windows.UI.Xaml.Controls.Maps;

  public sealed partial class MainPage : Page
  {
    public MainPage()
    {
      this.InitializeComponent();
    }
    void OnMapTapped(MapControl sender, MapInputEventArgs args)
    {
      if (this.map.MapElements.Count < 2)
      {
        var mapIcon = new MapIcon()
        {
          Location = args.Location,
          NormalizedAnchorPoint = new Point(0.5, 0.5)
        };
        this.map.MapElements.Add(mapIcon);

        // When you add a 2nd point of interest, we put an InkCanvas over the entire map
        // waiting to see what you do next. If your next move is to use a pen then 
        // we'll wait for an ink stroke. If your next move is to use something
        // other than a pen then we'll get rid of the InkCanvas and get out of your
        // way.
        if (this.map.MapElements.Count == 2)
        {
          this.InsertInkCanvasOverlay();
        }
      }
    }
    void InsertInkCanvasOverlay()
    {
      this.inkCanvas = new InkCanvas();

      // Cover the map. Would need to handle resize as well really.
      this.inkCanvas.Width = this.map.ActualWidth;
      this.inkCanvas.Height = this.map.ActualHeight;

      // We don't want it to ink just yet, we want to wait to see what happens next in
      // terms of input mechansim.    
      this.inkCanvas.InkPresenter.InputProcessingConfiguration.Mode = InkInputProcessingMode.None;

      // Make sure we watch all the events that interest us.
      this.inkCanvas.InkPresenter.StrokeInput.StrokeEnded += OnStrokeEnded;
      this.inkCanvas.InkPresenter.UnprocessedInput.PointerEntered += OnInkCanvasPointerEntered;

      // NB: with the pen, we will 'lose' the pointer if it moves much distance from the
      // screen after tapping the 2nd point of interest. We could do a more detailed
      // check using the "IsInContact" property but I haven't done yet.
      this.inkCanvas.InkPresenter.UnprocessedInput.PointerExited += OnPointerExited;
      this.inkCanvas.InkPresenter.UnprocessedInput.PointerLost += OnPointerLost;

      // Format it
      var drawingAttr = this.inkCanvas.InkPresenter.CopyDefaultDrawingAttributes();
      drawingAttr.PenTip = PenTipShape.Rectangle;
      drawingAttr.Size = new Size(4, 4);
      drawingAttr.IgnorePressure = true;
      drawingAttr.Color = Colors.Orange;
      this.inkCanvas.InkPresenter.UpdateDefaultDrawingAttributes(drawingAttr);

      // put it into the map control.
      this.map.Children.Add(this.inkCanvas);
    }

    void OnStrokeEnded(InkStrokeInput sender, PointerEventArgs args)
    {
      // TODO: Now there's just the small matter of taking that ink stroke
      // and somehow turning it into a route.
    }

    void OnPointerLost(InkUnprocessedInput sender, PointerEventArgs args)
    {
      this.RemoveInkCanvas();
    }
    void OnPointerExited(InkUnprocessedInput sender, PointerEventArgs args)
    {
      this.RemoveInkCanvas();
    }
    void OnInkCanvasPointerEntered(InkUnprocessedInput sender, PointerEventArgs args)
    {
      if (args.CurrentPoint.PointerDevice.PointerDeviceType == PointerDeviceType.Pen)
      {
        this.inkCanvas.InkPresenter.InputProcessingConfiguration.Mode = InkInputProcessingMode.Inking;
      }
      else
      {
        this.RemoveInkCanvas();
      }
    }
    void RemoveInkCanvas()
    {
      if (this.inkCanvas != null)
      {
        this.map.Children.Remove(this.inkCanvas);
        this.inkCanvas.InkPresenter.StrokeInput.StrokeEnded -= this.OnStrokeEnded;
        this.inkCanvas.InkPresenter.UnprocessedInput.PointerEntered -= this.OnInkCanvasPointerEntered;
        this.inkCanvas.InkPresenter.UnprocessedInput.PointerExited -= this.OnPointerExited;
        this.inkCanvas.InkPresenter.UnprocessedInput.PointerLost -= this.OnPointerLost;
        this.inkCanvas = null;

        // for now, I also remove any points of interest on the map but it's a TBD
        // whether you'd really want to do this.
        this.map.MapElements.Clear();
      }
    }
    InkCanvas inkCanvas;
  }
}

So, essentially this leaves the map open to regular use but it uses the MapTapped event to add up to 2 points of interest onto the map (with no real way of clearing them yet). At the point where the 2nd POI gets added, an InkCanvas is inserted over the Map in “stealth” mode in that it isn’t inking or erasing so all input should be unprocessed.

Event handlers are added to the “UnprocessedInput” events such that if the next event comes from a Pen, we switch the mode into Inking and allow the ink to happen whereas if the next event comes from mouse or touch then we get rid of the InkCanvas.

That allows me to draw between two points as below;

image

which is ‘a start’ and leaves the next step which would be to figure out whether this can be turned into route directions.

From Ink Line to Route Directions

I’m not sure of what the best way is to change the collection of strokes that I’ve now captured into a driving/walking route so I went for the simple option and decided that I’d make some assumptions;

  • It’s ok to take just a single ink stroke
  • It’s safe to assume that a single ink stroke is not going to be made up of too many rendering segments

but these assumptions may not hold and so the code below might well need changing. It’s worth saying that it’s possible to get all the ink points within an ink stroke so an alternate (or better) solution might be to take some sampling of the ink points but I haven’t tried that to date.

Here’s the code with additional handlers for the StrokesCollected event of the InkPresenter.

namespace MapApp
{
  using System;
  using System.Collections.Generic;
  using System.Linq;
  using Windows.Devices.Geolocation;
  using Windows.Devices.Input;
  using Windows.Foundation;
  using Windows.Services.Maps;
  using Windows.UI;
  using Windows.UI.Core;
  using Windows.UI.Input.Inking;
  using Windows.UI.Xaml;
  using Windows.UI.Xaml.Controls;
  using Windows.UI.Xaml.Controls.Maps;

  public sealed partial class MainPage : Page
  {
    public MainPage()
    {
      this.InitializeComponent();

      this.tapPositions = new MapInputEventArgs[POI_COUNT];

      this.map.SizeChanged += OnMapSizeChanged;
    }
    void OnMapTapped(MapControl sender, MapInputEventArgs args)
    {
      var mapElementCount = this.map.MapElements.Count;

      if (mapElementCount < POI_COUNT)
      {
        this.tapPositions[mapElementCount] = args;

        var mapIcon = new MapIcon()
        {
          Location = args.Location,
          NormalizedAnchorPoint = new Point(0.5, 0.5)
        };
        this.map.MapElements.Add(mapIcon);

        // When you add a 2nd point of interest, we put an InkCanvas over the entire map
        // waiting to see what you do next. If your next move is to use a pen then 
        // we'll wait for an ink stroke. If your next move is to use something
        // other than a pen then we'll get rid of the InkCanvas and get out of your
        // way.
        mapElementCount = this.map.MapElements.Count;

        if (mapElementCount == POI_COUNT)
        {
          this.InsertInkCanvasOverlay();
        }
      }
    }
    void InsertInkCanvasOverlay()
    {
      this.inkCanvas = new InkCanvas();

      // Cover the map.
      this.inkCanvas.Width = this.map.ActualWidth;
      this.inkCanvas.Height = this.map.ActualHeight;

      // We don't want it to ink just yet, we want to wait to see what happens next in
      // terms of input mechansim.    
      this.inkCanvas.InkPresenter.InputProcessingConfiguration.Mode = InkInputProcessingMode.None;

      // Make sure we watch all the events that interest us.
      this.inkCanvas.InkPresenter.StrokesCollected += OnStrokesCollected;
      this.inkCanvas.InkPresenter.UnprocessedInput.PointerEntered += OnInkCanvasPointerEntered;

      // NB: with the pen, we will 'lose' the pointer if it moves much distance from the
      // screen after tapping the 2nd point of interest. We could do a more detailed
      // check using the "IsInContact" property but I haven't done yet.
      this.inkCanvas.InkPresenter.UnprocessedInput.PointerExited += OnPointerExited;
      this.inkCanvas.InkPresenter.UnprocessedInput.PointerLost += OnPointerLost;

      // Format it
      var drawingAttr = this.inkCanvas.InkPresenter.CopyDefaultDrawingAttributes();
      drawingAttr.PenTip = PenTipShape.Rectangle;
      drawingAttr.Size = new Size(4, 4);
      drawingAttr.IgnorePressure = true;
      drawingAttr.Color = Colors.Orange;
      this.inkCanvas.InkPresenter.UpdateDefaultDrawingAttributes(drawingAttr);

      // put it into the map control.
      this.map.Children.Add(this.inkCanvas);
    }
    async void OnStrokesCollected(InkPresenter sender, InkStrokesCollectedEventArgs args)
    {
      // We're going to take the first stroke, we're not going to try and
      // somehow stitch together many strokes :-S
      var firstStroke = args.Strokes.FirstOrDefault();

      if (firstStroke != null)
      {
        // How to split this down into some waypoints?
        var geoPoints = new List<Geopoint>();
        geoPoints.Add(this.tapPositions[0].Location);

        // TBD: Is using the rendering segments a good idea here or would
        // it make more sense to take some sampling of the ink points
        // along the way? Not sure.
        foreach (var segment in firstStroke.GetRenderingSegments())
        {
          Geopoint geoPoint = null;

          this.map.GetLocationFromOffset(segment.Position, out geoPoint);

          geoPoints.Add(geoPoint);
        }

        geoPoints.Add(this.tapPositions[1].Location);

        var routeResult = await MapRouteFinder.GetDrivingRouteFromWaypointsAsync(geoPoints);

        // We should do something about failures too?:-)
        if (routeResult.Status == MapRouteFinderStatus.Success)
        {
          var routeView = new MapRouteView(routeResult.Route);
          routeView.OutlineColor = Colors.Black;
          routeView.RouteColor = Colors.Orange;
          this.map.Routes.Add(routeView);
        }
        this.RemoveInkCanvas();
      }
    }
    void OnMapSizeChanged(object sender, SizeChangedEventArgs e)
    {
      if (this.inkCanvas != null)
      {
        this.inkCanvas.Width = e.NewSize.Width;
        this.inkCanvas.Height = e.NewSize.Height;
      }
    }
    void OnPointerLost(InkUnprocessedInput sender, PointerEventArgs args)
    {
      this.RemoveInkCanvas();
      this.RemovePOI();
    }
    void OnPointerExited(InkUnprocessedInput sender, PointerEventArgs args)
    {
      this.RemoveInkCanvas();
      this.RemovePOI();
    }
    void OnInkCanvasPointerEntered(InkUnprocessedInput sender, PointerEventArgs args)
    {
      if (args.CurrentPoint.PointerDevice.PointerDeviceType == PointerDeviceType.Pen)
      {
        this.inkCanvas.InkPresenter.InputProcessingConfiguration.Mode = InkInputProcessingMode.Inking;
      }
      else
      {
        this.RemoveInkCanvas();
      }
    }
    void RemoveInkCanvas()
    {
      if (this.inkCanvas != null)
      {
        this.map.Children.Remove(this.inkCanvas);
        this.inkCanvas.InkPresenter.StrokesCollected -= this.OnStrokesCollected;
        this.inkCanvas.InkPresenter.UnprocessedInput.PointerEntered -= this.OnInkCanvasPointerEntered;
        this.inkCanvas.InkPresenter.UnprocessedInput.PointerExited -= this.OnPointerExited;
        this.inkCanvas.InkPresenter.UnprocessedInput.PointerLost -= this.OnPointerLost;
        this.inkCanvas = null;
      }
    }
    void RemovePOI()
    {
      this.map.MapElements.Clear();

      for (int i = 0; i < POI_COUNT; i++)
      {
        this.tapPositions[i] = null;
      }
    }
    void OnClear(object sender, RoutedEventArgs e)
    {
      this.RemovePOI();
      this.RemoveInkCanvas();
      this.map.Routes.Clear();
    }
    static readonly int POI_COUNT = 2;
    MapInputEventArgs[] tapPositions;
    InkCanvas inkCanvas;
  }
}

and that produces a “prototype” experience that you can see in the screen capture below;

That’s as far as I’ve gone with this to date, feel free to feedback in the comments here and let me know of better directions that I could have taken.