I wanted to experiment with how interactions work in the Visual Layer and so on the November update of Windows 10 (10586.x) I made a new, blank UWP app with a MainPage that presented a green rectangle as below;
<Page
x:Class="App33.MainPage"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:local="using:App33"
xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
mc:Ignorable="d">
<Grid
Background="{ThemeResource ApplicationPageBackgroundThemeBrush}"
x:Name="gridParent">
<Rectangle
x:Name="rectangle"
Stroke="Black"
StrokeThickness="4"
Width="388"
Height="388"
Fill="Lime"
RenderTransformOrigin="0.5,0.5"
Tapped="XamlRectangleTapped">
<Rectangle.RenderTransform>
<CompositeTransform
x:Name="transform" />
</Rectangle.RenderTransform>
</Rectangle>
</Grid>
</Page>
and I’ve tried to set this up so that I can use pinch/rotate gestures to manipulate the rectangle;
using Windows.UI.Xaml.Controls;
using Windows.UI.Xaml.Input;
namespace App33
{
public sealed partial class MainPage : Page
{
public MainPage()
{
this.InitializeComponent();
this.Loaded += OnLoaded;
}
void OnLoaded(object sender, Windows.UI.Xaml.RoutedEventArgs e)
{
this.rectangle.ManipulationMode =
ManipulationModes.Rotate | ManipulationModes.Scale;
this.rectangle.ManipulationDelta += OnManipulationDelta;
}
void OnManipulationDelta(object sender, ManipulationDeltaRoutedEventArgs e)
{
this.transform.Rotation += e.Delta.Rotation;
this.transform.ScaleX *= e.Delta.Scale;
this.transform.ScaleY *= e.Delta.Scale;
}
void XamlRectangleTapped(object sender, TappedRoutedEventArgs e)
{
}
}
}
and that works nicely enough;
and I added a little code to add another, orange rectangle painted by the Visual or Composition layer and created when the first tap happens on the green rectangle. At that point, the class file looks like;
namespace App33
{
using System.Numerics;
using Windows.UI;
using Windows.UI.Composition;
using Windows.UI.Xaml.Controls;
using Windows.UI.Xaml.Hosting;
using Windows.UI.Xaml.Input;
public sealed partial class MainPage : Page
{
public MainPage()
{
this.InitializeComponent();
this.Loaded += OnLoaded;
}
void OnLoaded(object sender, Windows.UI.Xaml.RoutedEventArgs e)
{
this.rectangle.ManipulationMode =
ManipulationModes.Rotate | ManipulationModes.Scale;
this.rectangle.ManipulationDelta += OnManipulationDelta;
this.compositor = ElementCompositionPreview.GetElementVisual(this.rectangle).Compositor;
}
void OnManipulationDelta(object sender, ManipulationDeltaRoutedEventArgs e)
{
this.transform.Rotation += e.Delta.Rotation;
this.transform.ScaleX *= e.Delta.Scale;
this.transform.ScaleY *= e.Delta.Scale;
}
void XamlRectangleTapped(object sender, TappedRoutedEventArgs e)
{
if (this.rectangleVisual == null)
{
this.rectangleVisual = this.compositor.CreateSpriteVisual();
this.rectangleVisual.Brush = this.compositor.CreateColorBrush(Colors.Orange);
this.rectangleVisual.Size = new Vector2(
(float)this.gridParent.ActualWidth / 2.0f,
(float)this.gridParent.ActualHeight / 2.0f);
this.rectangleVisual.Offset = new Vector3(
((float)this.gridParent.ActualWidth / 2.0f) - this.rectangleVisual.Size.X / 2.0f,
((float)this.gridParent.ActualHeight / 2.0f) - this.rectangleVisual.Size.Y / 2.0f,
0.0f);
ElementCompositionPreview.SetElementChildVisual(this.gridParent, this.rectangleVisual);
}
}
SpriteVisual rectangleVisual;
Compositor compositor;
}
}
and then I didn’t really know what to expect.
The orange ‘visual layer’ rectangle completely obscures the green ‘XAML’ and so I had a split expectation around whether the green rectangle would or would not still receive the touch events that let it be rotated and scaled.
That is – does the visual layer intercept those events? Here’s the answer;
So, from the point of view of the touch events travelling towards the green rectangle, the orange rectangle painted by the Visual layer seems to be invisible and the events pass through it.
That’s not entirely surprising and if the events had been impacted by the Visual layer then I’d have wondered how that might be possible but I wouldn’t have necessarily bet money on it prior to having tried it out 
With that trial out of the way, how would I actually process input on that orange rectangle? There aren’t any events visible on a Visual or its hierarchy;
and so maybe it’s time for a video that talks about interactions in the Visual Layer;
What did that tell me?
That some work was done in the November update such that some of the hidden inner-workings of a XAML ScrollViewer have been surfaced via a ManipulationPropertySet.
PropertySets are explained in that same article and I came across them in this post and they’re interesting in the way in which they can be used in expression based animations.
I’d like to come back and experiment with that ScrollViewer functionality but, for the moment, I went past it in the video and learned that in the Anniversary Update there’s a new “interaction tracker” component that it seems runs the algorithms to work out what type of interactions are going on and, following the diagram from the video, it looks like it can take input from different sources with one being a “Visual Interaction Source”;
Based on that initial description, in my mind I’m thinking of this a little bit like the XAML-based manipulation engine which I made use of to rotate and scale my green rectangle without ever having to write the explicit code to handle the pointer events and so on but it remains to be seen as to whether that’s a good or bad analogy.
Having watched the video, I felt that I mostly understood it from the point of view of a set of concepts but I didn’t feel that I’d actually be able to code it myself and so I thought that I should give that a whirl to get a feel for it and so I switched to my development machine running the Anniversary Update and tried to make a ‘Hello World’ on the latest preview SDK.
I made a completely blank UWP project (i.e. no XAML in there) with the single code file;
namespace App10
{
using System.Numerics;
using Windows.ApplicationModel.Core;
using Windows.Devices.Input;
using Windows.UI;
using Windows.UI.Composition;
using Windows.UI.Composition.Interactions;
using Windows.UI.Core;
class MainView : IFrameworkView, IFrameworkViewSource
{
// The first 2 functions in this source file are the 2 that really do anything
// 'new' from the point of view of this blog post. It's where the
// VisualInteractionSource and InteractionTracker are set up and used.
public void SetWindow(CoreWindow window)
{
this.window = window;
this.CreateCompositorAndRootContainer();
this.CreateSpriteVisualForRectangle();
this.CreateInteractionTracker();
this.window.PointerPressed += (s, e) =>
{
if (e.CurrentPoint.PointerDevice.PointerDeviceType == PointerDeviceType.Touch)
{
// TBD: I can get an UnauthorizedAccessException here if I pass
// a point that's beyond the bounds of my rectangle and I haven't
// figured out why yet.
this.interactionSource.TryRedirectForManipulation(e.CurrentPoint);
}
};
}
void CreateInteractionTracker()
{
this.tracker = InteractionTracker.Create(this.compositor);
// TBD: I'm unsure as to whether I want my source here to be the visual itself
// or the container it lives in but I've gone with the visual as I guess I
// really want interactions on that rather than the whole window.
this.interactionSource = VisualInteractionSource.Create(this.spriteVisual);
this.interactionSource.ScaleSourceMode = InteractionSourceMode.EnabledWithoutInertia;
this.tracker.InteractionSources.Add(this.interactionSource);
this.tracker.MinScale = 1.0f;
this.tracker.MaxScale = 2.0f;
var scaleAnimation = this.compositor.CreateExpressionAnimation("Vector3(tracker.Scale, tracker.Scale, 1.0f)");
scaleAnimation.SetReferenceParameter("tracker", this.tracker);
this.spriteVisual.StartAnimation("Scale", scaleAnimation);
}
// From hereon in, everything in this code file has been talked about in previous
// blog posts so becomes 'boilerplate' to some extent.
public static void Main()
{
CoreApplication.Run(new MainView());
}
public MainView()
{
}
public IFrameworkView CreateView()
{
return (this);
}
public void Run()
{
this.window.Activate();
this.window.Dispatcher.ProcessEvents(CoreProcessEventsOption.ProcessUntilQuit);
}
void CreateSpriteVisualForRectangle()
{
this.spriteVisual = this.compositor.CreateSpriteVisual();
this.spriteVisual.Size = new Vector2(
(float)this.window.Bounds.Width / 4.0f,
(float)this.window.Bounds.Height / 4.0f);
this.spriteVisual.Offset = new Vector3(
((float)this.window.Bounds.Width / 2.0f) - ((float)this.spriteVisual.Size.X / 2.0f),
((float)this.window.Bounds.Height / 2.0f) - ((float)this.spriteVisual.Size.Y / 2.0f),
0.0f);
this.spriteVisual.CenterPoint = new Vector3(
this.spriteVisual.Size.X / 2.0f,
this.spriteVisual.Size.Y / 2.0f,
0.0f);
this.spriteVisual.Brush = this.compositor.CreateColorBrush(Colors.Orange);
this.containerVisual.Children.InsertAtTop(this.spriteVisual);
}
void CreateCompositorAndRootContainer()
{
this.compositor = new Compositor();
this.containerVisual = this.compositor.CreateContainerVisual();
var target = this.compositor.CreateTargetForCurrentView();
target.Root = this.containerVisual;
}
public void Initialize(CoreApplicationView applicationView)
{
}
public void Uninitialize()
{
}
public void Load(string entryPoint)
{
}
VisualInteractionSource interactionSource;
InteractionTracker tracker;
SpriteVisual spriteVisual;
CoreWindow window;
ContainerVisual containerVisual;
Compositor compositor;
}
}
and it’s only the first 2 functions of this class (as listed above) that do anything that I haven’t mentioned in previous blog posts on this topic. It’s those first two functions that;
- Pass on pointer events from the window to the VisualInteractionSource.
- Set up the InteractionTracker in the first place along with the VisualInteractionSource which feeds it the events.
In setting up that InteractionTracker in the method CreateInteractionTracker, I try to limit it to scale between 1.0 and 2.0 and it’s worth saying that the UI allows me to go beyond those limits and then ‘snaps back’ to them when I complete a pinch gesture.
The other thing that happens within that method is the somewhat magical tie-up between the InteractionTracker and the visual which is done by the expression animation. That is – neither the Visual nor the animation need to know anything about the InteractionTracker because it’s just producing a bunch of properties (in a PropertySet) and the expression animation already knows how to deal with expressions that come from PropertySets.
I like that
The code works, I can scale the rectangle as intended. One thing I’d flag is that I expected to be able to do rotation here and either it’s not in the bits that I have here or I’m missing how to do it as I can only see how to pick up changes to the scale and to the position. I don’t know about other gestures like rotate, flick, etc.
I’d like to try and do something a bit more “realistic” with this capability. Naturally, there are the demos in the video referenced above which look really nice but I’d like to create something simpler of my own to try this out on.
That’ll be a follow on post as will some sort of experiment around what it’s like to have interactions at both the XAML layer and the Visual layer.
Mike Taulty 
Great text. How about InteractionTracker’s performance? Now in my app I’m using Xaml to catch ManipulationDelta event with Composition to change Visual’s position and sometimes it can be laggy.
Hi,
As you know, I’m only just coming to this InteractionTracker so I can’t speak definitively in any way 🙂
That said, I think the idea of the Visual Layer is to offer great performance and low latency and the idea of the InteractionTracker seems to be to support a scenario where the user’s interactions can be linked with Visual Layer manipulations *without* having to involve (switching to) the UI thread at all.
So, I’d expect that to be more performant than involving the ManipulationXXX events which involves switching over to that UI thread.
Mike.
Thank you for the reply.:)
Pingback: Windows 10, UWP and Composition– Experimenting with Interactions in the Visual Layer | Tech News
Thank you for the reply. 🙂
Pingback: Windows 10 Anniversary Update Preview, Visual Layer–Mocking Up the Lock Screen – Mike Taulty