Windows 10 Anniversary Update Preview–Composition and the CompositionBackdropBrush

Posted on June 6, 2016 by Mike Taulty

One of the Windows 10/UWP areas that I’m very interested in watching progress are the Composition APIs (the Visual Layer) that first appeared in the 1511 build but are continuing to grow in the upcoming Anniversary Edition.

I have watched a few of the sessions (including breakouts and pre-recorded) from //Build 2016 around composition and I was going to make a list of them here but then I realised that the composition team had already done this Smile

They have their github page;

https://github.com/Microsoft/WindowsUIDevLabs

and within there is the ‘newsletter’ that the guys are publishing with their updates which is something I found really valuable;

https://github.com/Microsoft/WindowsUIDevLabs/wiki/Windows-UI-Newsletter—May-Edition

and within that there is the list of all the sessions from //Build 2016 relating to the broader topic of Windows UI with around 8-9 relating directly to the visual layer topics;

https://github.com/Microsoft/WindowsUIDevLabs/wiki/Windows-UI-Newsletter—May-Edition#build-sessions

At the time of writing, I have watched some of these but not all of these but I will get there Smile

I also spotted that some of my earlier posts are referenced from the team’s github so I’d better try and keep up with what’s happening with these APIs!

Revisiting the Previous Visual Layer Post

The last post that I wrote in my earlier experiments with the APIs here was around how easy/difficult it might be to walk up to some arbitrary XAML element and add an effect to it like blur/saturation etc. to it.

At the time of that post, on the 10586 SDK I found that it wasn’t really so easy because it involved me as a developer having to render the XAML content to a bitmap before then rendering that bitmap back to the screen with the composition layer and that might be ok if the XAML in question was relatively static but it’s a bit of a challenge if the bitmap has to be constantly updated where the XAML content is dynamically changing.

I did it by using some kind of timer but that was a long way from ideal.

Moving on to the Current Preview SDK – New Brushes!

With the current preview of the 14332 SDK, I wanted to check whether this scenario had moved on and it looks like it has with the arrival of the CompositionBackdropBrush.

At the time of writing, I can’t find a proper documentation link around that class but I found a decent discussion around it on StackOverflow which pointed back to the sample code and so on and it intrigued me enough to want to try it out for myself and that’s what this post is about.

To recap, the Visual Layer deals with Visuals and there’s the derived SpriteVisual which is a Visual that is painted with a CompositionBrush and in the 10586 SDK the brushes are CompositionColorBrush, CompositionEffectBrush and CompositionSurfaceBrush.

In the 14332 SDK, the number of available brushes seems to double to include CompositionBackdropBrush, CompositionMaskBrush, CompositionNineGridBrush.

For this post, it’s the CompositionBackdropBrush that interests me and I found it to be a curious thing Smile

The CompositionBackdropBrush

Let’s say that I have this simple UI with a ViewBox and a TextBlock;

<Page
    x:Class="App3.MainPage"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:local="using:App3"
    xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
    xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
    mc:Ignorable="d">

  <Grid
    Background="{ThemeResource ApplicationPageBackgroundThemeBrush}"
    x:Name="myGrid">
    <Viewbox>
      <TextBlock
        x:Name="txtCount" />
    </Viewbox>
 </Grid>
</Page>

and some code-behind which updates on a timer;

  public sealed partial class MainPage : Page
  {
    public MainPage()
    {
      this.InitializeComponent();
      this.Loaded += OnLoaded;
    }

    void OnLoaded(object sender, RoutedEventArgs args)
    {
      int value = 0;
      this.timer = new DispatcherTimer();
      this.timer.Interval = TimeSpan.FromSeconds(1);
      this.timer.Tick += (s, e) =>
      {
        // not a great idea to ++ this variable in a format string here.
        this.txtCount.Text = $"{++value}";
      };
      this.timer.Start();
    }
    DispatcherTimer timer;
  }

then perhaps I want to take this whole ‘chunk’ of UI and use it as some kind of brush as input to an effect or maybe just to mirror the UI somewhere else on the screen as I can in WPF with a VisualBrush.

How I might do that wasn’t immediately obvious to me using CompositionBackgroundBrush because I could only find one way to create one.

Specifically;

      var gridVisual = ElementCompositionPreview.GetElementChildVisual(this.myGrid);

      var compositor = gridVisual.Compositor;

      var brush = compositor.CreateBackdropBrush();

and I found this one a bit puzzling in terms of that call to CreateBackdropBrush – i.e. which backdrop am I getting here because I’m making this call on the compositor itself and I don’t seem to tell the compositor about the element that I would like to get the backdrop from.

My expectation would be something like;

Get some visual for a UI element
Create a brush from that visual
Use that brush somewhere

but I don’t think that’s quite how things work here. I think the sequence is something more like;

Create a visual
Create a backdrop brush
Apply the brush to the visual
Position the visual ‘over’ the content which will become the brush

Applying an Effect

I thought I’d see how that worked out. I moved my existing grid into a row of a sub-grid and added a button to my UI;

<Page
    x:Class="App3.MainPage"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:local="using:App3"
    xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
    xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
    mc:Ignorable="d">

  <Grid
    Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">
    <Grid.RowDefinitions>
      <RowDefinition />
      <RowDefinition
        Height="Auto" />
    </Grid.RowDefinitions>
    <Grid x:Name="myGrid">
      <Viewbox>
        <TextBlock
          x:Name="txtCount" />
      </Viewbox>
    </Grid>
    <Button
      Grid.Row="1"
      Content="Blur"
      HorizontalAlignment="Center"
      Click="OnEffectButton" />
  </Grid>
</Page>

and then added some code behind the button;

namespace App3
{
  using Microsoft.Graphics.Canvas.Effects;
  using System;
  using System.Numerics;
  using Windows.UI.Composition;
  using Windows.UI.Xaml;
  using Windows.UI.Xaml.Controls;
  using Windows.UI.Xaml.Hosting;

  public sealed partial class MainPage : Page
  {
    public MainPage()
    {
      this.InitializeComponent();
      this.Loaded += OnLoaded;
    }

    void OnLoaded(object sender, RoutedEventArgs args)
    {
      int value = 0;
      this.timer = new DispatcherTimer();
      this.timer.Interval = TimeSpan.FromSeconds(1);
      this.timer.Tick += (s, e) =>
      {
        // not a great idea to ++ this variable in a format string here.
        this.txtCount.Text = $"{++value}";
      };
      this.timer.Start();
    }
    DispatcherTimer timer;

    void OnEffectButton(object sender, RoutedEventArgs e)
    {
      if (this.effectVisual != null)
      {
        this.effectVisual.Dispose();
        this.effectVisual = null;
      }
      else
      {
        var gridVisual = ElementCompositionPreview.GetElementVisual(this.myGrid);

        var compositor = gridVisual.Compositor;

        this.effectVisual = compositor.CreateSpriteVisual();

        // We'd need to resize this as/when the grid resized.
        this.effectVisual.Size = new Vector2(
          (float)this.myGrid.ActualWidth, 
          (float)this.myGrid.ActualHeight);

        GaussianBlurEffect blurEffect = new GaussianBlurEffect()
        {
          BorderMode = EffectBorderMode.Hard, // NB: default mode here isn't supported yet.
          Source = new CompositionEffectSourceParameter("source")
        };

        var effectFactory = compositor.CreateEffectFactory(blurEffect);
        var effectBrush = effectFactory.CreateBrush();
        effectBrush.SetSourceParameter("source", compositor.CreateBackdropBrush());

        this.effectVisual.Brush = effectBrush;

        ElementCompositionPreview.SetElementChildVisual(this.myGrid, this.effectVisual);
      }
    }
    SpriteVisual effectVisual;
  }
}

and now I’ve got a blurred counter updating in real time;

I then wondered whether that blur might be something I could drag around with my finger (or mouse/pen) and so I changed my UI;

<Page
    x:Class="App3.MainPage"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:local="using:App3"
    xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
    xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
    mc:Ignorable="d">

  <Grid
    Background="{ThemeResource ApplicationPageBackgroundThemeBrush}">
    <Grid.RowDefinitions>
      <RowDefinition />
    </Grid.RowDefinitions>
    <Grid
      x:Name="myGrid"
      PointerPressed="OnPointerPressed"
      PointerMoved="OnPointerMoved"
      PointerReleased="OnPointerReleased">
      <Viewbox>
        <TextBlock
          x:Name="txtCount" />
      </Viewbox>
    </Grid>
  </Grid>
</Page>

and the code behind it;

namespace App3
{
  using Microsoft.Graphics.Canvas.Effects;
  using System;
  using System.Numerics;
  using Windows.UI;
  using Windows.UI.Composition;
  using Windows.UI.Input;
  using Windows.UI.Xaml;
  using Windows.UI.Xaml.Controls;
  using Windows.UI.Xaml.Hosting;
  using Windows.UI.Xaml.Input;
  public sealed partial class MainPage : Page
  {
    public MainPage()
    {
      this.InitializeComponent();
      this.Loaded += OnLoaded;
      this.effectSize = new Vector2(200, 200);
    }

    void OnLoaded(object sender, RoutedEventArgs args)
    {
      int value = 0;
      this.timer = new DispatcherTimer();
      this.timer.Interval = TimeSpan.FromSeconds(1);
      this.timer.Tick += (s, e) =>
      {
        // not a great idea to ++ this variable in a format string here.
        this.txtCount.Text = $"{++value}";
      };
      this.timer.Start();
    }
    void AddEffect()
    {
      var gridVisual = ElementCompositionPreview.GetElementVisual(this.myGrid);

      var compositor = gridVisual.Compositor;

      this.effectVisual = compositor.CreateSpriteVisual();

      // We'd need to resize this as/when the grid resized.
      this.effectVisual.Size = this.effectSize;

      GaussianBlurEffect blurEffect = new GaussianBlurEffect()
      {
        BorderMode = EffectBorderMode.Hard, // NB: default mode here isn't supported yet.
        BlurAmount = 5.0f,
        Source = new CompositionEffectSourceParameter("source")
      };

      var effectFactory = compositor.CreateEffectFactory(blurEffect);
      var effectBrush = effectFactory.CreateBrush();
      effectBrush.SetSourceParameter("source", compositor.CreateBackdropBrush());

      this.effectVisual.Brush = effectBrush;

      ElementCompositionPreview.SetElementChildVisual(this.myGrid, this.effectVisual);
    }
    void OnPointerPressed(object sender, PointerRoutedEventArgs e)
    {
      this.AddEffect();
      this.PositionEffect(e.GetCurrentPoint(this.myGrid));
    }
    void PositionEffect(PointerPoint point)
    {
      var halfWidth = this.effectSize.X / 2;
      var halfHeight = this.effectSize.Y / 2;
      var idealPositionX = (float)point.Position.X - halfWidth;
      var idealPositionY = (float)point.Position.Y - halfHeight;
      idealPositionX = Math.Max(0, idealPositionX);
      idealPositionX = Math.Min(idealPositionX, (float)this.myGrid.ActualWidth - this.effectSize.X);
      idealPositionY = Math.Max(0, idealPositionY);
      idealPositionY = Math.Min(idealPositionY, (float)this.myGrid.ActualHeight - this.effectSize.Y);
      this.effectVisual.Offset = new Vector3(
        idealPositionX, idealPositionY, 0);
    }
    void OnPointerMoved(object sender, PointerRoutedEventArgs e)
    {
      if (this.effectVisual != null)
      {
        this.PositionEffect(e.GetCurrentPoint(this.myGrid));
      }
    }
    void OnPointerReleased(object sender, PointerRoutedEventArgs e)
    {
      this.effectVisual?.Dispose();
      this.effectVisual = null;
    }
    SpriteVisual effectVisual;
    Vector2 effectSize;
    DispatcherTimer timer;
  }
}

and, sure enough, I had a 200×200 pixel square of ‘blur effect’ that I could drag around my UI.

It would perhaps make sense if I had richer content (like an image) at this point but hopefully the 0 in the screenshot below looks suitably blurred at the top left corner.

As an aside, I noticed that my PointerMoved events didn’t fire when I was using this via touch (they did for mouse) and I’m guessing that’s because they were being intercepted by the visual layer in a way that I haven’t quite understood just yet so I need to return to that – I think this might relate to interactions at the visual layer.

A Non-Rectangular Effect

With this kind of working, I wondered whether I could make that effect brush fill something other than a rectangle – e.g. can I make it circular or do visuals always have to be rectangular?

I had a few attempts at this.

Initially, I thought I might try some XAML trick with a Path similar but I fairly quickly ground to a halt when I realised I ultimately needed to be able to clip the visual itself.
I then tried to look at the Clip property of a CompositionBrush but that seemed to limited to a rectangle.
I then started to look at the CompositionMaskBrush which seemed like the right thing to be using but I couldn’t find too many uses of it out there other than a sample here which seemed to be using a transparent PNG as the source which tells a mask how to clip.

I tried to follow that CompositionMaskBrush route based on what I saw in the sample but the sample was more about adding a shadow to an image. When I tried to bend this to suit my purposes, it seemed that the CompositionMaskBrush did not support taking its input from either a CompositionEffectBrush or a CompositionBackdropBrush.

I got a bit blocked at that point until…

A Non-Rectangular Visual

Being a bit stuck, I was about to ask the @windowsui Twitter account for help when I saw that they had retweeted Ratish’s tweet about;

“Custom Shaped Visuals with CanvasGeometry”

with links to his github and that sounded like exactly what I was trying to figure out – Visuals that aren’t square.

I had a quick poke into the code and it looks like the ‘trick’ that I was missing was to the one of rendering a geometry of some sort to a surface so that the composition APIs could use it as a mask.

Ratish’s code has that trick and so I thought that I’d give it a try and I did that for around 10 minutes before realising that I was heading back to the same place as my previous attempts – i.e. I was still going to get stuck by CompositionMaskBrush not allowing me to pass a Source brush that came from a CompositionBackdropBrush.

So…at the time of writing, I’m not sure whether I can/can’t apply an effect in a non-rectangular way here but I’m still pleased with how easy it becomes to add an effect to a ‘live’ XAML UI.

Finishing Up

I took my UI back to something really simple – just an image from the web;

<Page
    x:Class="App3.MainPage"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:local="using:App3"
    xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
    xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
    mc:Ignorable="d">

  <Grid
    Background="{ThemeResource ApplicationPageBackgroundThemeBrush}"
    x:Name="myGrid"
    SizeChanged="OnGridSizeChanged">
    <Image
      Source="http://www.mrwallpaper.com/wallpapers/mauritius-blue-bay-beach.jpg"
      Stretch="Uniform" />
  </Grid>
</Page>

and I changed my code to animate the blur over that UI (noting that the UI could be something more complex than just a static image);

namespace App3
{
  using Microsoft.Graphics.Canvas.Effects;
  using System;
  using System.Numerics;
  using Windows.UI.Composition;
  using Windows.UI.Xaml;
  using Windows.UI.Xaml.Controls;
  using Windows.UI.Xaml.Hosting;
  public sealed partial class MainPage : Page
  {
    public MainPage()
    {
      this.InitializeComponent();
      this.Loaded += OnLoaded;
    }
    void SizeVisual()
    {
      if (this.effectVisual != null)
      {
        this.effectVisual.Size = new Vector2(
          (float)this.myGrid.ActualWidth, (float)this.myGrid.ActualHeight);
      }
    }
    void OnLoaded(object sender, RoutedEventArgs args)
    {
      var gridVisual = ElementCompositionPreview.GetElementVisual(this.myGrid);

      var compositor = gridVisual.Compositor;

      this.effectVisual = compositor.CreateSpriteVisual();

      this.SizeVisual();

      GaussianBlurEffect blurEffect = new GaussianBlurEffect()
      {
        Name = "Blur",
        BorderMode = EffectBorderMode.Hard, // NB: default mode here isn't supported yet.
        BlurAmount = 0.0f,
        Source = new CompositionEffectSourceParameter("source")
      };

      var effectFactory = compositor.CreateEffectFactory(
        blurEffect,
        new string[] { "Blur.BlurAmount" });

      var effectBrush = effectFactory.CreateBrush();

      var blurAnimation = compositor.CreateScalarKeyFrameAnimation();
      var easingFunction = compositor.CreateLinearEasingFunction();

      blurAnimation.InsertKeyFrame(0.0f, 0.0f);
      blurAnimation.InsertKeyFrame(0.5f, 10.0f, easingFunction);
      blurAnimation.InsertKeyFrame(1.0f, 0.0f, easingFunction);
      blurAnimation.IterationBehavior = AnimationIterationBehavior.Forever;
      blurAnimation.Duration = TimeSpan.FromSeconds(5);
      effectBrush.StartAnimation("Blur.BlurAmount", blurAnimation);

      effectBrush.SetSourceParameter("source", compositor.CreateBackdropBrush());

      this.effectVisual.Brush = effectBrush;

      ElementCompositionPreview.SetElementChildVisual(this.myGrid, this.effectVisual);
    }
    void OnGridSizeChanged(object sender, SizeChangedEventArgs e)
    {
      this.SizeVisual();
    }
    SpriteVisual effectVisual;
  }
}

and I’m back to a nicely animated blur over whatever UI is resident in the Grid named ‘myGrid’.

What I’ve yet to figure out though is whether I can take one of these brushes and use it to paint some other ‘area’ of the screen – I’ll return to that in a later post.

Windows 10 Devices and ‘The Developer Portal’

Posted on February 3, 2016 by Mike Taulty

I haven’t found official documentation on this anywhere so I thought I’d write up my own quick notes which I can replace when I find the right web pages to link to – I’ve seen quite a few people talk about the portal that’s present on both IoT Core and on Windows Mobile and I’m assuming on Surface Hub but I haven’t checked that last one at the time of writing.

By way of rough notes…

I’m currently on a mixture of wired/WiFi networking where I have my Surface Pro 3 development machine on a wired network along with a Raspberry PI 2 running Windows IoT Core and a Windows Mobile 10 device (my phone) which connects to the same network but over WiFi.

If I use the IoT Dashboard application then it shows me a couple of devices on my network;

and so it’s showing both my PI and my phone.

If I use the “Open Device Portal” on the Raspberry PI 2 then a browser opens;

with the portal for the IoT core device and I can use this to manage aspects of the device as per the list on the left hand side of the screen – specifically, I can use this to add/remove/stop/start apps and I can use it to set up networking and bluetooth devices.

If you’ve seen me talk about IoT Core in public then you’ll have seen me talk my way through this portal and show a few things.

What I’ve never shown in public (AFAIK) though is what happens if I use the “Open Device Portal” on my phone. In the first instance, I see an error;

but I can fix that by losing the :8080 port on the address here although (right now) I still find myself with a warning;

but if I go around that I get to;

and this leads over to the pages on the phone where I can set all of this up. As you can see, I have the device in developer mode;

and you’ll notice that I have “Device discovery” switched on here;

and that it says that this is available over both USB and WiFi, that I have no devices paired and that the portal is also switched on and advertising itself at http://192.168.0.11.

If I then go through that ‘Pair’ process on the phone, I get a code;

which I can then type into the web page and I’m then onto the portal on the device and here’s the phone portal side by side with the IoT Core portal;

and from that Phone portal I can then go and add/remove/stop/start apps and I can look at performance, networking and so on for the phone just like I can on IoT Core device.

What I’d really like to do is to also be able to debug here over WiFi without having to connect a USB cable and I don’t know at the time of writing whether I can do that or not.

If I unplug the USB cable then Visual Studio certainly sees the device as a remote target over the network if I use the Universal authentication mode;

but attempting to deploy using this mechanism ends up in me being prompted for a PIN;

and I don’t seem to be able to provide any type of PIN here that makes the phone play nicely with me so I’m not sure whether this is just because I don’t know how to provide the right PIN or whether there isn’t a PIN that will work here right now.

Let me know if you know and I’ll update the post.

Windows 10 UWP–Migrating a Windows 8.1 App (Part 12 of 12)

Posted on January 11, 2016 by Mike Taulty

Following up on that last post, I wanted to finish up this series posts on moving a Windows 8.1 app to Windows 10 UWP.

Since the application is now showing up on Windows Mobile devices, I needed to do a little more work on the UI and, even with this small app, I find it to be quite a challenge to size things such that they look reasonable across a variety of devices. No-one should be telling anyone that this is ‘easy’ or ‘free’ but it’s certainly within the realm of the ‘possible’ Smile

For my own work here, I’m largely testing on a Surface Pro 3 with/without external monitors that are lower DPI than the tablet itself and then also on a Dell Venue Pro 8 and on a Lumia 920 running Windows 10 Mobile Preview.

Where I ended up with the UI was that I maintained a narrow/wide set of visual states rather than thinking in terms of the different mobile/desktop device families.

So, on the desktop, the app in its wide mode has a look;

and in its narrow mode it drops down to;

and that seems to work reasonably well across desktop mode, tablet mode and then down onto a phone device (screen-shotted here via the ‘project my phone’ app);

Beyond that, I had a few more areas that were buggy and that I needed to tidy up;

My camera preview handling had problems with front facing cameras and flipping/mirroring the video. I found the Camera Starter Kit sample to be really useful here.
I hit a problem with calling the GetNativeSystemInfo API on Mobile devices. I fixed that in this post.
I hit a bug in sharing QR code images into the app that only showed up on Windows Mobile but had been there since day 1 and so that was nice to find. I also reworked the share target UI as part of fixing that as I was a bit horrified by the XAML I’d made there.

and I’m also finding that my ‘rotary control’ for controlling the focus of the camera is a bit problematic on Windows Mobile right now;

in that its performance is pretty horrible and I haven’t yet come up with a solution to making that better but I’m going to revisit it.

Having got the app into a ‘reasonable’ state, I revisited the manifest because I knew that there would be a bunch of icon work to do and, sure enough, I spent a good hour or two changing images that I had previously drawn at scale factors like 140, 180 into images drawn at scale factors 125, 150, 200, 400 etc.

I’m not sure that everyone knows how much effort it is to complete a full application manifest but I find producing sheet after sheet of these sorts of images;

from their original vectors to be quite time consuming.

With that done, it was time to go and visit the Windows Store and set about an app submission. Here, the work goes into the highlighted sections;

and, specifically, in the Descriptions section I chose the option which let me create a ‘platform specific description’, I based it off my Windows 8.1 description and so I end up with 3 description areas here in the Store;

and then I went to work producing screenshots for the Windows 10 version of the app. Once again, if you filled in everything here then you’re looking at I think 26 separate images so it takes quite a bit of time (and imagination) to populate;

At the time of writing, I’ve still got some of that work to do but the app is heading back into Store (ready to get some more brutal feedback from its users Winking smile ) and I’ll then be back into bug-fix mode on it.

As I said at the start of this process, this was a simple app and it’s taken me 11 stages of blog posts to get from Windows 8.1 to Windows 10.0 albeit working on it only from time to time in some stolen moments but I’m glad that I worked through it as it gave me quite a bit of insight into what the process looks like and I hope there were a few things contained in these posts that helped others along the same journey.

Mike Taulty

I do some developer stuff on Microsoft technologies in the UK

Category Archives: WindowsPhone