In response to the previous post about the Visual Layer in Windows.UI.Composition a Tweet flooded in 
In my previous post, I’d used the composition APIs to draw some spinning rectangles whereas this question was about filling those rectangles with images and so I thought I’d bend the code that I’d written to render some images.
It’s not quite as easy as it sounds – the Visual Layer talks in terms of visuals being painted with brushes and so it initially seems natural to expect that there would be some kind of ‘image brush’ to do this for you.
However, as I found in my initial post on this topic, I don’t think that’s the case and it’s perhaps right that it isn’t the case. The composition APIs are perhaps more to do with composing things that have already been drawn than they are to do with drawing things in the first place.
So, how does a rasterizer that can draw content plug in to the composition APIs and paint a visual with what’s been drawn?
I think that the way this works (or at least one way) is to get hold of a CompositionSurfaceBrush which renders to an ICompositionSurface but the docs under those 2 hyperlinks very clearly say;
Note, ICompositionSurface is exposed in Native code only, hence LoadImage method is implemented in native code.
and the MSDN docs refer to a “LoadImage” function to bring in images. However, that “LoadImage” function is one that you won’t find anywhere in the APIs – it’s implemented in the C++ code that you’ll find in the composition samples on GitHub in this Toolkit code and the samples all seem to use this mechanism of painting visuals with images.
I didn’t necessarily want to include that bunch of C++ code every time that I wanted to draw an image and so I thought that I would use a different means to get a CompositionSurfaceBrush from an image and it’s very similar to some code that I wrote in my first post around Windows.UI.Composition.
I brought in Win2D via the Win2D.uwp package because there’s some support in there for rendering content with Win2D by being able to create a CompositionGraphicsDevice and then using that to create a drawing surface.
My early steps then in trying to use that to render an image (from a StorageFile) look like this;
using Microsoft.Graphics.Canvas;
using Microsoft.Graphics.Canvas.UI.Composition;
using System;
using System.Threading.Tasks;
using Windows.Foundation;
using Windows.Graphics.DirectX;
using Windows.Graphics.Imaging;
using Windows.Storage.Streams;
using Windows.UI.Composition;
class CompositionImageBrush : IDisposable
{
private CompositionImageBrush()
{
}
public CompositionBrush Brush
{
get
{
return (this.drawingBrush);
}
}
void CreateDevice(Compositor compositor)
{
this.graphicsDevice = CanvasComposition.CreateCompositionGraphicsDevice(
compositor, CanvasDevice.GetSharedDevice());
}
void CreateDrawingSurface(Size drawSize)
{
this.drawingSurface = this.graphicsDevice.CreateDrawingSurface(
drawSize,
DirectXPixelFormat.B8G8R8A8UIntNormalized,
DirectXAlphaMode.Premultiplied);
}
void CreateSurfaceBrush(Compositor compositor)
{
this.drawingBrush = compositor.CreateSurfaceBrush(
this.drawingSurface);
}
public async static Task<CompositionImageBrush> FromImageStreamAsync(
Compositor compositor,
IRandomAccessStream stream,
Size drawSize)
{
CompositionImageBrush brush = new CompositionImageBrush();
brush.CreateDevice(compositor);
stream.Seek(0);
var decoder = await BitmapDecoder.CreateAsync(stream);
using (var softwareBitmap = await decoder.GetSoftwareBitmapAsync(
BitmapPixelFormat.Bgra8,
BitmapAlphaMode.Premultiplied))
{
brush.CreateDrawingSurface(drawSize);
brush.DrawSoftwareBitmap(softwareBitmap, drawSize);
}
brush.CreateSurfaceBrush(compositor);
return (brush);
}
void DrawSoftwareBitmap(SoftwareBitmap softwareBitmap, Size drawSize)
{
using (var drawingSession = CanvasComposition.CreateDrawingSession(
this.drawingSurface))
{
using (var bitmap = CanvasBitmap.CreateFromSoftwareBitmap(drawingSession.Device,
softwareBitmap))
{
// I've made the drawing surface the same size as the visual on screen
// and so I'm now going to draw this bitmap to that surface at the
// same size too which means that I'm stretching it at this point
// whereas a smarter thing might be to let the Stretch property on
// the brush itself stretch it.
drawingSession.DrawImage(bitmap,
new Rect(0, 0, drawSize.Width, drawSize.Height));
}
}
}
public void Dispose()
{
// TODO: I'm unsure about the lifetime of these objects - is it ok for
// me to dispose of them here when I've done with them and, especially,
// the graphics device?
this.drawingBrush.Dispose();
this.drawingSurface.Dispose();
this.graphicsDevice.Dispose();
}
CompositionGraphicsDevice graphicsDevice;
CompositionDrawingSurface drawingSurface;
CompositionSurfaceBrush drawingBrush;
}
I’m not 100% sure that I have that right but it means that I can now ‘walk up’ to this CompositionImageBrush class that I’ve made and grab a brush from it for an image that I have in a file and it’s going to use Win2D to render that image at the size specified onto a drawing surface that can then be used to paint a visual.
With that in place, I added a picture into my project;
and then I reworked the code that I had from the previous post. My MainPage.xaml UI stayed as it was;
<Page
x:Class="App287.MainPage"
xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
xmlns:local="using:App287"
xmlns:d="http://schemas.microsoft.com/expression/blend/2008"
xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"
mc:Ignorable="d">
<Grid
Background="red">
<Viewbox>
<TextBlock
x:Name="txtCount" />
</Viewbox>
<Canvas
Background="Transparent"
PointerReleased="OnPointerReleased"
SizeChanged="OnCanvasSizeChanged"
x:Name="drawCanvas"/>
</Grid>
</Page>
and I reworked the interface that I wrote to ‘abstract’ the difference between a XAML implementation and a Composition implementation. It became;
using System.Threading.Tasks;
using Windows.Foundation;
using Windows.Storage;
using Windows.UI.Xaml.Controls;
interface IDrawCanvasImages
{
void InitialiseWithCanvas(Canvas canvas);
Task SetImageFileAndDrawSizeAsync(StorageFile imageFile, Size drawSize);
void DrawImage(int x, int y);
void ClearImages();
}
and so here there’s a method to set the Canvas to be drawn to, then there’s one to set up the image to be drawn and the dimensions that it’s to be drawn it. Then there’s a method to draw an individual image and another to clear all the images.
As in the previous post, I wrote the code behind my XAML file in terms of that interface;
using System;
using System.Threading.Tasks;
using Windows.Foundation;
using Windows.Storage;
using Windows.UI.Xaml;
using Windows.UI.Xaml.Controls;
using Windows.UI.Xaml.Input;
public sealed partial class MainPage : Page
{
public MainPage()
{
this.InitializeComponent();
#if VISUAL_DRAWING
this.drawer = new VisualImageDrawer();
#else
this.drawer = new XamlImageDrawer();
#endif // XAMLDRAWING
this.drawer.InitialiseWithCanvas(this.drawCanvas);
this.timer = new DispatcherTimer();
this.timer.Interval = TimeSpan.FromMilliseconds(10);
this.timer.Tick += OnTimerTick;
this.timer.Start();
}
void OnTimerTick(object sender, object e)
{
this.txtCount.Text = (++this.ticks).ToString();
}
async void OnPointerReleased(object sender, PointerRoutedEventArgs e)
{
this.rectangleCount *= 4;
await RedrawAsync();
}
async void OnCanvasSizeChanged(object sender, SizeChangedEventArgs e)
{
await this.InitialiseForNewSizeAsync();
await this.RedrawAsync();
}
async Task InitialiseForNewSizeAsync()
{
var file = await StorageFile.GetFileFromApplicationUriAsync(
new Uri("ms-appx:///Images/kermit.png"));
var fullWidth = this.drawCanvas.ActualWidth /
this.Sidelength;
var fullHeight = this.drawCanvas.ActualHeight /
this.Sidelength;
this.drawSize = new Size(fullWidth, fullHeight);
await this.drawer.SetImageFileAndDrawSizeAsync(
file,
new Size(
this.drawSize.Width * (1 - SPACING_FACTOR),
this.drawSize.Height * (1 - SPACING_FACTOR)));
}
async Task RedrawAsync()
{
this.drawer.ClearImages();
await this.InitialiseForNewSizeAsync();
var horizontalSpacing = SPACING_FACTOR * this.drawSize.Width;
var verticalSpacing = SPACING_FACTOR * this.drawSize.Height;
var sidelength = this.Sidelength;
for (int i = 0; i < sidelength; i++)
{
for (int j = 0; j < sidelength; j++)
{
this.drawer.DrawImage(
(int)((i * this.drawSize.Width) + (horizontalSpacing / 2)),
(int)((j * this.drawSize.Height) + (verticalSpacing / 2)));
}
}
}
int Sidelength
{
get
{
return ((int)Math.Sqrt(this.rectangleCount));
}
}
static readonly float SPACING_FACTOR = 0.25f;
Size drawSize;
DispatcherTimer timer;
long ticks;
IDrawCanvasImages drawer;
int rectangleCount = 1;
}
and wrote a little base class to pick up a tiny bit of the implementation;
using System.Threading.Tasks;
using Windows.Foundation;
using Windows.Storage;
using Windows.UI.Xaml.Controls;
abstract class ImageDrawerBase : IDrawCanvasImages
{
public abstract void ClearImages();
public abstract void DrawImage(int x, int y);
protected abstract Task SetImageFileAsync(StorageFile imageFile);
protected Canvas DrawCanvas
{
get;
set;
}
protected Size DrawSize
{
get;
set;
}
public virtual void InitialiseWithCanvas(Canvas canvas)
{
this.DrawCanvas = canvas;
}
public Task SetImageFileAndDrawSizeAsync(StorageFile imageFile, Size drawSize)
{
this.DrawSize = drawSize;
return (this.SetImageFileAsync(imageFile));
}
}
before implementing this in a XAML renderer where the vast majority of the code is the same as in the previous post;
using System;
using System.Threading.Tasks;
using Windows.Foundation;
using Windows.Storage;
using Windows.UI.Xaml.Controls;
using Windows.UI.Xaml.Media;
using Windows.UI.Xaml.Media.Animation;
using Windows.UI.Xaml.Media.Imaging;
using Windows.UI.Xaml.Shapes;
class XamlImageDrawer : ImageDrawerBase
{
public override void ClearImages()
{
this.DrawCanvas.Children.Clear();
}
protected override async Task SetImageFileAsync(StorageFile imageFile)
{
using (var stream = await imageFile.OpenReadAsync())
{
var bitmapImage = new BitmapImage();
bitmapImage.SetSource(stream);
this.FillBrush = new ImageBrush()
{
ImageSource = bitmapImage,
Stretch = Stretch.Fill
};
}
}
public override void DrawImage(int x, int y)
{
Rectangle rectangle = new Rectangle()
{
Width = this.DrawSize.Width,
Height = this.DrawSize.Height,
Fill = FillBrush
};
Canvas.SetLeft(rectangle, x);
Canvas.SetTop(rectangle, y);
AddSpinAnimationXaml(rectangle);
this.DrawCanvas.Children.Add(rectangle);
}
static void AddSpinAnimationXaml(Rectangle rectangle)
{
RotateTransform transform = new RotateTransform();
rectangle.RenderTransform = transform;
rectangle.RenderTransformOrigin = new Point(0.5d, 0.5d);
DoubleAnimation angleAnimation =
new DoubleAnimation()
{
From = 0.0d,
To = 360
};
angleAnimation.EasingFunction = new SineEase();
Storyboard.SetTarget(angleAnimation, transform);
Storyboard.SetTargetProperty(angleAnimation, "Angle");
DoubleAnimation opacityAnimation =
new DoubleAnimation()
{
From = 0.0d,
To = 1.0d
};
opacityAnimation.EasingFunction = new SineEase();
Storyboard.SetTarget(opacityAnimation, rectangle);
Storyboard.SetTargetProperty(opacityAnimation, "Opacity");
Storyboard storyBoard = new Storyboard();
storyBoard.Duration = TimeSpan.FromSeconds(1);
storyBoard.RepeatBehavior = RepeatBehavior.Forever;
storyBoard.Children.Add(angleAnimation);
storyBoard.Children.Add(opacityAnimation);
storyBoard.Begin();
}
ImageBrush FillBrush;
}
and in a Windows.UI.Composition renderer that’s using the CompositionImageBrush from the previous code snippet and is perhaps around 50% different code from what I had in the previous post;
using System;
using System.Numerics;
using System.Threading.Tasks;
using Windows.Storage;
using Windows.UI.Composition;
using Windows.UI.Xaml.Controls;
using Windows.UI.Xaml.Hosting;
class VisualImageDrawer : ImageDrawerBase
{
public override void InitialiseWithCanvas(Canvas canvas)
{
base.InitialiseWithCanvas(canvas);
var canvasVisual = ElementCompositionPreview.GetElementVisual(
base.DrawCanvas);
this.compositor = canvasVisual.Compositor;
this.containerVisual = this.compositor.CreateContainerVisual();
this.rotationAnimation = this.compositor.CreateScalarKeyFrameAnimation();
this.rotationAnimation.InsertKeyFrame(0.0f, 0.0f);
this.rotationAnimation.InsertKeyFrame(1.0f, 360.0f);
this.rotationAnimation.Duration = TimeSpan.FromSeconds(1);
this.rotationAnimation.IterationBehavior = AnimationIterationBehavior.Forever;
this.opacityAnimation = this.compositor.CreateScalarKeyFrameAnimation();
this.opacityAnimation.InsertKeyFrame(0.0f, 0.0f);
this.opacityAnimation.InsertKeyFrame(1.0f, 1.0f);
this.opacityAnimation.Duration = TimeSpan.FromSeconds(1);
this.opacityAnimation.IterationBehavior = AnimationIterationBehavior.Forever;
ElementCompositionPreview.SetElementChildVisual(this.DrawCanvas,
this.containerVisual);
}
public override void ClearImages()
{
this.containerVisual?.Children.RemoveAll();
}
public override void DrawImage(int x, int y)
{
var rectangleVisual = this.compositor.CreateSpriteVisual();
rectangleVisual.Offset = new Vector3(x, y, 0);
rectangleVisual.Size = new Vector2(
(float)this.DrawSize.Width, (float)this.DrawSize.Height);
rectangleVisual.CenterPoint = new Vector3(
(float)(this.DrawSize.Width / 2), (float)(this.DrawSize.Height / 2), 0);
rectangleVisual.Brush = this.imageBrush.Brush;
rectangleVisual.StartAnimation("RotationAngleInDegrees", this.rotationAnimation);
rectangleVisual.StartAnimation("Opacity", this.opacityAnimation);
containerVisual.Children.InsertAtBottom(rectangleVisual);
}
protected async override Task SetImageFileAsync(StorageFile imageFile)
{
this.imageBrush?.Dispose();
using (var stream = await imageFile.OpenReadAsync())
{
this.imageBrush = await CompositionImageBrush.FromImageStreamAsync(
this.compositor,
stream,
this.DrawSize);
}
}
ScalarKeyFrameAnimation opacityAnimation;
ScalarKeyFrameAnimation rotationAnimation;
Compositor compositor;
ContainerVisual containerVisual;
CompositionImageBrush imageBrush;
}
With that all in place, how does this run? I spun up the XAML based rendering (below is 16 kermits);
Initially, the app reports 30 frames per second and the system compositor reports 60 and that stays pretty constant until I reach 1024 kermits – i.e. at 1024 kermits I was seeing the app reporting 16fps and the system compositor was reporting 60.
My interpretation of that is that I’m taxing the UI thread too much.
At 4096 kermits the app was reporting around 3fps and the system (compositor) was again reporting 60fps.
With the Visual based rendering (i.e. where the compositor is doing the image drawing, the animation of the opacity and the animation of the rotation) I see that initially the performance counters report 30fps for the app and 60fps for the system and that seems to stay constant up to 16384 kermits and then things go a little crazy as I try and step to 65536 kermits.
So, that’s another little experiment with the Visual layer – the code for it is here if you want to download and the main thing for me is that I probably picked up some re-usable code for painting visuals with images here.
Mike Taulty 
Pingback: A Fourth Experiment with the Visual Layer–One Line Effects on XAML | Mike Taulty