Following on from that previous post I thought it might be interesting to look across the controls that ship with the various frameworks (i.e. WPF, Silverlight and Silverlight for WP7) and see how they respond to being dropped into a touch-based interface.

I thought I’d take a smattering of controls that I’ve come across in touch interfaces – Button, Calendar, Slider, ListBox, CheckBox/RadioButton, ComboBox, TreeView and see how they work out on the different platforms (if they exist).

WPF 4 Built-In Controls

I started with a ListBox which is just displaying a bunch of strings – displayed here in Blend;

as it stands, that’s not very pleasant to use from the point of view of touch but a few minor tweaks;

• make the font size big (16 in my case)
• edit the ItemContainerStyle to make the padding on the parent border larger (I went with 6)
• edit the ListBox template and the ScrollViewer template within it to change the width of the vertical scrollbar (it defaults to 16.8 and I set it to 36) and the height of the horizontal scrollbar.

and I ended up with;

which I found pretty usable on my touch monitor. It responds to pan gestures and the content scrolls with inertia and gives boundary feedback so it all feels pretty good.

It’s a similar story for a TreeView. In its raw incarnation it’s pretty much impossible to use;

but a few tweaks;

• make the font size big ( 16 )
• edit the ItemContainerStyle to;
  • add some margin ( 6 ) around the parent grid
  • add some padding to the border that parents the header
  • change the width/height of the expander (I set it to 32)
  • edit the internal template on the Expander to
    • change the size of the area that you’ve got to hit-test in order to expand/collapse
• edit the template of the TreeView and give the scrollbars more size (I set heights/widths to 32)

I ended up with;

which was quite usable once you could look past my ugly triangles.

For basic controls like Buttons I find that they work pretty well – no particular problems although I found myself wanting to make them bigger than I usually would;

and I also wondered whether they should really be re-templated as the Windows Phone 7 does so that they have a big border around them that makes them more hittable. That is – keep the button relatively small but be generous on the hit-testing. I took the default template and wrapped a Border around the Chrome within it;

then gave that Border a solid colour with a zero alpha value and made sure it had some padding;

and that seemed to make my button more “pressable” in the sense of allowing for a margin of error around the outside of it.

It’s a similar story for a CheckBox and a RadioButton. By default it doesn’t work too badly because there’s quite a lot of screen to aim at even with small fonts;

but a large font does a lot to help;

( that’s 16 point ) and that makes it pretty usable although I must admit that I’m not so keen that the other visuals of the CheckBox don’t scale up as the size of the control scales up.

It’s easy enough to wander into the template and resize it manually;

although that doesn’t really help because the little tickmark inside it doesn’t scale properly so at that point you maybe re-template the whole thing (I’m not too keen on the WPF approach with BulletDecorator to this anyway because it seems a little opaque to me) and you can then make your checkbox as hit-testable as you like.

For example;

makes it pretty easy to use from a touch perspective.

What about a ComboBox? I’m not 100% sure about the usability of a ComboBox in a touch interface but I was actually able to use it reasonably even at default font sizes although selections tended to hit the wrong items.

Large fonts pretty much seemed to be all I needed here but I also added in a little padding to the ItemContainerStyle and the control became very usable with the pan gesture scrolling up/down;

What about a Slider? By default, it’s pretty hard to get hold of of that thumb and drag it around;

and no amount of sizing on that Slider will make the thumb bigger although if I size it carefully and then wrap it into a ViewBox that’s a cheap way of making it size and work more easily with a finger on a touch screen;

but I suspect the right way to go here is to re-template.

Finally, what about a calendar? By default, it’s a bit tricky;

I started with my usual trick of trying to change the font size and it had zero effect on the control. I tried applying my trick of wrapping it in a ViewBox and telling that ViewBox to do Uniform scaling and it worked pretty well

In this screenshot, these items are approximately “finger sized” and it’s pretty easy to select dates;

and probably good enough that I wouldn’t be tempted to go digging off into the templates on this one.

So, all-in-all I’d say that the built-in WPF controls do a pretty decent job of being touch ready in .NET 4.0 and a combination of font sizes, margins, padding and some ViewBoxes can take you a long way.

Silverlight

I’ve a feeling that this is going to be painful but let’s see how I get on. Starting with a ListBox in its “vanilla” configuration just displaying some strings;

It wasn’t very pleasant. Scrolling is only achieved by sliding the scroll bar which is way too small and item selection is also tricky. I altered;

• Font Size to 16
• Margin around the ItemContainerStyle’s Grid set to 6 to space things out a little
• Width of the VerticalScrollBar in the ScrollViewer template for the ListBox set to 32 rather than 18 (similarly for the HorizontalScrollBar’s height).

and that was better but it still lacks the capability to use a flick/pan gesture to scroll the ListBox items up and down. You have to use the scrollbar

I thought it was time to revisit the Touch project on codeplex as I remembered that it had a touch-based scrolling behaviour somewhere in it.

It does – it’s called TouchScrollBehavior and I went through the process of downloading the Surface SDK sample for Silverlight which it depends upon and then rebuilding the Touch project code for Silverlight 4.

However, I soon realised that the behavior worked fine for ListBox but not for anything else. There’s some code in there that deals specifically with ListBox and I had a look at trying to change it but got a bit bogged down ( there’s some odd stuff in that code which does things like loading lumps of XAML from strings which I find a bit peculiar ).

So, I built my own behavior from scratch which is intended to be applied to any ScrollViewer – it’s not perfect by any means but it works up to a point.

Here’s the code for that – note that it relies on the Surface SDK sample for Silverlight;

namespace SilverlightApplication5
{
  using System;
  using System.Collections.Generic;
  using System.Linq;
  using System.Windows;
  using System.Windows.Controls;
  using System.Windows.Input;
  using System.Windows.Input.Manipulations;
  using System.Windows.Interactivity;
  using System.Windows.Media;
  using System.Windows.Threading;
  using System.Diagnostics;
  using System.Windows.Controls.Primitives;

  public class TouchScrollBehavior : Behavior<ScrollViewer>
  {
    static TouchScrollBehavior()
    {
       activeBehaviors = new List<TouchScrollBehavior>();
    }
    static void EnsureTouch()
    {
      Touch.FrameReported += OnFrameReported;
    }
    static void OnFrameReported(object sender, TouchFrameEventArgs e)
    {
      if (capturedBehavior == null)
      {
        TouchPoint pt = e.GetPrimaryTouchPoint(Application.Current.RootVisual);

        if ((pt != null) &&
          (pt.Action == TouchAction.Down))
        {
          // TODO: has to be a better way of doing this but I've tried a few
          // and haven't hit one just yet. 
          for (
            int i = 0;
            ((i < activeBehaviors.Count) && (capturedBehavior == null));
            i++)
          {
            IEnumerable<UIElement> elements = 
              VisualTreeHelper.FindElementsInHostCoordinates(
                pt.Position,
                Application.Current.RootVisual);

            if ((elements != null) && (elements.Count() > 0))
            {
              if (elements.Contains(activeBehaviors[i].AssociatedObject))
              {
                capturedBehavior = activeBehaviors[i];
              }
              else
              {
                foreach (UIElement element in elements)
                {
                  if (IsParentOf(element, activeBehaviors[i].AssociatedObject))
                  {
                    capturedBehavior = activeBehaviors[i];
                    break;
                  }
                }
              }
            }
          }
        }
      }

      if (capturedBehavior != null)
      {        
        TouchPointCollection points = e.GetTouchPoints(Application.Current.RootVisual);

        if ((points != null) && (points.Count > 0))
        {
          capturedBehavior.HandleTouchEvents(points);

          if ((points.Count == 1) && (points[0].Action == TouchAction.Up))
          {
            capturedBehavior.CompleteManipulation();
          }
        }
      }
    }
    static bool IsParentOf(DependencyObject parent, FrameworkElement child)
    {
      bool isParent = false;
      DependencyObject candidate = child.Parent;

      while (!isParent && (candidate != null))
      {
        isParent = candidate == parent;

        FrameworkElement nextCandidate = candidate as FrameworkElement;
        candidate = nextCandidate == null ? null : nextCandidate.Parent;
      }
      return (isParent);
    }
    public TouchScrollBehavior()
    {
      this.manipProcessor = new Lazy<ManipulationProcessor2D>(() =>
        {
          ManipulationProcessor2D proc = new ManipulationProcessor2D(
            Manipulations2D.Translate);

          proc.Delta += OnManipulationDelta;
          proc.Completed += OnManipulationCompleted;

          return (proc);
        });

      this.inertiaProcessor = new Lazy<InertiaProcessor2D>(() =>
        {
          InertiaProcessor2D proc = new InertiaProcessor2D();

          // TODO: this is a pure magic value of 0.01f
          proc.TranslationBehavior.DesiredDeceleration = 0.01f;
          proc.Delta += OnManipulationDelta;
          proc.Completed += OnInertiaCompleted;
          return (proc);
        });

      this.timer = new Lazy<DispatcherTimer>(() =>
        {
          DispatcherTimer newTimer = new DispatcherTimer();

          // TODO: another magic value
          newTimer.Interval = new TimeSpan(0, 0, 0, 0, 50);
          newTimer.Tick += OnTick;
          return (newTimer);
        });
    }
    void OnInertiaCompleted(object sender, Manipulation2DCompletedEventArgs e)
    {
      this.timer.Value.Stop();
    }
    void OnTick(object sender, EventArgs e)
    {
      this.inertiaProcessor.Value.Process(DateTime.Now.Ticks);
    }
    void OnManipulationCompleted(object sender, 
      Manipulation2DCompletedEventArgs args)
    {
      capturedBehavior = null;
      this.scrollDirection = null;

      if ((args.Total.TranslationX >= MOUSE_CLICK_TOLERANCE) ||
        (args.Total.TranslationY >= MOUSE_CLICK_TOLERANCE))
      {
        this.inertiaProcessor.Value.TranslationBehavior.InitialVelocityX =
          args.Velocities.LinearVelocityX;

        this.inertiaProcessor.Value.TranslationBehavior.InitialVelocityY =
          args.Velocities.LinearVelocityY;

        this.timer.Value.Start();
      }
    }
    void OnManipulationDelta(object sender, Manipulation2DDeltaEventArgs e)
    {
      if (!this.scrollDirection.HasValue)
      {
        // TODO: this is pretty arbitrary
        this.scrollDirection =
          Math.Abs(e.Velocities.LinearVelocityX) > Math.Abs(e.Velocities.LinearVelocityY) ?
            Orientation.Horizontal : Orientation.Vertical;
      }

      if (this.scrollDirection == Orientation.Vertical)
      {
        this.AssociatedObject.ScrollToVerticalOffset(
          this.AssociatedObject.VerticalOffset + e.Delta.TranslationY);
      }
      else
      {
        this.AssociatedObject.ScrollToHorizontalOffset(
          this.AssociatedObject.HorizontalOffset - e.Delta.TranslationX);
      }
    }
    void StopInertia()
    {
      if (this.timer.Value.IsEnabled)
      {
        this.inertiaProcessor.Value.Complete(DateTime.Now.Ticks);
        this.timer.Value.Stop();
      }
    }
    void HandleTouchEvents(TouchPointCollection points)
    {
      StopInertia();

      IEnumerable<Manipulator2D> manips = 
        from p in points
        select new Manipulator2D()
        {
          X = (float)p.Position.X,
          Y = (float)p.Position.Y
        };

      this.manipProcessor.Value.ProcessManipulators(
        DateTime.Now.Ticks, manips);
    }
    protected override void OnAttached()
    {
      base.OnAttached();
      EnsureTouch();
      activeBehaviors.Add(this);
    }
    protected override void OnDetaching()
    {
      base.OnDetaching();
      activeBehaviors.Remove(this);
    }
    void CompleteManipulation()
    {
      this.manipProcessor.Value.CompleteManipulation(
        DateTime.Now.Ticks);
    }
    Lazy<DispatcherTimer> timer;
    Lazy<InertiaProcessor2D> inertiaProcessor;
    Lazy<ManipulationProcessor2D> manipProcessor;
    static TouchScrollBehavior capturedBehavior;
    static List<TouchScrollBehavior> activeBehaviors;
    Orientation? scrollDirection;

    const double MOUSE_CLICK_TOLERANCE = 5.0;
  }
}

With that in place I can then make use of it by dragging it to my ListBox in Blend – that is;

and everything seemed to be working pretty well in my ListBox;

The TreeView followed a similar path. In short;

1. Large fonts.
2. Resized the horizontal and vertical scrollbars.
3. Gave vertical/horizontal margins to the items in the ItemContainerStyle.
4. Edited the ItemContainerStyle to change the width of the grid column containing the ExpanderButton and then the template of that control itself it such that it displayed the paths for expanded/collapsed at larger size.
5. Added the TouchScrollBehavior that I wrote for the ListBox example.

and that all worked pretty well too from a touch perspective;

The other controls – Button, CheckBox, Slider, Calendar, ComboBox all pretty much worked out just like they do for WPF. Here’s the notes that I made whilst experimenting with them;

• With Button I did the same trick of re-templating to wrap everything in a Border with an amount of padding and then set the alpha on that Border to 0 in order to provide a larger hit-test area. Seemed to work quite well.
• Re-templating the Silverlight CheckBox in order to get the visuals beyond the content to scale is different from the WPF one but it’s still a bit painful because there are lots of visuals in there with hard-coded widths (16) whereas I’d hoped to find perhaps a row/column with a hard-coded width and content within it that was designed to scale. No big deal though, it’s still not very hard to change as you can see here and I cheated by grouping the visuals into a Grid and then grouping that into a ViewBox;

• The trick of wrapping a Slider in a ViewBox seemed to work reasonably in Silverlight.
• When it came to the ComboBox, applying large fonts and some margins around the ItemContainerStyle content worked reasonably well but when it came to applying my TouchScrollBehavior to the ScrollViewer within the ComboBox the wheels came off a little in that the scrolling aspect of my behavior worked pretty well but item selection isn’t very good.
  • Why’s that?
    • My behavior very deliberately leaves mouse event promotion switched on which is great because the framework underneath sends the right mouse down/up events which makes selection work.
    • This works ok in a ListBox/TreeView scenario.
    • This does not work so well on a ScrollViewer in a popup because the mouse events tend to dismiss the popup and the selection doesn’t get changed. You can get it to work but it feels very awkward so my behavior would need more work here.
• When it came to the Calendar control, again the trick of dropping it into a ViewBox seemed to work quite well and I found the control pretty usable on my touch monitor when I lean across the desk.

Silverlight on Windows Phone 7

This is really only here for completeness. On Windows Phone 7 there is no assumption that you will have any input means other than touch and so,of course, the built in set of controls work well with touch input and there’s even that extra set of controls that really excel under touch input like the Panaroma control and the Pivot control.

It’d be great to see those controls show up for the full Silverlight framework on the desktop

Finishing Up

I’m not really planning to write any more of these “touch-centric” posts for the moment. I looked at;

1. What Comes For Free
2. Raw Touch Events
3. Manipulation and Gestures
4. A Simple Single-Code-Base Example for WPF, Silverlight and WP7
5. and then I took a look at how the built-in controls work in this post.

As an aside, a few people have asked me what hardware I’m using for this work and it’s a Dell SX2210T which I’m finding to be pretty good although there does seem to be a glitch when you run it in dual-monitor mode and the touch events seem to fire on the wrong monitor Running it in single monitor mode or a reboot seem to fix the problem.

Enjoy!