Silverlight and NESL Redux–Touch

Stephen pointed out last week that there’s a preview update to the “Native Extensions to Silverlight” up on CodePlex and this preview adds some capabilities around touch and having written a little around that topic recently;

I thought I’d take a look at what was going on in the NESL preview – this follows on from my previous post where I took more of an “overview” approach to NESL.

I downloaded the preview, installed it. Then I wanted to put together a simple example of a FlickR search with a touch gesture or two to flick through pictures.

I built a little basic support in order that my application had a few different pieces of UI depending on whether it was running in the browser and/or whether NESL was installed;

and that gets me to to a point where my app can know that NESL is installed and so start to rely on it – my previous post did this in more detail.

I hit a problem with the NESL installation on this preview in that the call Installer.CheckNESLInstalled failed on me and so I came up with my own hacky way of denoting whether NESL had or hadn’t been installed although this would only work for my own app rather than across the system. It’s probably a “preview” thing or a user error on my part as it worked fine the last time I used it.

With the NESL 2 preview installed, it’s then time to figure out what can be done about touch and there’s a new namespace Microsoft.Silverlight.Windows.Touch ( I ended up referencing Microsoft.Silverlight.Windows.dll and Microsoft.Silverlight.Windows.Touch.dll ) and within there you’ll find;

and these classes look pretty similar to what you see in WPF 4 in terms of a ManipulationProcessor and an InertiaProcessor although I think that the ManipulationProcessor is a little different here in that it also has the notion of raising Gestures whereas in WPF 4 you specify whether you want gestures or not up-front if I remember correctly. See my previous post for more on that.

This is convenient then as you have both options in one place. I figured that I’d try and put together a simple photo viewer which searches FlickR and then lets you move forwards through result-sets by using a simple flick gesture.

Signing up for gestures is pretty easy. If I’m running out of browser then I can set up a ManipulationProcessor and tell it what I want to do;

      // This is my own flag to determine app & NESL installation status
      if (this._appState == AppState.OutOfBrowserNeslInstalled)
      {
        this._manipProcessor = new ManipulationProcessor();

        this._manipProcessor.RaiseGestures =
          Gesture.Flick | Gesture.TwoFingerTap;

        Touch.FrameReported += (s, e) =>
          {
            this._manipProcessor.ProcessTouchPoints(
              e.GetTouchPoints(Application.Current.RootVisual),
              e.Timestamp);
          };

        this._manipProcessor.Flick += OnFlick;
        this._manipProcessor.TwoFingerTap += OnTap;
      }

and that should serve me fine.

I wanted to re-use the code that I’d worked up on this previous post which used the Reactive Extensions in order to bring back some images from FlickR.

They key of this code was a routine that made an observable sequence of images to be brought back from FlickR. This ended up looking like;

  class FlickrSearch
  {
    public FlickrSearch(string searchText,
      TimeSpan interval)
    {
      this.state = new FlickSearchState(searchText);
      this.interval = interval;

      this.photoResults = new Lazy<IObservable<Stream>>(
        MakePhotoResults);
    }
    public IObservable<Stream> PhotoResults
    {
      get
      {
        return (this.photoResults.Value);
      }
    }
    IObservable<Stream> MakePhotoResults()
    {
      var webRequests =
        this.state.GetObservableSearchUris().SelectMany(
          uri =>
          {
            HttpWebRequest wr = WebRequest.Create(uri) as HttpWebRequest;

            return (Observable.Defer(Observable.FromAsyncPattern<WebResponse>(
              wr.BeginGetResponse, wr.EndGetResponse)));
          });

      var xmlResponses =
        webRequests.Select(
          wr =>
          {
            XElement xml = null;
            FlickrPhotoPageResult result = null;

            using (Stream stream = wr.GetResponseStream())
            {
              xml = XElement.Load(stream);
              result = new FlickrPhotoPageResult(xml);
            };
            return (result);
          });

      var photoResponses =
        xmlResponses.SelectMany(
          pageResponse =>
          {
            return (pageResponse.GetPhotos());
          });

      var slowedDownResponses =
        photoResponses.Zip(
          Observable.Interval(this.interval),
          (photo, count) =>
          {
            return (photo);
          });

      var slowedDownLoadsNextPage =
        slowedDownResponses.Do(p =>
        {
          // TODO: not sure about this having a side-effect which
          // can advance/end the observable at the "tail of the
          // chain" here.
          if (p.IsLastImage)
          {
            this.state.Advance(end: true);
          }
          else if (p.IsLastImageOnCurrentPage)
          {
            this.state.Advance(end: false);
          }
        });

      var streams =
        slowedDownLoadsNextPage.SelectMany(
          photo =>
          {
            HttpWebRequest request = WebRequest.Create(photo.ImageUri) as HttpWebRequest;

            return(Observable.Defer(
              Observable.FromAsyncPattern<WebResponse>(request.BeginGetResponse,
                request.EndGetResponse)));
          });

      var byteArrays =
        streams.Select(
          resp =>
          {
            HttpWebResponse webResponse = (HttpWebResponse)resp;

            MemoryStream ms = new MemoryStream();

            using (Stream s = resp.GetResponseStream())
            {
              s.CopyTo(ms);
            }
            ms.Seek(0, SeekOrigin.Begin);
            return (ms);
          });

      return (byteArrays);
    }
    Lazy<IObservable<Stream>> photoResults;
    FlickSearchState state;
    TimeSpan interval;
  }

and what it is trying to do is bring back pages of results from FlickR and throttle them down so that they don’t arrive too frequently but, instead, only arrive at most every this.interval period where that period is configurable by whoever instantiates the search instance in the first place.

However, in my Silverlight app I didn’t want to produce a new image every so many seconds. I wanted to produce one every time the user did a flick gesture on the screen.

My FlickrSearch class needed to be able to adapt so that rather than taking an interval and then plugging it in to Observable.Interval it could perhaps take an IObservable<T> and use that as part of the Zip operation which it makes use of for throttling.

I also realised that my previous code throttled really at the wrong point in that it did;

Make new URI for FlickR search based on a page size and a page number
Make HTTP request for that URI
Parse Response into XML
Split Response into individual photos
Throttle
Make HTTP request for each photo’s image
Surface that image

and I figured I might move the throttling from step 5 here to after step 7 in order that a page of images are ready to go at pretty much the same time.

Here’s what I ended up with ( not too pretty – I’m sure there’s about 100 ways to do this better and/or correctly );

  class FlickrSearch<T>
  {
    public FlickrSearch(string searchText,
      IObservable<T> throttlingObservable)
    {
      this.state = new FlickSearchState(searchText);
      this.throttlingObservable = throttlingObservable;

      this.photoResults = new Lazy<IObservable<MemoryStream>>(
        MakePhotoResults);
    }
    public IObservable<MemoryStream> PhotoResults
    {
      get
      {
        return (this.photoResults.Value);
      }
    }
    IObservable<MemoryStream> MakePhotoResults()
    {
      // As new URIs are produced, we turn them into HttpWebResponses
      var webRequests =
        this.state.GetObservableSearchUris().SelectMany(
          uri =>
          {
            HttpWebRequest wr = WebRequest.Create(uri) as HttpWebRequest;

            return (Observable.Defer(Observable.FromAsyncPattern<WebResponse>(
              wr.BeginGetResponse, wr.EndGetResponse)));
          });

      // We take each of those responses and we load the XML back from it
      // as a page of data.
      var xmlResponses =
        webRequests.Select(
          wr =>
          {
            XElement xml = null;
            FlickrPhotoPageResult result = null;

            using (Stream stream = wr.GetResponseStream())
            {
              xml = XElement.Load(stream);
              result = new FlickrPhotoPageResult(xml);
            };
            return (result);
          });

      // We take that page of data and turn it into photograph instances.
      var photoResponses =
        xmlResponses.SelectMany(
          pageResponse =>
          {
            return (pageResponse.GetPhotos());
          });

      // We're happy to make the web requests in order to get the streams
      // containing the photo bits.
      var photoResponseStreams =
        photoResponses.SelectMany(
          photo =>
          {
            HttpWebRequest request = WebRequest.Create(photo.ImageUri) as HttpWebRequest;

            var response = Observable.FromAsyncPattern<WebResponse>(
              request.BeginGetResponse, request.EndGetResponse);

            return (Observable.Defer(response).Select(
              resp => Tuple.Create(photo, resp)));
          });

      // And produce a stream from it.
      var photoAndStreams =
        photoResponseStreams.Select(
          photoResp =>
          {
            HttpWebResponse webResponse = (HttpWebResponse)photoResp.Item2;

            MemoryStream ms = new MemoryStream();

            using (Stream s = webResponse.GetResponseStream())
            {
              s.CopyTo(ms);
            }
            ms.Seek(0, SeekOrigin.Begin);

            return (Tuple.Create(photoResp.Item1, ms));
          });

      // We always want the first one because the user has searched and
      // expects a result.
      var first =
        photoAndStreams.Take(1);

      // We also want the rest but we want to defer them until a later 
      // time.
      var rest =
        photoAndStreams.Skip(1);

      // We produce the first result followed by the remaining results
      // only when the "throttlingObservable" produces a value so the
      // remaining results won't come through until some timer tick
      // or event fires or whatever.
      var slowedDownResponses =
        first.Concat(
          rest.Zip(
            this.throttlingObservable,
            (photoResp, throttle) =>
            {
              return (photoResp);
            }));

      // We need to make sure that we request the next page if necessary
      // by updating the URI or that we signal the result-set is complete.
      var slowedDownLoadsNextPage =
        slowedDownResponses.Do(photoAndStream =>
        {
          // TODO: not sure about this having a side-effect which
          // can advance/end the observable at the "tail of the
          // chain" here.
          if (photoAndStream.Item1.IsLastImage)
          {
            this.state.Advance(end: true);
          }
          else if (photoAndStream.Item1.IsLastImageOnCurrentPage)
          {
            this.state.Advance(end: false);
          }
        });

      return (slowedDownLoadsNextPage.Select(
        photoAndStream => photoAndStream.Item2));
    }
    Lazy<IObservable<MemoryStream>> photoResults;
    FlickSearchState state;
    IObservable<T> throttlingObservable;
  }

and so what this is trying to do is to throttle back the web requests and the production of images such that we only move on to the next image when the IObservable<T> passed to the constructor produces some value.

I’m feeling that I need a lot more practise with Rx but I’ll get there one day

What it’s also trying to do ( around lines 92, 98 ) is trying to make sure that we always produce the first image regardless because if the user has done a search then they expect the first image to show up without having to do a flick gesture to see it.

With this in place, I did a clumsy looking combination such that I tied together the Flick event from the ManipulationProcessor with an instance of FlickrSearch and brought them together;

      this._flickEvents =
        Observable
          .FromEvent<FlickEventArgs>(this._manipProcessor, "Flick");

and then tying that into the search such that the Flick event becomes the throttle for the production of photos;

FlickrSearch<IEvent<FlickEventArgs>> search =
          new FlickrSearch<IEvent<FlickEventArgs>>(
            this.SearchText, this._flickEvents);

        this._searchSubscription =
          search
            .PhotoResults
            .ObserveOnDispatcher()
            .Subscribe(
            ms =>
            {
              BitmapImage image = new BitmapImage();
              image.SetSource(ms);
              this.CurrentImage = image;
            });

this all seems to hang together “reasonably well” and I also sync’d up the TwoFingerTap gesture to raise a search dialog box;

      this._manipProcessor.TwoFingerTap += OnTap;
    }
    void OnTap(object sender, TapEventArgs e)
    {
      this.ShowSearch = Visibility.Visible;
    }

which is simple enough and here’s a quick video of me using the thing and doing the two-finger-tap to raise the search box and then using a flick gesture to move to the next photos (I only support going forwards Smile );

In conclusion – the touch support here seems pretty nice and combines the 2 different options of going with pre-determined gestures like flick or doing manipulation and inertia processing on your own object. I haven’t explored those parts of the APIs because I’m assuming it’s pretty similar to doing the work in WPF 4.

When would I use this though? I’d use it in situations where I’d already decided that I was going to make use of the Native Extensions. These come with a fairly big additional installation requirement whereas there are steps that I can take in Silverlight 4 to make use of touch support ( as discussed at length in those previous posts at the top of this article ) without having to install anything or even leave the confines of a browser application so I’d probably follow the path of “least dependency” and go with those options if I could and go down the NESL path if my app was already using NESL for other reasons.

Here’s the source-code if you want to have an experiment yourself. You’ll need the NESL 2 Preview bits and Rx to build it.

Mike Taulty

I do some developer stuff on Microsoft technologies in the UK

Silverlight and NESL Redux–Touch

Share this:

Like this:

Related