This post really follows up on the demo that I built up in this previous post;
in that screen capture and blog post I walked through a demo of code where I was using AllJoyn in Windows 10 across desktop/phone and IoT devices (Raspberry PI 2 in my case).
It’s a simple scenario where I cooked up an imaginary ‘lightbulb’ interface which I then implement in an app in a couple of ways;
- If the app is running on a non Windows IoT device then the interface just changes the colour of a path on screen.
- If the app is running on a Windows IoT device then it will additionally attempt to light up a real LED.
I showed that working in the previous video and the code is all available for it on the blog post.
At the time though, what I really wanted to do was to control it with Cortana but I didn’t have Cortana working on my PC at the time that I wrote that post.
Since then, I’ve moved to Windows 10 build 10240 and I have a working Cortana (albeit after some struggles around UK/US regions and settings) and so I thought that I’d try it out with my goal being to enable this (made up) scenario;
- A lightbulb service comes onto the AllJoyn network and advertises itself in some location (e.g. office, kitchen, etc)
- A user should be able to ask Cortana to show all the lightbulbs that ‘she’ can see.
- A user should be able to ask Cortana to turn on/off lightbulbs.
I thought it might be a fun thing to play with and I could certainly see some future scenario of shouting at the XBOX One to turn the lights on outside or similar.
Everything I write here builds on the previous post and so I wouldn’t expect this to make any sense if you haven’t seen that post and I’m only going to write up the additional pieces here rather than run through the whole thing again.
Before I go there though, here’s a quick screencapture/video of me trying out the bits I put together;
Additions to the Existing Code
The first thing that I felt that I needed to do was to add a little more ‘functionality’ to my lightbulb interface beyond switch [on/off] and so I added a couple more simple methods;
<node name="/com/taulty/lightbulb">
<interface name="com.taulty.lightbulb">
<method name="switch">
<arg name="on" type="b" direction="in"/>
</method>
<method name="getLocation">
<arg name="returnvalue" type="s" direction="out"/>
</method>
<method name="getStatus">
<arg name="returnvalue" type="b" direction="out"/>
</method>
</interface>
</node>
and so now my interface encapsulates the idea of asking the lightbulb where it is located and whether it is already on/off. I then re-worked my 2 implementations of a lightbulb in order to implement these additional pieces of functionality but it’s just a small amount of extra code in each case.
Previously, I’d had the lightbulb app advertise itself with AllJoyn immediately that the app started up but, because the interface now has the idea of a location for the lightbulb I added a textbox and a button so that the location can be set before advertising;
Because my Raspberry PI doesn’t usually have a keyboard plugged in, I added some IoT Core specific code to set a timer such 5 seconds after startup it will act as though the button has been pressed and will go ahead and set the location of that lightbulb to be ‘kitchen’. Cheap and cheerful!
Beyond that, I didn’t make many (or probably) any mods to the code that I’d had in the previous post acting as a lightbulb service.
Addition 2 – Linking In Cortana
In trying to do something with Cortana, I made reference to the sample on GitHub but whereas that sample shows two scenarios;
- Voice commands being used to launch the foreground UI of an app and steer it to navigate to particular content.
- Voice commands being used to speak to a background service of an app and steer it to perform specific tasks with UI hosted on Cortana’s own canvas.
I focused entirely on (2) here and so my app is made up of 2 projects – a blank UWP app and a Windows Runtime Component to host my background task. I made the UWP app reference the component project so that the bits get copied over at build time.
and I made sure (using the manifest editor!) that I’d specified that this app had an app service;
and I’ll admit that I find it very odd that for this particular type of background task there’s no need (AFAIK) to write registration code to register the background task on a trigger. It just ‘exists’.
I then just threw a button on a form and had the click handler register my voice commands;
async void OnRegisterCommands(object sender, RoutedEventArgs e)
{
var file = await StorageFile.GetFileFromApplicationUriAsync(new Uri(@"ms-appx:///VoiceCommands.xml"));
await VoiceCommandDefinitionManager.InstallCommandDefinitionsFromStorageFileAsync(file);
}
with the voice commands looking like this;
<?xml version="1.0" encoding="utf-8" ?>
<VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.2">
<!-- NB: Not provided UK voice commands as, presently, my PC only
works with Cortana when I'm in UK mode
-->
<CommandSet xml:lang="en-us" Name="VoiceCommands">
<CommandPrefix>lightswitch</CommandPrefix>
<Example>show the lights</Example>
<Command Name="showLights">
<Example>show the lights</Example>
<ListenFor RequireAppName="BeforeOrAfterPhrase">show [the] lights</ListenFor>
<Feedback>Showing all the lights I can find...</Feedback>
<VoiceCommandService Target="VoiceCommandService"/>
</Command>
<Command Name="switchLight">
<Example>turn the bedroom lights on</Example>
<ListenFor RequireAppName="BeforeOrAfterPhrase">turn [the] {dictatedLocation} lights {onOffStatus}</ListenFor>
<Feedback>Just looking for that light for you...</Feedback>
<VoiceCommandService Target="VoiceCommandService"/>
</Command>
<PhraseList Label="onOffStatus">
<Item>On</Item>
<Item>Off</Item>
</PhraseList>
<PhraseTopic Label="dictatedLocation" Scenario="Natural Language">
</PhraseTopic>
</CommandSet>
</VoiceCommands>
with that in place, all of the remaining implementation lives in my background task project which proved to contain a few challenges…
The Background Task
Where I found writing a background task to be a bit of a challenge here is around the generated types that I seem to get from my AllJoyn interface via the alljoyncodegenerator.exe tool. What I need to do is something like;
- Receive a voice command from Cortana like ‘show all the lights’.
- Execute a query to find all the lightbulb services on the network.
- Format the results back to Cortana.
However…the class that I get generated from the AllJoyn interface is a lightbulbWatcher class and what it knows how to do is to wait around for lightbulb services to show up on the network.
What it doesn’t seem to be particularly set up to do is to get me all of the lightbulbs that are on the network at a specific point in time.
Meanwhile, Cortana is a demanding client – she’s not going to hang around for ever while I build a list of lightbulbs.
Consequently, I went with an approach something like this;
- When a voice command arrives at my background task, I construct a lightbulb watcher and ask it to look for lightbulbs.
- Meanwhile, I set about waiting for a 10 second delay and I stop watching for lightbulb arrivals after that 10 seconds has elapsed.
So far, this seems to work reasonably well but I don’t know whether it’d scale to 10-100 instances of the service on a slower network.
There was another challenge in building a list of lightbulb services here – it goes something like this;
- Voice command arrives.
- I create a lightbulb watcher and wait around for up to 10 seconds for lightbulb instances to be discovered.
- When an instance is discovered;
- I grab the offered lightbulb consumer as I need to talk to the lightbulb service to ask its location and status
- I put some details about that lightbulb onto a list.
- After around 10 seconds I then continue the work with Cortana which might be;
- Simply returning the list of status values and locations
- Communicating with a specific lightbulb to turn it on/off.
In my original implementation, I assumed that at step 3.2 above, I could get rid of the lightbulb service consumer and recreate it at step 4.2 if needed. However, I found that my attempts to do this were met with ‘you have already joined this AllJoyn session’ type errors and so in the end I keep around all the consumers from step 3.2 until step 4 is entirely finished and then I release them all.
The code for my background task then is as below;
namespace CortanaTestComponent
{
using com.taulty.lightbulb;
using System;
using System.Collections.Concurrent;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using Windows.ApplicationModel.AppService;
using Windows.ApplicationModel.Background;
using Windows.ApplicationModel.VoiceCommands;
using Windows.Devices.AllJoyn;
using Windows.Storage;
public sealed class TheTask : IBackgroundTask
{
private const int DELAY_TIMEOUT = 10000;
private const string VOICE_COMMAND_SHOW_LIGHTS = "showLights";
private const string VOICE_COMMAND_SWITCH_LIGHT = "switchLight";
private const string VOICE_COMMAND_LOCATION_KEY = "dictatedLocation";
private const string VOICE_COMMAND_ON_OFF_KEY = "onOffStatus";
class ServiceInfoWithLocation
{
public lightbulbConsumer Consumer { get; set; }
public string Location { get; set; }
public bool IsOn { get; set; }
}
public async void Run(IBackgroundTaskInstance taskInstance)
{
var triggerDetails = taskInstance.TriggerDetails as AppServiceTriggerDetails;
if (triggerDetails?.Name == "VoiceCommandService")
{
var deferral = taskInstance.GetDeferral();
var cancelledTokenSource = new CancellationTokenSource();
// Whatever the command is, we need a list of lightbulbs and their
// location names. Attemping to build that here but trying to make
// sure that we factor in cancellation.
taskInstance.Canceled += (s, e) =>
{
cancelledTokenSource.Cancel();
};
VoiceCommandServiceConnection voiceConnection =
VoiceCommandServiceConnection.FromAppServiceTriggerDetails(triggerDetails);
voiceConnection.VoiceCommandCompleted += (s, e) =>
{
cancelledTokenSource.Cancel();
};
var serviceInfoList = await this.GetLightListingAsync(DELAY_TIMEOUT, cancelledTokenSource.Token);
await this.ProcessVoiceCommandAsync(serviceInfoList, voiceConnection);
cancelledTokenSource.Dispose();
this.StopWatcherAndBusAttachment(serviceInfoList);
deferral.Complete();
}
}
async Task ProcessVoiceCommandAsync(List<ServiceInfoWithLocation> serviceInfoList,
VoiceCommandServiceConnection voiceConnection)
{
var command = await voiceConnection.GetVoiceCommandAsync();
switch (command.CommandName)
{
case VOICE_COMMAND_SHOW_LIGHTS:
await ProcessShowLightsCommandAsync(serviceInfoList, voiceConnection);
break;
case VOICE_COMMAND_SWITCH_LIGHT:
await ProcessSwitchLightCommandAsync(serviceInfoList, voiceConnection);
break;
default:
break;
}
}
async Task ProcessSwitchLightCommandAsync(
List<ServiceInfoWithLocation> serviceInfoList,
VoiceCommandServiceConnection voiceConnection)
{
var message = new VoiceCommandUserMessage();
var tiles = new List<VoiceCommandContentTile>();
bool worked = false;
if ((serviceInfoList == null) || (serviceInfoList.Count == 0))
{
message.SpokenMessage = "I couldn't find any lights at all, sorry";
message.DisplayMessage = "No lights could be found at any location";
}
else
{
var voiceCommand = await voiceConnection.GetVoiceCommandAsync();
var location = ExtractPropertyFromVoiceCommand(voiceCommand, VOICE_COMMAND_LOCATION_KEY);
var onOff = ExtractPropertyFromVoiceCommand(voiceCommand, VOICE_COMMAND_ON_OFF_KEY);
if (string.IsNullOrEmpty(location))
{
message.SpokenMessage = "I couldn't find a location in what you said, sorry";
message.DisplayMessage = "Interpreted text did not contain an audible location";
}
else if (string.IsNullOrEmpty(onOff))
{
message.SpokenMessage = "I couldn't figure out whether you said on or off, sorry";
message.DisplayMessage = "Not clear around on/off status";
}
else
{
var serviceInfo = serviceInfoList.SingleOrDefault(
sinfo => string.Compare(sinfo.Location.Trim(), location.Trim(), true) == 0);
if (serviceInfo == null)
{
message.SpokenMessage = $"I couldn't find any lights in the location {location}, sorry";
message.DisplayMessage = $"No lights in the {location}";
}
else
{
// It may just work...
await serviceInfo.Consumer.SwitchAsync(string.Compare(onOff, "on", true) == 0);
message.SpokenMessage = $"I think I did it! The light should now be {onOff}";
message.DisplayMessage = $"the light is now {onOff}";
}
}
}
var response = VoiceCommandResponse.CreateResponse(message);
if (worked)
{
await voiceConnection.ReportSuccessAsync(response);
}
else
{
await voiceConnection.ReportFailureAsync(response);
}
}
static string ExtractPropertyFromVoiceCommand(VoiceCommand voiceCommand, string propertyKey)
{
string result = string.Empty;
if (voiceCommand.Properties.ContainsKey(propertyKey))
{
var entries = voiceCommand.Properties[propertyKey];
if ((entries != null) && (entries.Count > 0))
{
result = entries[0];
}
}
return(result);
}
static async Task ProcessShowLightsCommandAsync(
List<ServiceInfoWithLocation> serviceInfoList,
VoiceCommandServiceConnection voiceConnection)
{
var onImageFile = await StorageFile.GetFileFromApplicationUriAsync(
new Uri("ms-appx:///Assets/Cortana68x68On.png"));
var offImageFile = await StorageFile.GetFileFromApplicationUriAsync(
new Uri("ms-appx:///Assets/Cortana68x68Off.png"));
var message = new VoiceCommandUserMessage();
var tiles = new List<VoiceCommandContentTile>();
if ((serviceInfoList == null) || (serviceInfoList.Count == 0))
{
message.SpokenMessage = "Either something went wrong, or there are no lights";
message.DisplayMessage = "I didn't find any lights, sorry";
}
else
{
message.SpokenMessage = "Yay! I found some lights. Here you go";
message.DisplayMessage = "Lights found in following places...";
foreach (var light in serviceInfoList)
{
tiles.Add(
new VoiceCommandContentTile()
{
Title = "Light",
TextLine1 = $"located in {light.Location}",
ContentTileType = VoiceCommandContentTileType.TitleWith68x68IconAndText,
Image = light.IsOn ? onImageFile : offImageFile
});
}
}
var response = VoiceCommandResponse.CreateResponse(message, tiles);
await voiceConnection.ReportSuccessAsync(response);
}
async Task<List<ServiceInfoWithLocation>> GetLightListingAsync(
int timeoutMs,
CancellationToken cancelledToken)
{
List<ServiceInfoWithLocation> list = null;
var safeBag = new ConcurrentBag<ServiceInfoWithLocation>();
this.busAttachment = new AllJoynBusAttachment();
this.watcher = new lightbulbWatcher(this.busAttachment);
this.watcher.Added += async (s,e) =>
{
var result = await lightbulbConsumer.JoinSessionAsync(e, s);
if (result.Status == AllJoynStatus.Ok)
{
var getLocationResult = await result.Consumer.GetLocationAsync();
var getStatusResult = await result.Consumer.GetStatusAsync();
if ((getLocationResult.Status == AllJoynStatus.Ok) &&
(getStatusResult.Status == AllJoynStatus.Ok))
{
safeBag.Add(new ServiceInfoWithLocation()
{
Location = getLocationResult.Returnvalue,
Consumer = result.Consumer,
IsOn = getStatusResult.Returnvalue
});
}
}
};
this.watcher.Start();
try
{
await Task.Delay(timeoutMs, cancelledToken);
list = new List<ServiceInfoWithLocation>(safeBag.ToArray());
}
catch (TaskCanceledException)
{
// the list remains null, we got cancelled.
}
return (list);
}
void StopWatcherAndBusAttachment(List<ServiceInfoWithLocation> serviceInfoList)
{
if (serviceInfoList != null)
{
foreach (var item in serviceInfoList)
{
item.Consumer.Dispose();
}
}
if (this.watcher != null)
{
this.watcher.Stop();
this.watcher.Dispose();
this.busAttachment.Disconnect();
}
}
lightbulbWatcher watcher;
AllJoynBusAttachment busAttachment;
}
}
and, clearly, some of this (particularly the message dialogs back to Cortana) could be encapsulated into some additional classes to make the code better all round but this was coded somewhat ‘on the hoof’.
I found it an interesting thing to experiment with though and I could definitely see this sort of idea being built into apps such that I can control devices around the house through home automation.
I’m fairly certain that companies like Insteon are doing work with both AllJoyn and Cortana so I’m sure it’s something that we’re going to see a lot more of.
If you want the code for all of this (warts and all!) then download from here.
Mike Taulty 