Technology Soup: Fun with Windows 8 Metro apps, WinRT, ESE, C++/CX, .NET and JavaScript

Posted on by

Ok, I just wanted to see how many technologies I could get into the title of a blog post 😉

This one really falls into the category of ‘just for fun’.

I wanted to experiment with writing a custom WinRT component in C++/CX and then try using it from .NET and JavaScript Metro style apps as I haven’t really had a chance to try out anything much in C++/CX just yet although I do have an old (and rusting) background in C++ and I’ve also written some C++/CLI in my time and I believe there’s commonalities between those two things even though they serve very different purposes (i.e. CX == native code, CLI == managed code).

You can find more about this kind of custom component writing on the dev centre for Windows 8 Metro style apps along with information about C++/CX;

 

I settled on trying to write a small part of a wrapper for the Extensible Storage Engine (ESE or Jet engine) that’s available to Windows 8 Metro style apps for storage;

Win32 and COM for Metro style apps

I daresay that there’s a WinRT wrapper for this out there already and there’s certainly a .NET wrapper for it;

http://managedesent.codeplex.com/

but I believe (i.e. haven’t looked) that wrapper goes beyond the APIs that are approved for Windows 8 Metro style apps so you might need to take a little care there.

Why play with ESE? It’s a local, file-based database technology and one of the areas that’s “interesting” about Metro style apps is structured, local storage and how that differs depending on the implementation technology you choose. That is;

 

I thought it might be interesting to see if I could take ESE, wrap it up as a custom WinRT component and then make use of it in the same way from all three of those environments.

There were three challenges there – the main ones being lack of knowledge on my part;

 

  1. I don’t really know anything about ESE and its API.
  2. I don’t really know anything about writing a custom WinRT component in C++/CX.
  3. You get into some choices around how to represent data with types because (e.g.) the JavaScript built-in types are a subset of what you’d see in .NET or native code.

But I stumbled along and this is why this post is for fun rather than for anything else. I actually found it fairly easy and most of the challenges I’ve encountered so far were around (1) rather than (2) although I have battled a little here and there to figure out what types I should use to pass across the Application Binary Interface (ABI) between a WinRT component and its consumer.

For simplicity, I arbitrarily decided that an app would need 1 and only 1 ESE database and that it wouldn’t want to explicitly name the files itself so I came up with this simple Store class to represent that database;

namespace JetStructuredStorage
{
	public ref class Store sealed 
    {
    public:
		static bool Open();
		static property TableSet^ Tables
		{
			TableSet^ get();
		}
		static void Close();

	private:
    };
}

Note that I’m omitting the private bits of the class and the implementation. What’s “interesting” about this class is that it’s a ref class and it’s sealed and that it gets compiled up to form a custom WinRT component. I can then reference this component from a [C++/.NET/JavaScript] Windows 8 Metro style app project.

If you’re a C++ programmer and pondering what ref class and TableSet^ might mean then, to my mind, ref class means WinRT component and TableSet is another of my ref classes and ^ means “handle” to which is like an opaque pointer. Beyond that, the C++/CX docs need a reading. ref classes need newing up with ref new and are reference counted.

The intention of the Open method is to open/create a DB and it returns true if it created the DB because it wasn’t already there for you. I can use it like this from JavaScript;

                   var jetStoreNs = JetStructuredStorage;

                        if (jetStoreNs.Store.open())
                        {
                            // DB created.
                        }
                        jetStoreNs.Store.close();

and from .NET (depends on a using JetStructuredStorage; statement);

          if (Store.Open())
          {
            // DB created.
          }
          Store.Close();

and from C++;

	if (JetStructuredStorage::Store::Open())
	{
		// DB created.
	}
	JetStructuredStorage::Store::Close();

Note that I didn’t use a using namespace for my C++ because it left me with an ambiguity around the name Store so I left it fully qualified. Note also that in a C++ Metro style app project adding a reference to another project which contains a WinRT component is a little different from in a JS or .NET project. The dialog just looks a bit different;

image

but other than that the experience is the common one.

Opening and closing a database is all well and good but there comes a time when you need to play around with tables and that’s where my TableSet class comes into play. It looks like this;

namespace JetStructuredStorage
{
	ref class Cursor;

	public ref class TableSet sealed
	{
	public:
		void Add(String^ tableName, IVector<ColumnDefinition>^ columns);
		void Remove(String^ tableName);		
		Cursor^ OpenCursor(String^ tableName);

	internal:
		TableSet(JET_SESID sessionId, JET_DBID dbId);
		virtual ~TableSet();

	private:
	};
};

Note that I left the internal parts visible in this class definition because it shows having the constructor in question public to other code inside my compilation unit but not visible for an external consumer of the component. This is like internal in .NET. Note that the ref class Cursor is just a forward declaration for another class called Cursor which I’ll come back to.

This class serves 3 purposes. Tables can be added and removed via the Add/Remove methods and a Cursor can be opened on a table via the OpenCursor method. This is a bit clunky and really points to the fact that I haven’t yet figured out how to make an indexer available from one of these components so code ends up looking like;

Store.tables.openCursor(“customers”);

Rather than;

Stores.tables[“customers”].openCursor();

In order to add a table, there’s a need to provide a name for the new table along with a basic schema and this caused me to try and figure out how to pass a list of table columns into a custom WinRT component and the mechanism I settled on was to use a IVector<T>.

My class ColumnDefinition which I populate that IVector with looks like this;

namespace JetStructuredStorage
{
	public value class ColumnDefinition 
	{
	public:
		Platform::String^ Name;
		ColumnType Type;
		IndexType IndexType;
		int MaxLength;
		bool AutoIncrement;
		bool NotNull;
	};
};

and I used a value class (http://msdn.microsoft.com/en-us/library/windows/apps/hh699870(v=vs.110).aspx) here because I was trying to end up with a situation in JavaScript where a consumer of the API could just pass an object literal rather than having to new up a specific type of object from my component library. I think a value struct would have been the same.

That worked ok but I learned a useful lesson along the way which I’ll come back to in a moment.

I have fairly limited column types and index types which I define as;

namespace JetStructuredStorage
{
	public enum class ColumnType 
	{
		colTypeBoolean = JET_coltypShort,
		colTypeInteger = JET_coltypLong,
		colTypeSingle = JET_coltypIEEESingle,
		colTypeDouble = JET_coltypIEEEDouble,
		colTypeText = JET_coltypText
	};
};
namespace JetStructuredStorage
{
	public enum class IndexType
	{
		None = 0,
		SingleColumnPrimaryKey = 1,
		SingleColumnIndexed = 2
	};
};

And so I’m making no attempt to fully pass on the ESE data types here and there’d be a lot of room for trying to figure out how to ‘do the right thing’ around various data-types like Guids and so on that I’ve chosen to ignore for the moment.

However, with that in place I can create a simple table – here in JavaScript;

                        var jetStoreNs = JetStructuredStorage;

                        if (jetStoreNs.Store.open())
                        {
                            jetStoreNs.Store.tables.add("contacts",
                                [
                                    {
                                        name: 'id',
                                        type: JetStructuredStorage.ColumnType.colTypeInteger,
                                        indexType: JetStructuredStorage.IndexType.singleColumnPrimaryKey,
                                        maxLength: 0,
                                        autoIncrement: true,
                                        notNull: false
                                    },
                                    {
                                        name: 'firstName',
                                        type: JetStructuredStorage.ColumnType.colTypeText,
                                        indexType: JetStructuredStorage.IndexType.singleColumnIndexed,
                                        maxLength: 100,
                                        autoIncrement: false,
                                        notNull: false
                                    },
                                    {
                                        name: 'lastName',
                                        type: JetStructuredStorage.ColumnType.colTypeText,
                                        indexType: JetStructuredStorage.IndexType.none,
                                        maxLength: 100,
                                        autoIncrement: false,
                                        notNull: false
                                    }
                                ]
                            );
                        }
                        jetStoreNs.Store.close();

Here I learned an important lesson (one of those ‘three hours to learn’ style lessons :-)) in that I needed to ensure that every field of my ColumnDefinition was filled in via the object-initialisation syntax otherwise I hit exceptions. This is by design and is stated on MSDN (http://msdn.microsoft.com/en-us/library/hh710232(v=vs.94).aspx) which says;

A Windows Runtime Structure is a heterogeneous collection of named data fields. In JavaScript, a structure is represented as a JavaScript object that contains a named property for each field in the structure. If the conversion of any structure value fails, a marshaling error is returned. Fields in the JavaScript object that do not have equivalents in the Windows Runtime structure are ignored. Note: A Windows Runtime structure cannot be instantiated in JavaScript with the new keyword.

Here’s that same table creation code in .NET; 

    if (Store.Open())
          {
           Store.Tables.Add("contacts",
              new List<JetStructuredStorage.ColumnDefinition>() {
                new JetStructuredStorage.ColumnDefinition()
                {
                  Name = "id",
                  Type = JetStructuredStorage.ColumnType.colTypeInteger,
                  AutoIncrement = true,
                  IndexType = JetStructuredStorage.IndexType.SingleColumnPrimaryKey,
                  NotNull = true
                },
                new JetStructuredStorage.ColumnDefinition()
                {
                  Name = "firstName",
                  Type = JetStructuredStorage.ColumnType.colTypeText,
                  AutoIncrement = false,
                  IndexType = JetStructuredStorage.IndexType.SingleColumnIndexed,
                  NotNull = true
                },
                new JetStructuredStorage.ColumnDefinition()
                {
                  Name = "lastName",
                  Type = JetStructuredStorage.ColumnType.colTypeText,
                  AutoIncrement = false,
                  IndexType = JetStructuredStorage.IndexType.None,
                  NotNull = true
                }
              }
            );
          }
          Store.Close();

and in C++;

	if (JetStructuredStorage::Store::Open())
	{
		JetStructuredStorage::ColumnDefinition idCol;
		idCol.Name = L"id";
		idCol.Type = JetStructuredStorage::ColumnType::colTypeInteger;
		idCol.AutoIncrement = true;
		idCol.IndexType = JetStructuredStorage::IndexType::SingleColumnPrimaryKey;
		idCol.NotNull = true;

		JetStructuredStorage::ColumnDefinition firstNameCol;
		firstNameCol.Name = L"firstName";
		firstNameCol.Type = JetStructuredStorage::ColumnType::colTypeText;
		firstNameCol.AutoIncrement = false;
		firstNameCol.MaxLength = 100;
		firstNameCol.IndexType = JetStructuredStorage::IndexType::SingleColumnIndexed;
		firstNameCol.NotNull = false;

		JetStructuredStorage::ColumnDefinition lastNameCol;
		lastNameCol.Name = L"lastName";
		lastNameCol.Type = JetStructuredStorage::ColumnType::colTypeText;
		lastNameCol.AutoIncrement = false;
		lastNameCol.MaxLength = 100;
		lastNameCol.IndexType = JetStructuredStorage::IndexType::None;
		lastNameCol.NotNull = false;

		auto columns = ref new Platform::Collections::Vector<JetStructuredStorage::ColumnDefinition, myEqual>();

		columns->Append(idCol);
		columns->Append(firstNameCol);
		columns->Append(lastNameCol);

		JetStructuredStorage::Store::Tables->Add("contacts", columns);
	}
	JetStructuredStorage::Store::Close();

Note that in order to use Platform::Collections::Vector<T> on my type I had to provide an equality comparer because that class needs one even though I don’t think ( i.e. I hope 🙂 ) in my circumstance it’s never going to actually get used. I cobbled together a fake comparer; 

struct myEqual : 
	std::binary_function<JetStructuredStorage::ColumnDefinition, JetStructuredStorage::ColumnDefinition, bool>
{
	bool operator()(const JetStructuredStorage::ColumnDefinition& x, const JetStructuredStorage::ColumnDefinition& y)
	{
		return(true);
	};
};

When an index is requested on a column named C, I take a short-cut and always call that index C_index rather than making the calling code explicitly name the index although that’d be easy enough to add. Multi-column indexes would be another addition.

Having added a table, I could remove it if I didn’t like the look of it using JavaScript; 

                     var jetStoreNs = JetStructuredStorage;

                        jetStoreNs.Store.open();
                        jetStoreNs.Store.tables.remove("contacts");
                        jetStoreNs.Store.close();

or .NET; 

          Store.Open();
          Store.Tables.Remove("contacts");
          Store.Close();

or C++; 

	JetStructuredStorage::Store::Open();
	JetStructuredStorage::Store::Tables->Remove(L"contacts");
	JetStructuredStorage::Store::Close();

And if I re-create the table to have the same little schema that I use previous [id (pkey), firstName (indexed), lastName] then I can use the OpenCursor method previously discussed to get hold of a Cursor which is defined as;

namespace JetStructuredStorage
{
	public ref class Cursor sealed
	{
	public:
		void SetIndexFilter(String^ indexName, Object^ indexValue);
		void ClearIndexFilter();
		property bool HasCurrent { bool get(); };
		PropertySet^ GetCurrent();
		bool MoveFirst();
		bool MoveLast();
		bool MovePrevious();
		bool MoveNext();
		void Remove();
		void Insert(PropertySet^ properties);
		void Close();
};

And then use that to insert some data. For me, the “interesting” thing here was in how to pass a “record” of data across the boundary into the different languages given that the structure of that record is dynamic. It calls for some kind of dictionary <string,object> and I used the built-in PropertySet type to do that for me. I hadn’t really looked at PropertySet very much until this point but I had spotted previously that it shows up in places like your application’s settings data.

When you’ve got a PropertySet on the native side of the boundary you can look enumerate its members and try and figure out what types they are. Here’s a piece of code from my implementation which does some of that;

void Cursor::Insert(PropertySet^ properties)
{

	try
	{
		auto first = properties->First();

		while (first->HasCurrent)
		{
			String^ key = first->Current->Key;
			IPropertyValue^ value = safe_cast<IPropertyValue^>(first->Current->Value);
			PropertyType valueType = value->Type;

And you can see that I can grab a Key/Value pair as in most dictionary types and that the value is represented as an IPropertyValue and one thing that allows me to do is try to figure the PropertyType.

I’m not certain that trying to infer the type of the value is really the right way to go but it’s ok if your caller sticks to simple types (number, string, Boolean) but I suspect it’d hit its limits pretty soon if I tried to make the library more realistic and deal with DateTimes and so on.

That code fragment above comes from the implementation of my Cursor::Insert method and I can populate a table using Insert as this example JavaScript shows; 

                        var jetStoreNs = JetStructuredStorage;                        
                        var firstNames = [ 'jim', 'bob', 'sue', 'tom', 'don', 'liz', 'tim'];
                        var lastNames = ['smith', 'brown', 'jones', 'williams', 'jackson'];
                        var i = 0;
                        var cursor;
                        var index = 0;
                        var ps;

                        jetStoreNs.Store.open();

                        cursor = jetStoreNs.Store.tables.openCursor("contacts");
                        

                        for (i = 0; i < 1000; i++)
                        {                            
                            ps = new Windows.Foundation.Collections.PropertySet();
                            index = Math.floor(Math.random() * (firstNames.length));
                            ps['firstName'] = firstNames[index];
                            index = Math.floor(Math.random() * (lastNames.length));
                            ps['lastName'] = lastNames[index];
                            cursor.insert(ps);
                        }
                        cursor.close();

                        jetStoreNs.Store.close();

It’s a bit tedious to have to manually create a PropertySet so I guess it might be nice to have some convenience function that turned an object into a PropertySet. Maybe something along the lines of; 

    WinJS.Namespace.define(
        "JetUtility",
        {
            toPropertySet: function (obj)
            {
                var ps = new Windows.Foundation.Collections.PropertySet();

                Object.getOwnPropertyNames(obj).forEach(
                    function (field)
                    {
                        ps[field] = obj[field];
                    }
                );
                return (ps);
            }
        }
    );

And then that can be used in the previous loop; 

cursor = jetStoreNs.Store.tables.openCursor("contacts");
                        for (i = 0; i < 1000; i++)
                        {                            
                            fIndex = Math.floor(Math.random() * (firstNames.length));
                            lIndex = Math.floor(Math.random() * (lastNames.length));
                            ps = JetUtility.toPropertySet(
                                    {
                                        firstName: firstNames[fIndex],
                                        lastName: lastNames[lIndex]
                                    }
                                );

                            cursor.insert(ps);
                        }
                        cursor.close();

Doing the same set of random inserts with .NET code looks like; 

          var firstNames = new string[] { "jim", "bob", "sue", "tom", "don", "liz", "tim" };
          var lastNames = new string[] { "smith", "brown", "jones", "williams", "jackson" };
          var random = new Random();

          Store.Open();

          var cursor = Store.Tables.OpenCursor("contacts");

          for (int i = 0; i < 1000; i++)
          {
            PropertySet ps = new PropertySet();
            ps["firstName"] = firstNames[random.Next(0, firstNames.Length - 1)];
            ps["lastName"] = lastNames[random.Next(0, lastNames.Length - 1)];
            cursor.Insert(ps);
          }
Cursor.close();
          Store.Close();

Again, it’s a bit tedious to have to populate a PropertySet from an object so maybe there’s some shortcut way to do that like add an extension method to PropertySet or similar. I could put together something like the function below (note that I’m not suggestion an extension to object here is the right thing to do and nor perhaps is to use DeclaredProperties like this) – it’s just an example; 

    public static class PropertySetExtensions
    {
      public static PropertySet ToPropertySet(this object o)
      {
        PropertySet ps = new PropertySet();
        TypeInfo typeInfo = IntrospectionExtensions.GetTypeInfo(o.GetType());
        
        // this choice of using DeclardProperties is somewhat arbitrary to say
        // the least.
        foreach (var propertyInfo in typeInfo.DeclaredProperties)
        {
          ps[propertyInfo.Name] = propertyInfo.GetValue(o);
        }
        return (ps);
      }
    }

And then I can rework the code in my loop to something like;

  for (int i = 0; i < 1000; i++)
          {
            var data = new {
              FirstName = firstNames[random.Next(0, firstNames.Length - 1)],
              LastName = lastNames[random.Next(0, lastNames.Length - 1)]
            };
            cursor.Insert(data.ToPropertySet());
          }

The insert code in C++ looks like; 

JetStructuredStorage::Store::Open();
	
	String^ firstNames[] = { L"jim", L"bob", L"sue", L"tom", L"don", L"liz", L"tim" };
	String^ lastNames[] = { L"smith", L"brown", L"jones", L"williams", L"jackson" };

	auto cursor = JetStructuredStorage::Store::Tables->OpenCursor(L"contacts");

	for (int i = 0; i < 1000; i++)
	{
		PropertySet^ ps = ref new PropertySet();
		ps->Insert(L"firstName", firstNames[rand() % ARRAY_SIZE(firstNames)]);
		ps->Insert(L"lastName", lastNames[rand() % ARRAY_SIZE(lastNames)]);
		cursor->Insert(ps);
	}
	cursor->Close();

	JetStructuredStorage::Store::Close();

When it comes to reading back the records that I’ve written, in JavaScript code I can do something like; 

                      jetStoreNs.Store.open();
                        cursor = jetStoreNs.Store.tables.openCursor("contacts");

                        while (cursor.hasCurrent)
                        {
                            ps = cursor.getCurrent();
                            console.log("Record " + ps['id'] + ':' + ps['firstName'] + ',' + ps['lastName']);
                            cursor.moveNext();
                        }

                        cursor.close();
                        jetStoreNs.Store.close();

and the same thing in .NET; 

          Store.Open();

          var cursor = Store.Tables.OpenCursor("contacts");

          while (cursor.HasCurrent)
          {
            var ps = cursor.GetCurrent();
            Debug.WriteLine("{0}:{1},{2}", ps["id"], ps["firstName"], ps["lastName"]);
            cursor.MoveNext();
          }

         Store.Close();

And the same thing in C++ where I’m not 100% sure about my safe_casts(); 

	wchar_t debugBuffer[1024];
	JetStructuredStorage::Store::Open();
	
	auto cursor = JetStructuredStorage::Store::Tables->OpenCursor(L"contacts");

	while (cursor->HasCurrent)
	{
		auto ps = cursor->GetCurrent();

		swprintf_s(debugBuffer, 
			sizeof(debugBuffer) / sizeof(wchar_t),
			L"%d: %s,%s\n", 
			safe_cast<int>(ps->Lookup(L"id")),
			(safe_cast<String^>(ps->Lookup(L"firstName"))->Data()),
			(safe_cast<String^>(ps->Lookup(L"lastName"))->Data())
		);

		::OutputDebugStringW(debugBuffer);

		cursor->MoveNext();
	}
	cursor->Close();

	JetStructuredStorage::Store::Close();

When it comes to finding data, I can narrow down the index and the key value that my cursor is using on that index. So, to search for all people with a first name of ‘sue’ I can try and restrict my cursor on the firstName_index. This is JavaScript; 

     jetStoreNs.Store.open();
                        cursor = jetStoreNs.Store.tables.openCursor("contacts");

                        cursor.setIndexFilter("firstName_index", "sue");

                        while (cursor.hasCurrent)
                        {
                            ps = cursor.getCurrent();
                            console.log("Record " + ps['id'] + ':' + ps['firstName'] + ',' + ps['lastName']);
                            cursor.moveNext();
                        }
                        cursor.clearIndexFilter();
                        cursor.close();
                        jetStoreNs.Store.close();

and it’d be the same API call in both .NET and C++.

There’s a bit of a TODO: left around updates at the moment – I haven’t done that bit although it wouldn’t be rocket science to add it into the Cursor class that I’ve got. I did add Remove and that would let me clear out a table as below (JavaScript);

      var jetStoreNs = JetStructuredStorage;                        
                        var cursor; 
                        
                        jetStoreNs.Store.open();
                        cursor = jetStoreNs.Store.tables.openCursor("contacts");
                        cursor.moveLast();
                        while (cursor.hasCurrent)
                        {
                            cursor.remove();
                        }                       
                        cursor.close();
                        jetStoreNs.Store.close();

That’s about all I have at the time of writing – there’s a whole bunch of work I could do here – some obvious things might be;

 

  1. Implement Update 🙂
  2. Do a much better job about implementing various data-types like binary types and GUIDs and DateTimes and so on.
  3. Do a similar thing around NULLs.
  4. Perhaps do something to move the API calls to a separate thread because this API is not likely to lead to a very ‘fast and fluid’ experience as everything is synchronous.
  5. Provide a better programming model around tables/cursors and so on.
  6. Provide a better system for dealing with errors – at the moment every error shows up as the same exception which makes for any debugging without the source code a non-starter.

And no doubt expose more of what the ESE can do for you as a native store.

With all those caveats though, I’ve enjoyed playing with wrapping up these bits of ESE as a WinRT component and it’s taught me quite a lot about writing those components and then using them from the different environments.

I’ve also really enjoyed working with C++ in Visual Studio 2012. I’ve only been working with a tiny set of source files so it’s hard to know whether the goodness I see would scale if you were working on large code-bases but if it does then I think the C++ team have made massive steps forward in terms of IntelliSense and so on.

I’ve provided the source for the my sketchy ESE wrapper here for download. It’s nowhere near finished so apply a very large pinch of salt to what you see.

I may take it further if I get a little time to spend on it but if someone wants to take it and make a proper, full WinRT job of it (or wants to point me at another WinRT wrapper for ESE) then that’d be great. I’d love to see a wrapper that makes the DB available in all coding environments.

Enjoy 🙂