Following up from my previous post, I wanted to experiment a little more with the object model offered by the RealSense SDK. While I’d managed to get some video onto the screen in the previous post via the PXCMSenseManager class, I hadn’t really felt that I’d known what it was doing for me.
With that in mind, I went back to writing a simple console application to see if I could spend a little more time on some of the classes being used below the PXCMSenseManager without actually bringing that into play and I moved on to the PXCMSession object which the SDK tells me is the place to start if I want to get some configured I/O modules up and running.
I started off by creating one which didn’t feel too challenging;
using (PXCMSession session = PXCMSession.CreateInstance())
{
}
and once I’ve got hold of a session like this, it becomes a form of ‘container’ which can be queried for the modules it contains and then can be asked to instantiate those modules. It feels a little like a specific form of IoC container but I don’t think the analogy is a perfect one.
I have to be honest and say that I found the method by which you execute queries in this SDK to be a bit tedious. It’s not very ‘modern .NET’ in that there’s not a set of enumerables or queryables but, instead, there are APIs that feel to me very much like old-school Windows APIs that do enumeration (like EnumWindows, EnumPrinters type APIs). That is, the query APIs take the form of;
- (optionally) define some descriptor of what you're looking to find – e.g. which type of module you want
- call some Query function with that descriptor and an index 0,1,2,3…and passing a reference parameter to capture the result
- wait for the Query function to return an error as that will let you know that you've hit the end of the enumeration
Additionally, the API functions tend now to throw exceptions, they return a pxcmStatus value which is like an HRESULT and which also means that a lot of API calls take out parameters which makes for longer code. My first attempt at dealing with this was to define these 2 delegates and functions;
delegate pxcmStatus DescQueryFunction<T,U>(T desc, int index, out U returnValue);
delegate pxcmStatus QueryFunction<T>(int index, out T returnValue);
and I defined these functions;
static IEnumerable<T> Enumerate<T>(QueryFunction<T> queryFunction)
{
int i = 0;
T queryResult;
while (queryFunction(i++, out queryResult) == pxcmStatus.PXCM_STATUS_NO_ERROR)
{
yield return queryResult;
}
}
static IEnumerable<U> EnumerateWithDescription<T,U>(T description, DescQueryFunction<T,U> queryFunction)
{
int i = 0;
U queryResult;
while (queryFunction(description, i++, out queryResult) == pxcmStatus.PXCM_STATUS_NO_ERROR)
{
yield return queryResult;
}
}
to try and help in generally calling these types of enumeration functions and, with that in place, I can then write some console code which tries to enumerate all the modules on my installation;
using (PXCMSession session = PXCMSession.CreateInstance())
{
// dump all the modules the session knows about
var description = MakeImplementationDescription(
PXCMSession.ImplGroup.IMPL_GROUP_ANY,
PXCMSession.ImplSubgroup.IMPL_SUBGROUP_ANY);
var modules = EnumerateWithDescription<PXCMSession.ImplDesc, PXCMSession.ImplDesc>(
description, session.QueryImpl);
Console.WriteLine("===All Modules");
foreach (var module in modules)
{
Console.WriteLine(module.friendlyName);
}
}
and on my system I see;
that code makes use of this simple function to create a thing called a PXCMSession.ImplDesc – again, the SDK has some very ‘interesting’ names for types here and it uses a lot of nested types which I don’t find particularly discoverable or even memorable. Regardless, that little utility function is just hiding a constructor call for me;
static PXCMSession.ImplDesc MakeImplementationDescription(
PXCMSession.ImplGroup group,
PXCMSession.ImplSubgroup subgroup)
{
return (
new PXCMSession.ImplDesc()
{
group = group,
subgroup = subgroup
}
);
}
and if I wanted to be more specific and look for modules related to video capture then I believe that I can do it as;
using (PXCMSession session = PXCMSession.CreateInstance())
{
// dump all the modules the session knows about
var description = MakeImplementationDescription(
PXCMSession.ImplGroup.IMPL_GROUP_SENSOR,
PXCMSession.ImplSubgroup.IMPL_SUBGROUP_VIDEO_CAPTURE);
var modules = EnumerateWithDescription<PXCMSession.ImplDesc, PXCMSession.ImplDesc>(
description, session.QueryImpl);
Console.WriteLine("===Video Capture Modules");
foreach (var module in modules)
{
Console.WriteLine(module.friendlyName);
}
}
which tells me;
Once I've got those modules, I can navigate to find a lot more info from them – this larger lump of code;
static void Main(string[] args)
{
// create session
using (PXCMSession session = PXCMSession.CreateInstance())
{
// dump all the modules the session knows about
var description = MakeImplementationDescription(
PXCMSession.ImplGroup.IMPL_GROUP_SENSOR,
PXCMSession.ImplSubgroup.IMPL_SUBGROUP_VIDEO_CAPTURE);
var modules = EnumerateWithDescription<PXCMSession.ImplDesc, PXCMSession.ImplDesc>(
description, session.QueryImpl);
Console.WriteLine("===Video Capture Modules");
foreach (var module in modules)
{
Console.WriteLine("===Module [{0}]", module.friendlyName);
// create one of those
PXCMCapture captureModule;
if (session.CreateImpl<PXCMCapture>(out captureModule) == pxcmStatus.PXCM_STATUS_NO_ERROR)
{
Console.WriteLine(" ===Devices");
var deviceInfos = Enumerate<PXCMCapture.DeviceInfo>(captureModule.QueryDeviceInfo);
foreach (var deviceInfo in deviceInfos)
{
Console.WriteLine(" [{0}]", deviceInfo.name);
Console.WriteLine(" [{0}]", deviceInfo.model);
Console.WriteLine(" [{0}]", deviceInfo.location);
Console.WriteLine(" [{0}]", deviceInfo.orientation);
Console.WriteLine(" [{0}]", deviceInfo.streams);
using (var device = captureModule.CreateDevice(deviceInfo.didx))
{
for (int i = 0; i < PXCMCapture.STREAM_LIMIT; i++)
{
PXCMCapture.StreamType type = PXCMCapture.StreamTypeFromIndex(i);
Console.WriteLine(" ===PROFILES FOR TYPE [{0}]", type);
var profiles = EnumerateWithDescription<PXCMCapture.StreamType, PXCMCapture.Device.StreamProfileSet>(
type,
device.QueryStreamProfileSet);
foreach (var profile in profiles)
{
var relevantSet = profile[type];
Console.WriteLine(
" PROFILE: [{0}], [{1}], [{2},{3}]",
relevantSet.frameRate.max,
relevantSet.imageInfo.format,
relevantSet.imageInfo.width,
relevantSet.imageInfo.height);
}
}
}
}
captureModule.Dispose();
}
}
}
}
queries out all video capture models, creates them in turn, figures out any devices associated with them and then creates those devices and asks them what streaming profiles they support.
That gives me a long list of streaming profiles as below;
===Video Capture Modules
===Module [IVCam I/O Proxy (Xc)]
===Devices
[Intel(R) RealSense(TM) 3D Camera]
[DEVICE_MODEL_IVCAM]
[PXCMPointF32]
[DEVICE_ORIENTATION_FRONT_FACING]
[STREAM_TYPE_COLOR, STREAM_TYPE_DEPTH, STREAM_TYPE_IR]
===PROFILES FOR TYPE [STREAM_TYPE_COLOR]
PROFILE: [30], [PIXEL_FORMAT_YUY2], [1920,1080]
PROFILE: [30], [PIXEL_FORMAT_YUY2], [1280,720]
PROFILE: [30], [PIXEL_FORMAT_YUY2], [960,540]
PROFILE: [30], [PIXEL_FORMAT_YUY2], [848,480]
PROFILE: [30], [PIXEL_FORMAT_YUY2], [640,480]
PROFILE: [30], [PIXEL_FORMAT_YUY2], [640,360]
PROFILE: [30], [PIXEL_FORMAT_YUY2], [424,240]
PROFILE: [30], [PIXEL_FORMAT_YUY2], [320,240]
PROFILE: [30], [PIXEL_FORMAT_YUY2], [320,180]
PROFILE: [60], [PIXEL_FORMAT_YUY2], [848,480]
PROFILE: [60], [PIXEL_FORMAT_YUY2], [640,480]
PROFILE: [60], [PIXEL_FORMAT_YUY2], [640,360]
PROFILE: [60], [PIXEL_FORMAT_YUY2], [424,240]
PROFILE: [60], [PIXEL_FORMAT_YUY2], [320,240]
PROFILE: [60], [PIXEL_FORMAT_YUY2], [320,180]
PROFILE: [30], [PIXEL_FORMAT_RGB24], [1920,1080]
PROFILE: [30], [PIXEL_FORMAT_RGB24], [1280,720]
PROFILE: [30], [PIXEL_FORMAT_RGB24], [960,540]
PROFILE: [30], [PIXEL_FORMAT_RGB24], [848,480]
PROFILE: [30], [PIXEL_FORMAT_RGB24], [640,480]
PROFILE: [30], [PIXEL_FORMAT_RGB24], [640,360]
PROFILE: [30], [PIXEL_FORMAT_RGB24], [424,240]
PROFILE: [30], [PIXEL_FORMAT_RGB24], [320,240]
PROFILE: [30], [PIXEL_FORMAT_RGB24], [320,180]
PROFILE: [60], [PIXEL_FORMAT_RGB24], [848,480]
PROFILE: [60], [PIXEL_FORMAT_RGB24], [640,480]
PROFILE: [60], [PIXEL_FORMAT_RGB24], [640,360]
PROFILE: [60], [PIXEL_FORMAT_RGB24], [424,240]
PROFILE: [60], [PIXEL_FORMAT_RGB24], [320,240]
PROFILE: [60], [PIXEL_FORMAT_RGB24], [320,180]
PROFILE: [30], [PIXEL_FORMAT_RGB32], [1920,1080]
PROFILE: [30], [PIXEL_FORMAT_RGB32], [1280,720]
PROFILE: [30], [PIXEL_FORMAT_RGB32], [960,540]
PROFILE: [30], [PIXEL_FORMAT_RGB32], [848,480]
PROFILE: [30], [PIXEL_FORMAT_RGB32], [640,480]
PROFILE: [30], [PIXEL_FORMAT_RGB32], [640,360]
PROFILE: [30], [PIXEL_FORMAT_RGB32], [424,240]
PROFILE: [30], [PIXEL_FORMAT_RGB32], [320,240]
PROFILE: [30], [PIXEL_FORMAT_RGB32], [320,180]
PROFILE: [60], [PIXEL_FORMAT_RGB32], [848,480]
PROFILE: [60], [PIXEL_FORMAT_RGB32], [640,480]
PROFILE: [60], [PIXEL_FORMAT_RGB32], [640,360]
PROFILE: [60], [PIXEL_FORMAT_RGB32], [424,240]
PROFILE: [60], [PIXEL_FORMAT_RGB32], [320,240]
PROFILE: [60], [PIXEL_FORMAT_RGB32], [320,180]
===PROFILES FOR TYPE [STREAM_TYPE_DEPTH]
PROFILE: [30], [PIXEL_FORMAT_DEPTH], [640,480]
PROFILE: [30], [PIXEL_FORMAT_DEPTH], [640,240]
PROFILE: [60], [PIXEL_FORMAT_DEPTH], [640,480]
PROFILE: [60], [PIXEL_FORMAT_DEPTH], [640,240]
PROFILE: [110], [PIXEL_FORMAT_DEPTH], [640,240]
===PROFILES FOR TYPE [STREAM_TYPE_IR]
PROFILE: [30], [PIXEL_FORMAT_Y8], [640,480]
PROFILE: [30], [PIXEL_FORMAT_Y8], [640,240]
PROFILE: [60], [PIXEL_FORMAT_Y8], [640,480]
PROFILE: [60], [PIXEL_FORMAT_Y8], [640,240]
PROFILE: [110], [PIXEL_FORMAT_Y8], [640,240]
PROFILE: [300], [PIXEL_FORMAT_Y8], [640,480]
PROFILE: [200], [PIXEL_FORMAT_Y8_IR_RELATIVE], [640,480]
PROFILE: [100], [PIXEL_FORMAT_Y8_IR_RELATIVE], [640,480]
PROFILE: [60], [PIXEL_FORMAT_Y8_IR_RELATIVE], [640,480]
PROFILE: [30], [PIXEL_FORMAT_Y8_IR_RELATIVE], [640,480]
===PROFILES FOR TYPE [STREAM_TYPE_LEFT]
===PROFILES FOR TYPE [STREAM_TYPE_RIGHT]
===PROFILES FOR TYPE [32]
===PROFILES FOR TYPE [64]
===PROFILES FOR TYPE [128]
===Module [Video Capture (Media Foundation)]
===Devices
[Intel(R) RealSense(TM) 3D Camera]
[DEVICE_MODEL_IVCAM]
[PXCMPointF32]
[DEVICE_ORIENTATION_FRONT_FACING]
[STREAM_TYPE_COLOR, STREAM_TYPE_DEPTH, STREAM_TYPE_IR]
===PROFILES FOR TYPE [STREAM_TYPE_COLOR]
PROFILE: [30], [PIXEL_FORMAT_YUY2], [1920,1080]
PROFILE: [30], [PIXEL_FORMAT_YUY2], [1280,720]
PROFILE: [30], [PIXEL_FORMAT_YUY2], [960,540]
PROFILE: [30], [PIXEL_FORMAT_YUY2], [848,480]
PROFILE: [30], [PIXEL_FORMAT_YUY2], [640,480]
PROFILE: [30], [PIXEL_FORMAT_YUY2], [640,360]
PROFILE: [30], [PIXEL_FORMAT_YUY2], [424,240]
PROFILE: [30], [PIXEL_FORMAT_YUY2], [320,240]
PROFILE: [30], [PIXEL_FORMAT_YUY2], [320,180]
PROFILE: [60], [PIXEL_FORMAT_YUY2], [848,480]
PROFILE: [60], [PIXEL_FORMAT_YUY2], [640,480]
PROFILE: [60], [PIXEL_FORMAT_YUY2], [640,360]
PROFILE: [60], [PIXEL_FORMAT_YUY2], [424,240]
PROFILE: [60], [PIXEL_FORMAT_YUY2], [320,240]
PROFILE: [60], [PIXEL_FORMAT_YUY2], [320,180]
PROFILE: [30], [PIXEL_FORMAT_RGB24], [1920,1080]
PROFILE: [30], [PIXEL_FORMAT_RGB24], [1280,720]
PROFILE: [30], [PIXEL_FORMAT_RGB24], [960,540]
PROFILE: [30], [PIXEL_FORMAT_RGB24], [848,480]
PROFILE: [30], [PIXEL_FORMAT_RGB24], [640,480]
PROFILE: [30], [PIXEL_FORMAT_RGB24], [640,360]
PROFILE: [30], [PIXEL_FORMAT_RGB24], [424,240]
PROFILE: [30], [PIXEL_FORMAT_RGB24], [320,240]
PROFILE: [30], [PIXEL_FORMAT_RGB24], [320,180]
PROFILE: [60], [PIXEL_FORMAT_RGB24], [848,480]
PROFILE: [60], [PIXEL_FORMAT_RGB24], [640,480]
PROFILE: [60], [PIXEL_FORMAT_RGB24], [640,360]
PROFILE: [60], [PIXEL_FORMAT_RGB24], [424,240]
PROFILE: [60], [PIXEL_FORMAT_RGB24], [320,240]
PROFILE: [60], [PIXEL_FORMAT_RGB24], [320,180]
PROFILE: [30], [PIXEL_FORMAT_RGB32], [1920,1080]
PROFILE: [30], [PIXEL_FORMAT_RGB32], [1280,720]
PROFILE: [30], [PIXEL_FORMAT_RGB32], [960,540]
PROFILE: [30], [PIXEL_FORMAT_RGB32], [848,480]
PROFILE: [30], [PIXEL_FORMAT_RGB32], [640,480]
PROFILE: [30], [PIXEL_FORMAT_RGB32], [640,360]
PROFILE: [30], [PIXEL_FORMAT_RGB32], [424,240]
PROFILE: [30], [PIXEL_FORMAT_RGB32], [320,240]
PROFILE: [30], [PIXEL_FORMAT_RGB32], [320,180]
PROFILE: [60], [PIXEL_FORMAT_RGB32], [848,480]
PROFILE: [60], [PIXEL_FORMAT_RGB32], [640,480]
PROFILE: [60], [PIXEL_FORMAT_RGB32], [640,360]
PROFILE: [60], [PIXEL_FORMAT_RGB32], [424,240]
PROFILE: [60], [PIXEL_FORMAT_RGB32], [320,240]
PROFILE: [60], [PIXEL_FORMAT_RGB32], [320,180]
===PROFILES FOR TYPE [STREAM_TYPE_DEPTH]
PROFILE: [30], [PIXEL_FORMAT_DEPTH], [640,480]
PROFILE: [30], [PIXEL_FORMAT_DEPTH], [640,240]
PROFILE: [60], [PIXEL_FORMAT_DEPTH], [640,480]
PROFILE: [60], [PIXEL_FORMAT_DEPTH], [640,240]
PROFILE: [110], [PIXEL_FORMAT_DEPTH], [640,240]
===PROFILES FOR TYPE [STREAM_TYPE_IR]
PROFILE: [30], [PIXEL_FORMAT_Y8], [640,480]
PROFILE: [30], [PIXEL_FORMAT_Y8], [640,240]
PROFILE: [60], [PIXEL_FORMAT_Y8], [640,480]
PROFILE: [60], [PIXEL_FORMAT_Y8], [640,240]
PROFILE: [110], [PIXEL_FORMAT_Y8], [640,240]
PROFILE: [300], [PIXEL_FORMAT_Y8], [640,480]
PROFILE: [200], [PIXEL_FORMAT_Y8_IR_RELATIVE], [640,480]
PROFILE: [100], [PIXEL_FORMAT_Y8_IR_RELATIVE], [640,480]
PROFILE: [60], [PIXEL_FORMAT_Y8_IR_RELATIVE], [640,480]
PROFILE: [30], [PIXEL_FORMAT_Y8_IR_RELATIVE], [640,480]
===PROFILES FOR TYPE [STREAM_TYPE_LEFT]
===PROFILES FOR TYPE [STREAM_TYPE_RIGHT]
===PROFILES FOR TYPE [32]
===PROFILES FOR TYPE [64]
===PROFILES FOR TYPE [128]
I said it was a long list! It’s worth saying that these profiles can be combined so it’s possible to ask the SDK to return COLOR, IR and DEPTH at the same time but not every combination of every profile is going to work and the object model has a method that can be called to ask whether a mix of profiles is valid or not.
Once I have created the device, as in the code above, I can start to acquire frames of data from it and I can also query/set a tonne of properties on it such as;
device.QueryMirrorMode(); device.QueryColorAutoExposure(); device.QueryColorAutoWhiteBalance();
and there are many, many more of these each with a corresponding Set method.
An important one of these Set methods seems to be the SetStreamProfileSet method which appears to tell the device what streaming profiles I'm interested in using to read streamed data. I find that if I don't set this then on my device any attempt to read streaming data throws a memory allocation exception and I suspect that might be because it's trying to allocate memory for every different kind of stream available and, as you can see, it’s a long list so if that’s what it’s doing then I’m not surprised it runs out of memory.
To take that extra step and to try and acquire frames from the device, I could change the code to do less enumerating and just accept the first module/device/profile that it comes across and then try to use it to grab data as;
static void Main(string[] args)
{
// create session
using (PXCMSession session = PXCMSession.CreateInstance())
{
// get the first video capture module descriptor.
var captureModuleDesc =
EnumerateWithDescription<PXCMSession.ImplDesc, PXCMSession.ImplDesc>(
MakeImplementationDescription(PXCMSession.ImplGroup.IMPL_GROUP_SENSOR, PXCMSession.ImplSubgroup.IMPL_SUBGROUP_VIDEO_CAPTURE),
session.QueryImpl)
.First();
PXCMCapture captureModule;
// create an instance
if (session.CreateImpl<PXCMCapture>(out captureModule) == pxcmStatus.PXCM_STATUS_NO_ERROR)
{
// get the first device info
var deviceInfo = Enumerate<PXCMCapture.DeviceInfo>(captureModule.QueryDeviceInfo).First();
// create an instance
using (var device = captureModule.CreateDevice(deviceInfo.didx))
{
// get the first stream profile for color.
PXCMCapture.Device.StreamProfileSet streamProfileSet =
EnumerateWithDescription<PXCMCapture.StreamType, PXCMCapture.Device.StreamProfileSet>(
PXCMCapture.StreamType.STREAM_TYPE_COLOR,
device.QueryStreamProfileSet)
.First();
Console.WriteLine("Using stream profile for [{0}Hz], [{1}], [{2} x {3}]",
streamProfileSet.color.frameRate,
streamProfileSet.color.imageInfo.format,
streamProfileSet.color.imageInfo.width,
streamProfileSet.color.imageInfo.height);
// filter the device to use that profile set.
if (device.SetStreamProfileSet(streamProfileSet) == pxcmStatus.PXCM_STATUS_NO_ERROR)
{
PXCMCapture.Sample sample = new PXCMCapture.Sample();
// capture 10 frames from the device.
for (int i = 0; i < 10; i++)
{
// note, this blocks until it gets a frame...
if (device.ReadStreams(PXCMCapture.StreamType.STREAM_TYPE_COLOR, sample) == pxcmStatus.PXCM_STATUS_NO_ERROR)
{
Console.WriteLine("Read frame [{0}] from device", i);
PXCMImage.ImageData imageData;
if (sample.color.AcquireAccess(PXCMImage.Access.ACCESS_READ, out imageData) == pxcmStatus.PXCM_STATUS_NO_ERROR)
{
Console.WriteLine("Image is in format [{0}]", imageData.format);
sample.color.ReleaseAccess(imageData);
}
}
}
}
}
captureModule.Dispose();
}
}
}
and, sure enough, this does seem to capture 10 frames of COLOR data for me;
I learned quite a bit from playing around with the object model here around modules/devices/profiles. I don’t think I have it entirely right in my head at the time of writing but, having taken this step, I then wanted to put a UI back onto it to explore further. I’ll write that up in the next post.
Mike Taulty 