Build Day 3 – In With the New: Microsoft Edge, More HoloLens, Azure Logic Apps, and Direct2D

[This post is by Jeff Mlakar, a member of the Business Intelligence Team at Bennett Adelson.  Follow us @BIatBA and @JeffMlakar]

This was my third and final day at the Microsoft Build Conference. My first was almost all HoloLens. My second was all data. I tried to find a theme in the sessions I saw today. Best I could come up with was that they were all dealing with something new. Of course that could also be said for any collection of sessions at this conference, but anyway… Session 1:

The Microsoft Edge Rendering Engine That Makes the Web Just Work

Presented by David Catuhe and David Rousset.

I gotta say, on Wednesday it was pretty exciting to be in the room for a name announcement. And yes, we have our new browser name: Microsoft Edge, with a logo that looks not far off from IE. Which has to make me wonder: IE has a strong presence within the enterprise, right? A lot of companies have old versions of IE as part of their approved software. It’s probably not going to be going away from the enterprise any time soon. A lot of employees have their old IE versions for business applications and then install a newer/other browser for going out to the web. So what will happen if these employees start to choose Edge as their other browser? Will they now have two different browsers of near-identical logos in their work computer taskbar? Gotta wonder. Anyway, on to the session.

The Davids talked about the history of IE, with its trident engine and many document modes. They talked about how Edge’s new engine, a fork of trident called EdgeHTML, has one document mode and currently benchmarks better than Chrome or Firefox. A lot of the session was on WebGL performance and gaming in the browser, with built-in game pad support and all. This led to an impressive spooky-scene game demo in Edge where the biggest cheer was for a gravestone reading “RIP IE 6”.

One interesting thing was the discussion of IE on Windows Phones currently having issues with mobile displays as some sites check for Android or iOS as a criterion for displaying mobile. Edge corrects this by almost faking as Android or iOS:

Jeff Mlaker 1

The Davids stressed the use of CSS filters. And feature detection, which hopefully developers are doing already. Edge seems to be catching up a couple years to the other browsers in the use of @supports to do feature detection in CSS.

All in all, Edge looks very promising. I’m looking forward to learning more specifics about its features, seeing how it will work for business applications, and trying out its dev tools.

Case Studies of HoloLens App Development

This was a panel discussion where we got to text questions to people who have worked on HoloLens projects already. It was moderated by Rukari Austin, a Microsoft Community Manager for HoloLens. The panel included Dr. Jeff Norris of NASA, Aviad Almagor of Trimble Navigation, Professor Mark Griswold of Case Western Reserve University in Cleveland, and Microsoft Studios Manager Megan Saunders. You’ve probably already seen the videos of the projects these panelists have been working on, so I won’t elaborate on their work. If you haven’t seen the videos, I’d highly recommend you see them. I’ll mostly go through and outline some of the points that resonated with me as I listened to the panelists. These are somewhat paraphrased and could be out of order, but I’ll stick to the spirit of what the speaker was saying. This is not complete, so I’ll urge you to watch the video.

Jeff Mlaker 2

(from left: Rukari Austin, Jeff Norris, Aviad Almagor, Mark Griswold, and Megan Saunders)

Jeff described how HoloLens is helping the two main initiatives of NASA: to explore and to share the experiences of that exploration. He noted that when first developing for such a new technology, all his instincts were wrong. Megan elaborated on the issues of defining a new kind of UI and UX. Aviad said that in this new world, a proper user experience is critical. Mark postulated that as we develop for the technology, we’re probably going to start out making it look like a traditional computer, but we’re not going to end there.

Mark’s background is in building MRIs. In addition to his physics, radiology, and engineering background, he had programming experience and had worked with Unity. He talked about how unsure he was of how long it would take to get set up and working with a new technology like HoloLens. Installing SDKs, configuring, troubleshooting, doing the “hard math” of 3D graphics transformations, etc. He decided to time how long it would take him to go from first sitting down, to making a hologram, to viewing it in HoloLens. It was less than 5 minutes. Which is amazing. Jeff described an experience of intuitively wanting to move his mouse off the screen to interact with the hologram. He suggested that to a member of the team, who implemented it in an hour or two, and he now has mouse interaction with the holograms. So ease of development in the Windows Holographic platform is a big talking point.

All speakers talked of the importance of collaborating via holograms. Jeff said we take for granted how important physical co-presence is. Mark described it as the only way he can educate students, and he can see applications for museums and other such institutions coming out of the technology. Mark also reminded us of the importance of the spatial sound feature of the HoloLens for further immersing you in the augmented reality.

Aviad discussed the inherent human difficulty in translating 2D images into imagined 3D. For a device to do that is huge for “real architects”, as he said (not “system architects” or such). I wonder: if Microsoft is working with someone from Trimble on HoloLens, and Trimble bought SketchUp from Google, will we see SketchUp integrating with HoloLens at some point? I’m just wondering.

No one could answer the big question of when we’ll be able to get our hands on a HoloLens with which to develop. Or when we might see an emulator or such. It’s interesting to me that as this new technology comes out, our gaze is not on how it works but on how we can develop and expand on it.

After the session, we got a chance for one-on-one questions with the panelists. I’m surprised at how the choice of platform is always Unity for their applications. So basically, if you want to get ready for HoloLens development, learn Unity. (http://unity3d.com/)

Logic Apps

Presented by Ilya Grebnov and Stephen Siciliano.

Logic Apps are a part of the relatively new Azure App Service. They enable a developer to create business process workflows in a designer. Business processes are thus automated as a SaaS service. In this session, Ilya and Stephen gave us an introduction to the concept and showed us how to lay out our workflows in a designer, and how to code them.

They talked about all key app services currently available. I learned a new acronym: IPaaS, where the I is for Integration. Ilya and Stephen also taught me about Azure Pack, basically a “mini Azure” that can run on prem. In this way, Logic Apps could run on premises.

Over the course of the demo, they defined Logic Apps in the designer and through code. It’s nice that the Logic Apps can be completely defined in JSON. This means that whatever you can do with Logic Apps, you can configure via the RESTful API of the Logic App.

There are a handful of components in Logic Apps. First Triggers, which might be a bit of a misnomer. Because let’s say you use a Twitter trigger, it is still polling on a schedule, say every minute. We define a Recurrence trigger. Next is Actions, which is a set of steps. Possible types are Http, ApiApp, or even a Workflow itself. In the demo, we wire up a Twitter connector and a DropBox connector.

At this point, I can’t help but be struck by the number of sessions I’ve seen that are using Twitter in their demos. Does this mean it’s a treasure-trove of valuable data? Or is it just really good for pedagogical purposes? Either way, I’m glad that in at least one session, I was able to get a tweet saying “Lee-rooooy Jenkins!!” onto the presenter’s screen.

It’s nice that in Logic Apps everything is metadata driven, defined in Swagger. This is nice because then everything can wire up apparently pretty seamlessly. At least it did in the demo. More complex logic can be accomplished through conditions and JSON transformations. As for what you have to do to prepare your own API to be used by a Logic App, as long as you have a RESTful, JSON-returning API using OAuth or another common Auth provider, you should be able to use it in a Logic App.

As far as a new possible location for your Logic App, just this week, Azure App Service Environments were released, which encapsulate a fully isolated and dedicated environment to run all your apps, be they Web, Mobile, API, or Logic.

It was a great session on a cool technology with good demos. Though I do have to wonder about Logic Apps’ similarity to Azure Data Factories. There are certainly similarities, as each are visual workflows of data in the cloud. But Data Factories seem to be more about the Analytic Data Pipeline and have better connections to SQL and other data stores. Logic Apps seem to be all about APIs and smaller volume business process data flows. It could just be all about size of data. I’ll have to play with each more to give a better comparison.

What’s New in Direct2D and DirectWrite for Windows 10

Presented by Anthony Hodsdon.

This was my final session at Build and it was nice coverage of some fundamental graphics enhancements that can really make a difference in your Windows apps. Imagine you have document text in your app and you provide pinch-zoom-in to enlarge your text. Odds are the text is pixelated as you’re zooming and then snaps to a much cleaner view when you let go. (I just confirmed it on a Word document on my Windows 8.1 slate). This is because in Windows 8.1, by default, text was constantly being re-rasterized, putting a lot of work on the CPU that gets uploaded to the GPU. So, the app developer probably did what was common practice, which was to render the document to a bitmap during the scaling, hence the pixilation on zoom-in. The solution: Hardware accelerated vector-based text rendering in Windows 10. Anthony gave an impressive demonstration of zooming in very close to a document in a Windows 10 app and seeing the text remain clear with nice, round Béziers.

I think it’s little formatting things of this sort that really make for an enjoyable app experience for users. This is what the Direct2D and DirectWrite enhancements were bringing in this session. Direct2D and DirectWrite are hardware accelerated graphics APIs for 2D graphics and text display, respectively. Enhancements that come for them in Windows 10 are going to help displays not just on PCs, but phones as well.

For example, most phones don’t want to give up the ½ GB of space needed for all desktop fonts. If they don’t have the font your app is using, it could cause serious formatting issues. The solution: asynchronous font downloading and rendering built in not only to DirectX, but directly to XAML as well. You can configure your app to display a default font as you download the appropriate font, as in the following code:

class MyFontDownloadListener : IDWriteFontDownloadListener
{
STDMETHOD_(void, DownloadCompleted)( /* … */)
{
RerenderDocument();
}
}
spDwriteFactory->GetFontDownloadQueue(/* … */, &spQueue);
spQueue->AddListener(/* … */);
spQueue->BeginDownload();

Non-text graphics enhancements include things like built-in shader linking for optimization, and a simple, efficient API for loading and rendering images which reduces what was chucks of code down to a manageable amount.

Inking capabilities exist as a new 2D primitive that is hardware accelerated and provides low latency, high fidelity, and no polymerization. Anthony gave a great demo with his own handwriting showing how one can manipulate ink thickness and ink rotation. Makes me bummed for missing the DirectInk talk yesterday; I’ll have to go back and watch it.

In summary, as Anthony put it, these enhancements will mean your apps on DirectX will be faster, more maintainable with less code, more consistent across devices, and more beautiful.

Final thought:

So that’s it. Heck of a 3 days. Build is the kind of conference that makes you realize how much more you’ve got to learn. I’d say that’s a good thing.

My Day at Holographic Academy

[This entry was written by Jeff Mlakar, a member of the Business Intelligence Team at Bennett Adelson.]

Today was day 1 at the Microsoft Build Conference.  While there were many exciting things that came out of the Keynote like an Android subsystem on Windows, Objective-C apps, and, I’m happy to report, lots pieces for the data geek like me such as Azure SQL Data Warehouse and Data Lake.  But by far the most exciting was again Windows Holographic and HoloLens.

If you’re not familiar, take a look:

And I’m also proud to say Cleveland represented.  Both Case Western Reserve University and the location of my last consulting job, the Cleveland Clinic, were front and center for the main demo:

I was eagerly jonesing to get my hands on the device and had no idea if I’d get a chance to, let alone code for it.  So when they announced at the end of the Keynote that they were now taking registration for HoloLens events at the conference, I couldn’t register fast enough.  Literally, the site was slammed with requests and by the time I finished I was certain I didn’t get in.  You can imagine my excitement when I’m sitting in my next session on the Microsoft Band when I get an email that my registration was approved and that I was to report to the Intercontinental Hotel in a half an hour.  After figuring out exactly where that was, I skipped lunch and made a beeline for it.

There were 3 possible sessions:  a demo, a one-on-one, and a 4-and-a-half hour “Holographic Academy”, where you’ll actually learn to code for the thing.  Naturally, the 3rd option was my 1st choice.

After I got to the hotel, there was actually a little bit of waiting, so I ordered a burger at the bar.  Only a minute later I was approached to take part in a user experience study on HoloLens prior to the Academy.  My desire to get the thing on my head greatly outweighed my desire for the burger, so I left before I even had a bite. 

I had to sign an NDA on the experience, so I can’t speak too much on it, except to say that it was mainly about how a first time user would react to working with the device.  Nothing too exciting; didn’t even get to see a hologram yet.

So it’s on now to the Holographic Academy (after a few fast bites of the burger the barman graciously saved for me).  We had to check all cell phones and devices before going in, so please forgive the lack of pictures or proper code samples.  The code samples I provide will be from memory and what’s jotted by hand in my notebook, so I can neither confirm nor deny if they’re correct at this point.  And the only picture I can give you is of my badge:

clip_image001

 The 65 sticker on there isn’t my attendee number but my measured PD (Pupillary Distance), 6.5 centimeters.  I don’t know if that’s good or not…

They marched us in 2-by-2 into a large computer lab lit like a hip night club.  There were tables each with 2 large desktop workstations and couches besides coffee tables behind us which we were told we would use to try out the HoloLens’ interaction with physical objects.  My partner, a Kinect MVP I met in line, and I met our guide for the session.  It was 1 guide per 2 attendees with one speaker leading from the center. 

With cheers from the attendees, they start the session.  The intro is quick, with emphasis on HoloLens being the first device of its kind and the ease at which one will be able to develop and release apps for it, since apps will be on the Universal Windows Platform (UWP) with an existing store.  The Windows Holographic team talks about how every major advancement in computing has been a change to Input and Output, a good way to look at it.  HoloLens’ Inputs:

  •          Gaze, Gesture, Voice
  •          Spatial Mapping
  •          Holographic Camera

Its Outputs:

  •          Scalable Augmented Reality
  •          Light and Color locked to the Physical World
  •          Spatial Sound

We start our demos with a “Holo World” app.  I’m not usually one for puns or wordplay, but given how excited I was to actually hold the device, I’ll allow it.  By plugging our HoloLens’ into our lab machines via a micro-USB connection in the back right, we can open a browser and navigate to http://127.0.0.1:10080/AppXManager.htm to administer the device.  We find there is one application already loaded, and we start it from the website.

Putting on the device is harder than I expected.  But maybe that’s just me.  You start be tilting the inner headband portion that moves separately from the device, and then turning a wheel in the back to loosen it.  Your head almost feels like it’s climbing into the device as you slip the headband around the back of your head and hairline.  You turn the wheel to tighten the headband and then adjust the lens down and forward.  It doesn’t have to, and shouldn’t, rest on your nose.

The first thing we see is a blue windows logo in the middle of a blue rectangle.  We’re told to perform the most common clicking gesture, which is holding your hand out, pointing your index finger up and then pinching it down to your thumb.  It’s basically the “I’m crushing your heads” motion.  After performing this, the app starts and we see a three dimensional jeep floating in front of us.  We move our heads and find that as the virtual jeep approaches the physical coffee table in front of us, the surface of the table is permeated with small virtual triangles, indicating that the jeep is near a surface.  We click again and the jeep falls to the physical table.  Now that it is placed we can walk around it and observe our augmented reality.

First impressions:  I’ve seen 3D before, but I’m surprised how quickly my eyes and brain accept a virtual object in a physical world.  It’s very impressive.  There is one big limitation I see with the device right now and that is the clipping boundaries.  When you’re wearing the device, there’s only a relatively small rectangle of your field of vision that can see the virtual objects.  If you’re not looking in that rectangle, you don’t see the objects.  It’s hard to say the size of this rectangle, but I would say you can think of sitting on your couch watching a decent-sized TV.  The TV screen is roughly the amount of your vision within this clipping boundary.  So as I’m walking around looking at the virtual jeep, I notice it is sometimes clipped.  I’m sure the technology will adapt to expand this soon.

We find we can place pins on the table for the jeep to drive to.  I put a pin on the neighboring couch and watch the jeep jump from the coffee table to the couch.  I’m giddy.

After playing with this demo for a bit, it’s time to make an app of our own.  We take the device off and plug it back in to the usb.  From the webpage, we stop the app.

The app we’ll be building is called “Project Origami”, and we’ll be building it in Unity.  Developing graphics in HoloLens is as simple as using DirectX with some Holographic APIs.  So this means we have the usual options for developing graphics applications against it.  You could imagine making a graphics layer in C++/DirectX and then the majority of your application’s code in C#.  I ask how migrating XAML apps to HoloLens will work, as it seems from the keynote that 2D Windows Universal apps will just run 2D in the 3D HoloLens world.  I’m told this won’t be covered today, but should be a fairly seamless migration.  We’ll be using Unity today.  Unity is there and they give a brief overview for anyone who hasn’t used it.  It’s very nice that before the device is barely able to be seen by the public that it’s already getting support from a platform as respected as Unity.

We work in a Unity project, “Project Origami”, which is already started for us.  Over the course of the session, we don’t really do anything out of the ordinary in Unity.  We have meshes, game objects, and write scripts to control the behavior of the objects and accept user input.  The only big differences are Unity connecting a camera to the Holographic camera, our scripts containing a reference to the HoloToolkit namespace, and a Unity mesh that is dynamically built from the Spatial Data returned from the HoloLens.

We drag some pre-built meshes into a new Unity scene, set up our cameras, and preview the scene in Unity.  To send the device to HoloLens is a 3-step process.

1)      Build the Project from Unity

2)      Load from Visual Studio and configure the project properties to run in a remote device

3)      Start the project from Visual Studio with HoloLens connected

We do these and see the mesh in front of us in our HoloLens.  It is two Origami balls floating above a few other origami objects on a white canvas with a drain in the middle.  We disconnect our device from the usb, walk around and view the scene from all angles.

The next step is to give ourselves a cursor with which to interact with the world.  We create a small red torus and create a C# script for its behavior.  We add the following using directive:

using HoloToolkit;

And in its update method we put in the following code:

var head = StereoCamera.Instance.Head

head in this case is of type UnityEngine.Transform.  This gives us the location and direction of our gaze from HoloLens.  We can then do a ray trace to find what object our gaze is on with the following code:

if(Physics.Raycast(head.position, head.forward, out hitInfo)
{
FocusedObject = hitInfo.collider.gameObject;
}

We put in some more code to position the cursor torus based on the normals of the mesh it’s hitting, but I won’t include that here, as there is nothing HoloLens specific about it.

Our next step is to add some select code.  We add a script called SphereCommands and attach it to our origami spheres.  We put in an OnSelect() Method and invoke it when we detect the user input of a click from the click gesture.  If there is collision between our cursor and the sphere, we release the sphere to gravity.  We try it out and experience selecting the hovering spheres and watching them hit our surface, rolling off it, and then falling through the physical floor.

For our next demo, we use a mesh in Unity that represents the spatial data brought in from HoloLens.  We set up collisions with our objects.  We now demo and watch our origami balls fall to our virtual canvas, and then fall and roll on the virtual floor.  I play with it trying to get the balls to collide with the couch and other objects.

We perform more demos using the spatial data.  First we simply set our Unity visualization of the spatial mapping data to the triangular mesh so we can see how HoloLens has interpreted the physical objects it sees around it.  Of course, it’s not perfect.  But still, I’m mesmerized by it.  In our next demo we practice moving our scene around the room using the spatial data.  For all this, we use the following:

SpatialMapping.Physics.RaycastMask

We do some sound and voice recognition.  For sound, we do some ambient and impact sound and even demonstrate how we can dim sound based on distance from the scene.  Here is a snippet of the impact sound code:

SpatialSound.Play(“Impact.wav”, this.gameObject, vol: 0.3f)

We add voice commands to drop our objects and reset our scene like so:

KeywordRecognizer.Instance.AddKeyword(“Reset world”, (sender, e) =>
{
Resetting = true;
} , null);

We wrap up by adding a pre-built scene to demonstrate HoloLens creating a scene where you look “through” physical objects.  As in, where it adds reality that appears to be behind physical objects.  We watch our origami balls fall through the drain hole to a whole scene beneath the computer lab floor, complete with a green origami landscape and red flying origami birds.

All-in-all, it was amazing stuff.  We wrapped up getting to meet the team that built the following:

And now I’m exhausted in my hotel room and ready to turn in for another day.  Tomorrow’s activities for me are mostly data-related: sessions on Azure SQL Database, HDInsight, AzureML, PowerBI and such.  Exciting, but probably not as dazzling as today’s HoloLens activities.