hololens

The Amazing Hololens: Microsoft Shows us the Future

When I saw the first promo videos of the Microsoft Hololens I honestly thought this was another one of those “visions of the future” videos they do every now and then. If you want a bit of a laugh, check out this old one:

It’s not that they got it wrong, just that it’s all so hilariously clunky. In any case, the Hololens demo came across as that sort of thing. Here comes a guy wearing a headset and the whole world around him turns into this holodeck-like world of floating graphics and virtual objects.

“This is just another concept” I thought to myself, but I could not have been more wrong. It turns out that not only was the Hololens not just a concept, but the working prototype used in live demos would soon be replaced by an actual commercial product small enough to fit everything into a headset.

What is the Microsoft Hololens?

The Hololens is a “mixed-reality” headset. You’ll hear about this concept a lot on this site, but mixed reality is a more advanced implementation of augmented reality. Mixed reality systems generally have technology in them that can map and measure the real world. This allows them to adjust the rendered imagery in such a way so that it seems to blend into the world seamlessly.

For example, if the system has a rangefinder to detect flat surfaces such as walls and tables, it can then render virtual objects that persist in their relationship with those surfaces. A virtual TV screen will appear to hang on the wall just like a real one; doing essentially the same job.

The Microsoft Hololens is one of the first truly practical mixed-reality headsets made, but it turns out it probably isn’t for the likes of you or me. At least not yet. It’s not exactly a made-up vision of the future, but it’s a reality only for those who can afford the $3000 price tag.

the Microsoft Hololens

All-In-Wonder

The Microsoft Hololens is a completely self-contained device. It doesn’t have a tether to a bigger, more powerful computer. There’s no backpack or wireless display transmission. What you see is what you get.

The headset isn’t exactly svelte though. It’s a chunky beast, but at least it isn’t too heavy. It comes in just a smidge under 600 grams. Compare that to something like the DJI Goggles drone cameras headset which weighs twice that much. That porker is plenty comfortable, according to users, so Microsoft hasn’t weighed early adopters down.

The design is pretty simple, though. A single headband wraps around the user’s head, with a semi-transparent visor covering the eyes. It’s actually quite elegant and contemporary. The headband is not monolithic. Instead there’s an inner ring that is what’s actually resting on your head. This inner ring can tilt back and forth, meaning you can adjust the position of the Hololens without having to shift its position on your head. That’s a big departure from the current generation of VR and AR headsets that use either the clunky three-strap design or the monolithic headband with tilt-up visor.

It goes beyond even that. What appears to be a solid band can actually EXPAND to accommodate bigger noggins. Good news for me, since I’ve never found an off-the-shelf hat that actually fits. I have a seriously large head, guys.

A Closer Look

If we really look at the device beyond that initial impression, it quickly becomes clear that there is a lot going on under the hood. Over the user’s eyes are what appear to be a pair of glasses. Above the “browline” of the visor we see several camera lenses. The optical systems are quite involved.

Looking at an exploded view of the Hololens, it turns out those little camera blisters we see are part of a complex sensor array nestled in a compartment above the visor. It’s like looking into the eyes of some strange techno-spider. These sensors are what let the Hololens physically detect the world around it. The two lenses that sit right in front of the eyes are referred to as “holographic” by Microsoft. They seem to be a type of optical waveguide that has high-quality imagery projected onto it.

The reason the imagery projected onto those lenses seem to fit so perfectly into the real world around us is thanks to a dedicated processor. This little electronic brain weaves together all that sensor data with the renderer and projection system to create mixed-reality images. Microsoft calls it the “HPU” or Holographics Processing Unit. There’s really nothing to compare it to, since it sits next to the CPU and GPU as its own distinct thing. It’s not quite light-field technology, as Magic Leap is attempting to do, but the HPU is certainly an incredibly impressive dedicated hardware component. It shows that specialized hardware has a future in VR, much in the same way we moved from software rendering to dedicated GPUs.

The one big drawback of the current hardware is its incredibly narrow field of view. While Microsoft has not given an official number, independent people who have used the Microsoft Hololens estimate that its FOV is around 35 degrees. Contrast that with premium VR headsets that start at 90 degrees and typically have a horizontal FOV of 110-degrees. It also means that many of the promotional videos that show us what users are seeing are way too optimistic about the field of view. A key improvement that has to be worked on is the widening of the FOV.

Hololens Apps

Of course, without actual software applications the Hololens is just a goofy-looking head ornament. However, with a 3K price tag it’s hard to imagine a flood of applications for the machine. That’s not to say there isn’t anything for the Hololens. The product is of keen interest to people in corporate environments or scientific fields. Even in education and training spheres you’ll find a market for the Hololens. It makes much more sense if you think of it as a shared organizational resource rather than something each individual person is expected to own. So companies, labs, and universities can purchase a small number of units to be shared.

So what apps are there worth knowing about? Let’s have a look at the most interesting Hololens apps.

Skype for Hololens

skype

Skype is a pretty mundane part of our lives these days, although 15 years ago I would have been flabbergasted by mainstream video-calling technology. Now I have Skype on my phone, tablet, and PC. Using Skype on a Hololens turns the magic back on again. While it doesn’t create a Star Wars-like holographic illusion, the Skype video box and UI elements are projected into the space surrounding you.

Of course, since you are wearing the Hololens the people you are speaking with can’t actually see your face, but Skype for Hololens has another trick up its sleeve that might make up for that particular shortcoming. You see, the people on the other end of the call who are just using a normal Skype device can see through the Hololen’s “eyes”. In other words, they can see what you are seeing.

Even better, they can use their tablet to draw directly onto your video feed. So it’s possible for someone to teach or instruct you remotely. The idea, it seems, is that people can get a better type of technical support. There are also business use cases where people have to collaborate on projects remotely. On top of this it features spatial sound, so you only hear the audio coming from the actual chat window. The app also allows for hand-tracking so that you too can draw and visually communicate with the other person.

Insight Heart

insight heart

I’m always fascinated by AR and VR applications that teach us about the physical world. I especially like anatomy applications. There are few things as beautiful as the complex mechanics of the human body.

Insight Heart is also available as an Apple ARKit program, but it’s the Hololens version that really caught my attention. The program focuses specifically on the human heart and does so in great detail. The heart is an organ that has played a central part in human culture. Before we knew what it actually did, many thought that the soul or mind was seated there. Even today, it has a special place in the mind, since although you can live with one lung or kidney a busted heart is a one-way ticket to Deadsville.

The actual model of the heart is beautifully detailed with great animation and sound. It shows that this application only focuses on one organ. I also think that Insight Heart is a great example of how to create an MR interface. The menus look intuitive and are crystal-clear.

Holobeam Tech

HoloBeam Tech

Remember how I said that Skype for Hololens didn’t do the whole Star Wars hologram thing? Well there’s an app for that and it is called Holobeam Tech. In fact, it’s not just me – the actual app description directly references Star Wars.

Holobeam Tech replicates that experience by using a special 3D capture method. The person you are speaking to is captured, digitized, and sent across the internet. To you it looks like their ghostly apparition is there in the room with you. It’s a little creepy, but also really cool. We’ve seen this sort of holographic telepresence thing in sci-fi movies for ages, but it’s awesome to see in real life. Of course, if this were to work both ways then both people would appear wearing their
Hololens units. A little lame, but future versions of the technology are likely to be much more compact and less intrusive.

HoloStudy (Demo)

HoloStudy

One of the most exciting applications of VR and MR technology is in education. Putting students in a virtual environment where you can control their first-person experience in detail allows for unique and powerful teaching potential. There’s a tech-demo out of exactly this sort of approach for the Hololens. It’s called HoloStudy and showcases a number of MR lessons in things like biology, chemistry, physics, and geology.

The imagination that went into these lessons is amazing. For example, the chemistry portion of the app starts you off with a virtual periodic table projected onto the wall. Then you can call up individual chemicals and view them as atomic or molecular models in 3D, 2D, or as “real” matter.

The geology lessons have an amazing 3D model that breaks the Earth up into its individual layers. The animation and solidity of the graphics are amazing, especially considering that this is mobile rendering hardware we’re talking about here. You can also see amazing models indicating how the Earth’s gravity puts a dent in space-time, and a great visualization of the planet's magnetic field.

HoloAnatomy

HoloAnatomy

I have at least two anatomy apps on my iPad and one of them has a pretty sweet ARkit-powered mode where you can lay the virtual body down on a table and then examine all its bits and pieces. It’s an awesome experience and I have literally spent hours learning about all the parts of the body through pure discovery. I’d see something in the model and then wonder “What does that do?” with only a tap of the finger separating the question from the answer.

HoloAnatomy blows that app out of the water and then off the planet. It’s the product of Cleveland Clinic and was designed specifically for use in medical training. Groups of medical students who are wearing Hololens units can all see the same shared graphics. They can discuss and interact with the renderings of working, detailed human anatomy. The level of detail is simply astonishing and I really believe tools like these will become standard in medical training practice.

Shared mixed-reality experiences are a cornerstone of the future “AR world” I talk about in a different article on this site. Extend this concept to other fields and even just to our recreational activities, and you’ll have a good idea of what things will be like in the future.

MineCraft for Hololens

This is without a doubt the Hololens application that got the most attention. Minecraft is already such a phenomenon, but it’s inclusion in a mixed reality platform makes so much sense the minute you see it. Don’t take my word for it. Check out the official demo:

In the demo, the player walks up to a real physical table that the Hololens recognizes. It then proceeds to build a Minecraft world on that table. It looks like a little diorama or some tiny world lorded over by a mighty, glasses-wearing god. The persistence and detail are just out of this world. I’m not even that big a fan of Minecraft, but seeing this demo actually made me want to play this version.

This was also the game demo that made certain genres such as real-time strategy really click for me. Most of us tended to think only of first-person games. Seeing Minecraft like this is beyond cool, and many gamers who saw that original demo audibly gasped when it all came together.

Windows Mixed Reality to the Rescue

The Hololens is an astounding piece of technology that simply isn’t ready for the mainstream. The technology just isn’t there yet. It’s too expensive, for one thing, and the FOV limitations really hurt how attractive it is for consumer purposes.

Halolens Demonstration

Luckily, Microsoft did not chuck all of their eggs in one basket. They’ve expanded their mixed-reality ecosystem to include more traditional mixed-reality headsets. I discuss the platform more fully in another article in this section, but basically it’s like a normal VR headset with two cameras and a few other sensors stuck on the outside. In fact, I believe that right now those MR headsets have far more impact than the Hololens itself. They are the most practical and elegant VR/MR headsets I have seen so far. On top of that, they cost nearly ten times less than a hololens.

Of course, these are still tethered devices, whereas the Hololens is a complete unit. So it’s not exactly an apples-to-apples comparison. These two product lines are, however, linked in one very important way – applications. Strictly speaking, if an app works on a Windows MR headset then it can be ported to Hololens and vice versa.

The Windows MR headsets are set to become mainstream products and that means much more incentive for developers to write applications for them. One day the technology in the Hololens will be cheap enough that almost anyone can afford it. It would be good if a mature and wide variety of applications is ready and waiting when Hololens technology is ready for prime time, but only time will tell if that’s how things will turn out. For now, most of us can only marvel that such a thing actually exists.


Android Vr

The Best VR Games and Apps on Android

Mobile VR, the likes of Google Cardboard and Gear VR, is the red-headed stepchild of VR. Google Cardboard kicked off the mobile VR revolution, but it started as a sort of joke-y afterthought. Today everyone is getting into mobile VR and several standalone VR headsets have been announced that are anything but a joke.

Mobile VR that requires a phone to be slipped into a headset case has come a long way from the early days of Cardboard. Developers are taking this new industry very seriously and the array of apps that make use of VR in some way has grown by leaps and bounds. Picking what VR apps are worth taking a look at from this mountain isn’t easy, and obviously I haven’t tried every app in the world, but all of these fine products are special in some way and will definitely help you get your money’s worth from that new mobile VR case you bought out of curiosity.

The 7 Best Android VR Games


Sisters

Sisters

Sisters is another horror experience on mobile VR that has gathered quite a following. This isn’t really a game so much as a short VR experience. It’s a ghost story that you get to experience in the first person, up close to the creepy characters.

There are now a number of sequels to Sisters which improve on it in many technical ways, but this original game has been updated with some of that eye candy and is still a creepy start to some creepy games. When you start up this app you find yourself seated on a couch. A storm is raging outside and the TV is on. You seem to be alone in this forbidding old house, but from the corner of your eye you spot a small figure. One of two weird little girl dolls. The sisters are here to play and there’s nothing you can do but stay around for the ride.

Sisters was one of the first truly creepy things I tried when I got a Google Cardboard. It’s free now, so there’s no reason you shouldn’t give it a go too.

Download From Google Playstore


InMind VR

InMind VR

I have two movies I really love that deal with shrinking someone down and then having them enter the human body as explorers. These movies are The Fantastic Voyage and, of course, Innerspace. I’ve always wanted to experience some form of that, and this game gives you a little taste of what it might be like inside the human body from a microscopic point of view.

OK, it’s not a realistic 3D rendering, but you get to travel into the brain and follow along the neural pathways until you run into something that shouldn’t be there. Yes, this is a game about making someone’s brain healthy again by zapping the parts that don't work right until they do.

The same people who made this also made a game called InCell, and I guess you can figure out the premise of that game just from the name.

InMind may just be an on-rails rollercoaster shooter, but it’s a well-made and imaginatively-themed one. So give it a go – it doesn’t cost anything.

Download From Google Playstore


Insidious VR

Insidious VR

I have a lot of faith in VR as a storytelling medium, and products like Insidious VR help keep that faith alive. It’s another scary story, but this time told using a mix of digital and live action media.

Insidious VR is basically a non-interactive VR movie with really good production values. How scary you’ll find it, especially on repeat viewing, is debatable. But half the fun is roping in other people who haven’t tried it and then letting them have a go while you watch.

Insidious is hopefully a sign of things to come, and it shows that VR can be a great storytelling medium in its own right.

Download From Google Playstore


Netflix VR

Netflix VR

Oh, boy. This is one of my absolute favorite VR applications. I’m already quite a Netflix junkie, but this app makes it possible to watch Netflix in style even when you’re on a plane or otherwise away from home. In fact, you might want to use it at home anyway, because my own place certainly isn’t as fancy as the virtual big-city high rise Netflix treats us to.

Unfortunately, this app only works on the Gear VR through the Oculus app, but if you’re lucky enough to have access, this is a must-have piece of software. The first time the lights dim down in the virtual theater is pretty thrilling, and I honestly think this is one of the best VR applications out at the moment.

Download From Google Playstore


Trinus VR

Trinus VR

Trinus VR is one of the smartest VR applications I’ve seen. It takes your mobile VR headset and tries to turn it into a tethered VR HMD. Basically, there’s the app itself and then a client program that goes onto your PC. You can connect the two devices with a USB cable (which works the best) or you can use a WiFi connection, which can be good enough depending your specific network hardware. The software is getting better all the time, but there’s no guarantee you’ll get good result with what you have at the moment. The good news is that the app isn’t that expensive and it also comes with a free demo version that allows you to test it before spending any money.

I really think that anyone who has an Android VR setup owes it to themselves to give Trinus a try. It’s the cheapest way to expand what affordable VR can do. The only glaring issue is the lack of external tracking, but it’s a small price to pay in order to pay such a small price.

Visit Website


A Chair in a Room

A Chair in a Room

One of the most popular VR genres on mobile is horror. There have been some crude apps in the past that were cute, but only scary in the sense that a high-school haunted house production is scary. You spend more time giggling than anything else.

A Chair in a Room is different. This game is widely acknowledged as one of the first truly scary VR games on mobile and uses the medium to it’s maximum potential. In this game you play a corporate lawyer who is visited by ghosts who try to make you change your ways. The game unfolds as a sinister story told over segments, each an opportunity to scare the pants off you.

This game uses some pretty advanced (for a mobile game) graphics. So only people with a true beast of a phone need apply.

Visit Website


Dinotrek

Dinotrek

I’m not such a hardcore fan of the Jurassic Park films, but I am married to one. That first scene in the first movie, where the members of the main cast first see the Brachiosaurus, is one that has instilled the fantasy of walking with dinosaurs into a whole generation of people.

Dinotrek lets you do just that. It puts you right there on the ground as various dinosaurs traverse the landscape. Being a mobile VR application, the graphics are hardly photorealistic. But they’re pretty darn compelling and it’s a great kid-friendly yet exciting experience. Think of it as a sort of personal Universal Studios ride.

Visit Website


google glass

Google Glass is the Best Failure in AR History

First of all, before we even get into anything about the Google Glass, I need to clarify that the product is actually still around. After going back to the drawing board, Google actually re-released the product, except it’s not available to the public. Now it’s exclusively marketed towards enterprise customers.

So this article is about the Google Glass that we can no longer get – the one that no one wanted, which caused Google to take it off the shelf for years. With that cleared up, let’s get into the history of this enigmatic product.

google glass on table

Through the Looking Glass

The Glass is the product of Google X, a sort of skunkworks development arm of the company that is today known as Alphabet. These are the guys who push the envelope and work on things like self-driving cars and internet balloons. Yes, I’m serious – look it up.

The first prototype of what would become the Glass was finished sometime in 2011 and weighed about eight pounds, which doesn’t sound very compact! The final version of the Glass had about the same shape and weight as a normal pair of eyeglasses. The only real clue that it was more than plain eyewear was a small module on one side of the glasses sporting a small prism and a camera. That bit of the puzzle is actually the Glass proper, since the module could in principle go on any pair of glasses. Presumably, if Glass ever became a mainstream device that’s exactly what Google planned to do.

By 2013 early adopters could apply for a Glass unit as long as they had the princely sum of $1500 to spend on it. Google opened the Glass up to even more people until around 2015 when it went back for a redesign. In 2017, the Glass was re-released, but this time as a product only for companies like Boeing.

People in Glass Houses

The Glass itself had some pretty impressive specifications, given how far mobile technology has come since 2013. It ran a custom operating system known as “Glass OS” and packed a system on a chip with two CPU cores. It also had 2GB of RAM and 16GB of flash storage. That may seem a little anemic in 2017, but putting it all into such a small form factor seemed like black magic at the time.

Those were just the conventional hardware specs. Google also managed to put in a bone conduction transducer for audio. I once owned a set of bone conducting headphones and they allowed me to hear everything going on around me while still letting me hear music and other audio. The visual part of the system was even more fancy, using a prism projector with a resolution of 640x360. To the user it appears to be a floating image the size of a 25” screen eight feet away.

There are plenty of inputs too. There’s a microphone for voice commands, a touchpad for menu navigation, and a raft of sensors. I’m not kidding – inside the glass there’s an accelerometer, gyro, magnetometer, light sensor, and proximity sensor. That’s a lot of tech packed into a tiny device. This space has to be shared with a lithium ion battery too, and don’t forget about the 5MP/720p camera!

The most mind-boggling thing about it all is the fact that the total Glass package ended up weighing 36 grams, or just over an ounce. See? Black Magic.

Don’t Be a Glasshole

While the hardware in the Glass is certainly impressive, given its size and weight, it’s a fair question to wonder what the thing is actually for. The Glass is technically an AR device, but it’s the type of AR that doesn’t really integrate with what you are looking at. It’s essentially a wearable display system that acts as an extension of your smartphone; you can discreetly read your emails, watch YouTube videos, and do other stuff you’d usually have to take your phone out for.

google glass

It also served as a platform for the Google voice assistant so that users could get a hands-free interface for their wearable computer. This is obviously pretty exciting, especially if you work as a surgeon or as another type of professional who needs to access information while also working with both hands.

One of the key features of the Glass is that big camera on the front. Really, I think this aspect of the device was its biggest downfall. It turns a person wearing a Glass into a roving surveillance platform. Talking with someone wearing a Glass can be pretty uncomfortable, as you are very aware of that camera. Privacy issues were always at the forefront of Glass criticisms. It’s funny – people seemed to be OK with flaunting every detail of their lives on social media, but the presence of the Glass camera made them aware of the fact that they could be watched. It’s illogical, but it’s also a very human reaction; one which Google never foresaw for some reason.

Psychologically, people seem to think differently about a camera positioned to be ready at any time and the fact that any of us can whip out a smartphone and start recording at any time. This resentment is one of the reason people who were early adopters of the Glass were nicknamed “Glassholes”. Society seemed unwilling to accept the technology.

Google Glass In Context

When Google Glass came to the attention of the public, it really was a novel concept to regular, non-tech geeks. While researchers at places like MIT have been playing with wearable computers for decades, the public was generally unaware. The sudden disruption to social norms made people uncomfortable, but this was before the VR and AR revolution of 2016. Let’s be honest, the Google Glass had some pretty mundane use cases. It’s an awesome piece of technology, but it was just a fancier smartphone interface.

Today, people might be much more willing to try or accept wearables in public. Modern AR is also much more sophisticated and compelling than simply projecting a small screen into space. Glass failed as a product because it was simply ahead of its time. The idea of Glass has not failed, however, and the work Google did on it played an important part in modern devices such as the Microsoft Hololens.


vr tracking

Tracking Your Muscles for VR Motion

Did you know that you are just full of electricity? Electrical impulses play an incredibly important job in your body. Inside your body you carry all sorts of electrical signals; mainly, these allow your body to perform internal communications. Electrical impulses are carried between the various parts of your body by the nervous system – a network of specialized cells that shoot electric impulses down their length.

This is how you can command your arm to move and pick up an object. Your brain contains trillions of these cells packed together, where they talk to each other in order to solve various problems. You and I are the product of this intercommunication between the cells. This is where our minds are generated. The human brain and nervous system is one of the most complex structures we know of in the universe. It’s a delicate yet powerful system that will still take decades, if not centuries, to fully understand.

VRroom

Messing with Perfection

Not understanding it all perfectly hasn’t stopped human beings from messing with it by pumping electricity into the nerves and muscles. As you probably learned in science class, it was an Italian genius named Luigi Galvani that discovered in 1780 that if you put electricity into the muscle tissue of dead frogs their legs would twitch. Although they were dead, there was still some chemical energy left in the muscles; when they received an electrical impulse they would dutifully contract.

Galvani basically invented the field of bioelectricity and one of the (dumber) results of this science are those exercise machines that make your muscles contract while you sit on the couch and watch TV. However, it turns out it works the other way as well; every time you move your arm or other part of your body a small electrical signal is generated. With the right sensor this signal can be recorded and used for all sorts of purposes. What we care about, however, is how this could be used in VR, and that’s where the magic can really happen.

Myoelectric Control

At the moment, motion control in VR is usually achieved by something like a Touch Controller. It’s something you hold, and the motion sensors inside the device translate the forces that your arm exerts on it into mirror-motion in your virtual arm. Alternatively, they might use a glove or other form-fitting wearable to measure fine movement in high detail. These days there are even high-resolution cameras that do a pretty good job of watching your movements and then relaying them to your avatar.

Now, an alternative is to introduce sensors that identify a particular muscle’s movement by measuring its electrical activity and then amplifying and converting that into motion data that a computer can understand. There are many advantages to this. For one thing, you don’t have to use external tracking hardware, which means increased freedom of movement and the possibility of portable control. In this case it would also be important to note that myoelectric sensor devices could be worn underneath the clothes so they don’t need to get in the way of your daily activities.

The Real Deal

Myo Gesture

Does this all sound a little experimental? Well, if you have a spare two hundred bucks or so lying around you can buy a commercial muscle tracker right now. It’s called the Myo Gesture Control Armband; with it you can control stuff in supported VR apps or in other applications just by moving your arm, hand, or fingers. It’s some serious Sci-Fi stuff, if you ask me. Right now the Myo is a little niche, with only about a 100 applications of all types supported, but the technology is here. It’s affordable, and if it becomes popular enough it could change VR forever.

Buy On Amazon


Apples ARKit

What is Apple's ARKit and Why is it Important?

Augmented reality on mobile phones has been a fun, but very flawed, experiment so far. Applications such as Pokemon Go have pushed the idea of AR with mass market appeal ahead by leaps and bounds, but the actual technology hasn’t been all that impressive.

In the Pokemon app, for example, you have to hold your device just right for the illusion that Pikachu is really squatting on your carpet to seem at all real. It’s very apparent that the projection is dumbly layered onto the image captured by the camera. For the projection to really work well, you need to have a digitized map of the environment. If the software can identify those elements of the scene, it can render the digital imagery in a far more realistic way.

For one thing, objects can be persistent in relation to other objects. So if you put a virtual coffee cup on your real table in a particular spot, it doesn’t seem to move relevant to that table. You can walk around it, lean into it, and generally treat it like a real object from an observational perspective.

ARKit logo

Doing the Tango

Google has been working on creating this sort of concrete and environmentally-aware AR for a while. One line of development is Project Tango, which describes a hardware platform full of sensors. These are used to directly map things like the ceiling, walls, furniture, and so on. We’re talking infrared cameras, ultrasonic sensors, and a lot of sophisticated code and processing muscle to make it all work in real time.

Tango is incredibly impressive, but its main problem is that it will only work on a Tango device. There have been some Tango phones released, but they hardly have the sorts of sales numbers to make them common. With a limited hardware install base, developers aren’t keen to make compelling apps. It’s an age-old technology chicken and egg scenario.

Apple Thinks Differently

Enter Apple, which has approached the whole problem from an entirely different perspective. They wanted to see if they could create a software solution that could use the hardware people already had in their existing Apple devices to do the same sorts of things that Tango does. Using nothing but a camera and the internal motion sensors of the device, they would attempt to create a more lifelike AR experience. That effort and years of development and acquisitions has led to what we know today as Apple ARkit.

So What Is It?

ARkit is a software development kit. In other words, Apple has taken all its software technology that enables this new generation of AR and packaged in such a way that third-party software developers can use it in their own apps. They don’t have to worry about the AR aspects of their app and can concentrate on simply designing something awesome. Any features and abilities that Apple has built into ARkit is theirs to play with and use.

Basically, when the developer wants to do something like map a flat surface and work out the mathematics of calculating the angle of the AR projection and how it should be lit, they don’t have to figure any of that out themselves. They can just call on the functions from the ARKit library and the job’s done. That’s obviously a big oversimplification, but the point is that ARKit makes high-end markerless AR much easier to make on Apple platforms.

So let’s go over what exactly it is that ARKit can do and what makes it so special.

A World’s Eye View

The biggest hurdle that the ARKit technology tackles is world tracking. What is world tracking? Remember that augmented reality works by marrying digital projection in virtual space with points in real space. In other words, you need some way to map and measure the real world for your digital projections to have any hope of the AR experience feeling real and solid.

As I mentioned above, Google has basically achieved this with its Project Tango technology, but ARKit can perform advanced world tracking on any Apple device with at least an A9 processor and a camera.

amazon AR update

With older approaches to AR using nothing but the device’s camera, motion sensors, and software, it was common to use a “marker”. This would be an object with QR codes or specific unique pictures printed in it that the software could easily recognize. That would then be used as a low-impact tracking method. Alternatively, you could cheat a little with markerless projection and create a pretty good illusion. The AR Pokemon from Pokemon Go is a good example. These little guys can actually look like they are sitting on your desk or the floor. Move the phone, however, and the whole illusion breaks since the Pokemon just gets overlaid on whatever the camera points at.

World tracking in ARKit uses a method known as visual-inertial odometry. This is an approach that takes a visual analysis of the current camera feed and combines that with motion sensor data. The secret-sauce math of ARKit is obviously the key to it all. The way that Apple explains it in its documentation for developers is that their world tracking system identifies landmarks or features in the image. These can be things like the edge of a desk or the spot where the wall and floor meet. It tracks the difference in those little landmarks across the frames of the video and then compares it to the info coming from the motion sensors. By doing this it can quickly get a good sense of where things are in the real world.

It’s Not Perfect

Now we have to be clear that ARKit is not as good as Tango. The Google system can directly measure height, distances, and other physical properties. The amazing thing is that ARKit comes so close and can run on millions of devices that people already own.

The developer can, for example, ask the world tracking function to identify flat surfaces and that information can be used to map 3D projections. But there are limits to what the technology can do. For example, if there isn’t any detail to pick out in the picture there isn’t much you can do. For example, if you have a dark floor and dark walls, the software won’t be able to tell where one begins and the other ends.

So to create those truly convincing AR experiences with ARKit, developers have to plan for the lighting conditions they require and encourage types of motion that help ARKit make the most accurate maps.

ARKit and the Mixed Reality Future

If you’ve seen the Google Cardboard or a few of the plastic clones that others have made of it, you might have noticed that it often has a hole in the front where the camera can poke through. This is meant for AR purposes, but until now the sort of AR experiences that you could achieve using just a phone camera weren’t very compelling. With ARKit it's not hard to imagine Apple introducing an iPhone VR case into market that will really bring this new AR solution to life.

For now, however, there are a ton of ARKit apps you can try out if you have one of the compatible ARKit devices. That list includes the iPhone SE, 6s, and up. It also includes the entry-level 2017 iPad and all the iPad Pros.

It's still early days for ARKit as I write this, but I already have a few favorite applications, so here they are.


Stack AR

Stack AR iOS

Stack AR is basically just a tech demo, but I have found that this little free app is one of the most effective ways to immediately understand and show what makes ARKit different from everything that’s come before.

The “game” part of this app is pretty simple. You stack digital blocks on a surface. The topmost block is constantly moving to and fro. Every time you tap, it stops above the next block down. Any part of the block that doesn’t align with the lower block falls away. When you run out of block material the game ends. The object of the game is to build the highest tower.

Now, Stack AR can actually be quite a lot of fun, but as you play be sure to try a few things. Mainly, when you’re done building your tower, don’t immediately reset. Instead take a walk around it. Look away and look back. Marvel at the fact that the blocks are still where they were. Note that the shadow cast fits in with the actual lighting in the scene. Back away from the tower and it all still scales properly. These things would have been unthinkable in the past with the previous generation of AR technology.

Download From App Store


AR MeasureKit

AR MeasureKit iOS

AR MeasureKit is one of the smartest applications of the ARKit technology I’ve seen, which is surprising since it's also an ARKit launch app. This is an application that uses the world tracking technology of ARKit to help you measure stuff in the real world. Want to know how high something is? Maybe if something is level? What about a person’s height? These are all possible with a fair bit of accuracy just using the application. I used the height measurement on a whole bunch of people and the reported height was usually correct to within a centimeter.

The best part of it all is that the measurements are persistent, so as you look around the room they all stick exactly to where they were. It’s a great example of the world tracking technology in action.

Download From App Store


World Brush

World Brush iOS

If you know anything about PC-based VR you’ve probably heard of Tilt-Brush. It’s a Vive and Oculus application that lets you use the motion controllers of each system to paint artwork in 3D space. World Brush lets you create drawings in 3D space using an iPhone or iPad. The drawing exists within 3D space and you can walk around it as it sits in fixed relation to the environment.

That’s not even the best part of this application, however. It uses the cloud and geo-tagging so that anyone else who moves through that same space can see your actual drawings. I think the potential of something like World Brush is pretty revolutionary. For one thing, it is one of the first apps to allow users content-creation powers within a shared AR world.

Download From App Store


HouseCraft

HouseCraft iOS

Are you one of the millions of people worldwide who enjoy playing the Sims? Do you like spending your time buying virtual furniture and then moving it around obsessively. Well then, you are going to love Housecraft. This app provides you with a library of objects that you can place in your real home using ARKit. Thanks to this new technology from Apple, the scale of the virtual furniture actually looks realistic. IKEA has a similar app out too, but in my opinion Housecraft does a better job despite not selling real furniture as does the Swedish retail chain.

Download From App Store


Thomas & Friends Minis

Thomas & Friends Minis iOS

Who hasn’t watched Thomas the Tank Engine as a kid? Like most children, I had an affinity for trains when I was little. I even had a toy train set with real smoke and everything. Now as an adult I don’t have the space or time to play with toy trains anymore, which is why I think this app is pretty cool. Yes, it’s for children and normally I wouldn’t have given it another thought, but the addition of an AR mode makes it worth trying out, even if only for a few minutes. I can only imagine using this technology with a proper head-mounted AR setup and playing with a realistic model train. Honestly, I hope a more serious app is inspired by this first attempt at an AR train set.

Download From App Store


The AR We Deserve

Say what you want about Apple, but they’ve done us all an incredible favor here. Not only have they solved so many of the problems standing in the way of convincing AR, they’ve also done it by using the hardware people already have in their possession. I’ve played with ARKit apps extensively now and there’s no doubt that it finally brings that minimum level of AR technology to make you feel like you’re looking at a real object rather than just a picture on a screen. I find it hard to believe that Apple will be stopping at just a phone or tablet implementation, so don’t be surprised if there’s an Apple AR HMD in our near future.