Augmented reality on mobile phones has been a fun, but very flawed, experiment so far. Applications such as Pokemon Go have pushed the idea of AR with mass market appeal ahead by leaps and bounds, but the actual technology hasn’t been all that impressive.
In the Pokemon app, for example, you have to hold your device just right for the illusion that Pikachu is really squatting on your carpet to seem at all real. It’s very apparent that the projection is dumbly layered onto the image captured by the camera. For the projection to really work well, you need to have a digitized map of the environment. If the software can identify those elements of the scene, it can render the digital imagery in a far more realistic way.
For one thing, objects can be persistent in relation to other objects. So if you put a virtual coffee cup on your real table in a particular spot, it doesn’t seem to move relevant to that table. You can walk around it, lean into it, and generally treat it like a real object from an observational perspective.
Doing the Tango
Google has been working on creating this sort of concrete and environmentally-aware AR for a while. One line of development is Project Tango, which describes a hardware platform full of sensors. These are used to directly map things like the ceiling, walls, furniture, and so on. We’re talking infrared cameras, ultrasonic sensors, and a lot of sophisticated code and processing muscle to make it all work in real time.
Tango is incredibly impressive, but its main problem is that it will only work on a Tango device. There have been some Tango phones released, but they hardly have the sorts of sales numbers to make them common. With a limited hardware install base, developers aren’t keen to make compelling apps. It’s an age-old technology chicken and egg scenario.
Apple Thinks Differently
Enter Apple, which has approached the whole problem from an entirely different perspective. They wanted to see if they could create a software solution that could use the hardware people already had in their existing Apple devices to do the same sorts of things that Tango does. Using nothing but a camera and the internal motion sensors of the device, they would attempt to create a more lifelike AR experience. That effort and years of development and acquisitions has led to what we know today as Apple ARkit.
So What Is It?
ARkit is a software development kit. In other words, Apple has taken all its software technology that enables this new generation of AR and packaged in such a way that third-party software developers can use it in their own apps. They don’t have to worry about the AR aspects of their app and can concentrate on simply designing something awesome. Any features and abilities that Apple has built into ARkit is theirs to play with and use.
Basically, when the developer wants to do something like map a flat surface and work out the mathematics of calculating the angle of the AR projection and how it should be lit, they don’t have to figure any of that out themselves. They can just call on the functions from the ARKit library and the job’s done. That’s obviously a big oversimplification, but the point is that ARKit makes high-end markerless AR much easier to make on Apple platforms.
So let’s go over what exactly it is that ARKit can do and what makes it so special.
A World’s Eye View
The biggest hurdle that the ARKit technology tackles is world tracking. What is world tracking? Remember that augmented reality works by marrying digital projection in virtual space with points in real space. In other words, you need some way to map and measure the real world for your digital projections to have any hope of the AR experience feeling real and solid.
As I mentioned above, Google has basically achieved this with its Project Tango technology, but ARKit can perform advanced world tracking on any Apple device with at least an A9 processor and a camera.
With older approaches to AR using nothing but the device’s camera, motion sensors, and software, it was common to use a “marker”. This would be an object with QR codes or specific unique pictures printed in it that the software could easily recognize. That would then be used as a low-impact tracking method. Alternatively, you could cheat a little with markerless projection and create a pretty good illusion. The AR Pokemon from Pokemon Go is a good example. These little guys can actually look like they are sitting on your desk or the floor. Move the phone, however, and the whole illusion breaks since the Pokemon just gets overlaid on whatever the camera points at.
World tracking in ARKit uses a method known as visual-inertial odometry. This is an approach that takes a visual analysis of the current camera feed and combines that with motion sensor data. The secret-sauce math of ARKit is obviously the key to it all. The way that Apple explains it in its documentation for developers is that their world tracking system identifies landmarks or features in the image. These can be things like the edge of a desk or the spot where the wall and floor meet. It tracks the difference in those little landmarks across the frames of the video and then compares it to the info coming from the motion sensors. By doing this it can quickly get a good sense of where things are in the real world.
It’s Not Perfect
Now we have to be clear that ARKit is not as good as Tango. The Google system can directly measure height, distances, and other physical properties. The amazing thing is that ARKit comes so close and can run on millions of devices that people already own.
The developer can, for example, ask the world tracking function to identify flat surfaces and that information can be used to map 3D projections. But there are limits to what the technology can do. For example, if there isn’t any detail to pick out in the picture there isn’t much you can do. For example, if you have a dark floor and dark walls, the software won’t be able to tell where one begins and the other ends.
So to create those truly convincing AR experiences with ARKit, developers have to plan for the lighting conditions they require and encourage types of motion that help ARKit make the most accurate maps.
ARKit and the Mixed Reality Future
If you’ve seen the Google Cardboard or a few of the plastic clones that others have made of it, you might have noticed that it often has a hole in the front where the camera can poke through. This is meant for AR purposes, but until now the sort of AR experiences that you could achieve using just a phone camera weren’t very compelling. With ARKit it’s not hard to imagine Apple introducing an iPhone VR case into market that will really bring this new AR solution to life.
For now, however, there are a ton of ARKit apps you can try out if you have one of the compatible ARKit devices. That list includes the iPhone SE, 6s, and up. It also includes the entry-level 2017 iPad and all the iPad Pros.
It’s still early days for ARKit as I write this, but I already have a few favorite applications, so here they are.
Stack AR is basically just a tech demo, but I have found that this little free app is one of the most effective ways to immediately understand and show what makes ARKit different from everything that’s come before.
The “game” part of this app is pretty simple. You stack digital blocks on a surface. The topmost block is constantly moving to and fro. Every time you tap, it stops above the next block down. Any part of the block that doesn’t align with the lower block falls away. When you run out of block material the game ends. The object of the game is to build the highest tower.
Now, Stack AR can actually be quite a lot of fun, but as you play be sure to try a few things. Mainly, when you’re done building your tower, don’t immediately reset. Instead take a walk around it. Look away and look back. Marvel at the fact that the blocks are still where they were. Note that the shadow cast fits in with the actual lighting in the scene. Back away from the tower and it all still scales properly. These things would have been unthinkable in the past with the previous generation of AR technology.
Download From App Store
AR MeasureKit is one of the smartest applications of the ARKit technology I’ve seen, which is surprising since it’s also an ARKit launch app. This is an application that uses the world tracking technology of ARKit to help you measure stuff in the real world. Want to know how high something is? Maybe if something is level? What about a person’s height? These are all possible with a fair bit of accuracy just using the application. I used the height measurement on a whole bunch of people and the reported height was usually correct to within a centimeter.
The best part of it all is that the measurements are persistent, so as you look around the room they all stick exactly to where they were. It’s a great example of the world tracking technology in action.
Download From App Store
If you know anything about PC-based VR you’ve probably heard of Tilt-Brush. It’s a Vive and Oculus application that lets you use the motion controllers of each system to paint artwork in 3D space. World Brush lets you create drawings in 3D space using an iPhone or iPad. The drawing exists within 3D space and you can walk around it as it sits in fixed relation to the environment.
That’s not even the best part of this application, however. It uses the cloud and geo-tagging so that anyone else who moves through that same space can see your actual drawings. I think the potential of something like World Brush is pretty revolutionary. For one thing, it is one of the first apps to allow users content-creation powers within a shared AR world.
Download From App Store
Are you one of the millions of people worldwide who enjoy playing the Sims? Do you like spending your time buying virtual furniture and then moving it around obsessively. Well then, you are going to love Housecraft. This app provides you with a library of objects that you can place in your real home using ARKit. Thanks to this new technology from Apple, the scale of the virtual furniture actually looks realistic. IKEA has a similar app out too, but in my opinion Housecraft does a better job despite not selling real furniture as does the Swedish retail chain.
Download From App Store
Thomas & Friends Minis
Who hasn’t watched Thomas the Tank Engine as a kid? Like most children, I had an affinity for trains when I was little. I even had a toy train set with real smoke and everything. Now as an adult I don’t have the space or time to play with toy trains anymore, which is why I think this app is pretty cool. Yes, it’s for children and normally I wouldn’t have given it another thought, but the addition of an AR mode makes it worth trying out, even if only for a few minutes. I can only imagine using this technology with a proper head-mounted AR setup and playing with a realistic model train. Honestly, I hope a more serious app is inspired by this first attempt at an AR train set.
Download From App Store
The AR We Deserve
Say what you want about Apple, but they’ve done us all an incredible favor here. Not only have they solved so many of the problems standing in the way of convincing AR, they’ve also done it by using the hardware people already have in their possession. I’ve played with ARKit apps extensively now and there’s no doubt that it finally brings that minimum level of AR technology to make you feel like you’re looking at a real object rather than just a picture on a screen. I find it hard to believe that Apple will be stopping at just a phone or tablet implementation, so don’t be surprised if there’s an Apple AR HMD in our near future.