If you’ve spent any time listening to the conversations in the modern VR scene, you may have heard the term “presence” repeated over and over again. It’s talked about as if it’s the holy grail of VR. That’s because in many ways it really is that important to what makes VR tick. So if you’re wondering what exactly “presence” is but have been too embarrassed to ask, here come the answers.
Taking the Red Pill?
In the movie The Matrix, the hero is trapped in a virtual world meant to keep humanity a prisoner. A man named Morpheus offers him the choice of escaping this false reality by drinking a “red pill” or staying in the dream world with the “blue pill”.
The question of how we know what’s real and what isn’t is one that has occupied philosophers and drunk university students for centuries. The problem is that none of us have any actual proof that the things we experience come from some sort of reality. Our brains receive signals that we interpret as sensory data. If that sensory data is what the brain expects to receive, then it doesn’t matter if it comes from the real world or if it’s simulated in some way.
That combination of correct signals to the brain that triggers the feeling of something being real is what the VR industry refers to as “presence”. It’s that inescapable feeling that you really are where your eyes and ears say that you are. Even if you know what you are seeing is an illusion, there’s no way to shake the actual feeling, just as knowing something is an optical illusion or a magic trick doesn’t suddenly make the illusion itself disappear.
Fake Like a Three Dollar Bill
A sense of “fakeness” has been the Achilles heel of VR since the very beginning. Obviously the crude vector graphics, inaccurate head tracking, and incredibly janky frame rate were never going to fool a sophisticated structure like the brain. Even as these things improved over time, that threshold seemed out of reach. But sometime after 2014 the issue was essentially cracked and that’s one of the main reasons why today’s VR is so good compared to anything that’s come before.
Even so, the recipe for achieving presence isn’t simple to bake into a VR experience. Maybe that’s why companies like Oculus, which did the bulk of the research and development work needed to achieve presence, don’t really mind explaining what the ingredients are; simply knowing the ingredients doesn’t mean you’ll make something yummy.
So what are the ingredients of presence? Let’s go over them one by one.
Premium Head Tracking
The quality and accuracy of head-tracking is a key component to achieving VR presence. Oculus and HTC have found that the tracking has to be accurate to the sub-millimeter level before it feels natural and real. Not only this, when the object that’s being tracked (such as your head or hands) is not moving there must be no false tracking. Earlier tracking technology has something known as “jitter”, which meant that even when you held completely still there would still be “noise” in the movement data. This is one area where there seems to be no compromise, and it’s why the premium systems use special external IR trackers.
The final piece of the head tracking puzzle is the axes that your movements can be tracked. The gold standard is six degrees of freedom. In other words, tracking along the X,Y and Z axes all at the same time.
External tracking has also been the key to allowing for this sort of multi-axial tracking. Most mobile VR solutions don’t have six degrees of freedom. You can look around, but you can step in more closely. The motion sensors inside the phone can’t yet detect that lateral head motion without some sort of help.
Interestingly, the new generation of standalone mobile VR HMD (head-mounted display) seems to have solved this with very advanced external sensors. In other words, there are cameras on the HMD that track the room, so it doesn’t matter where you are.
That Old Enemy – Latency
Latency, lag, delay – whatever you want to call it, it’s the time difference between when you initiate something and when the system responds. If you turn your head and it takes a second or two for the scenery to catch up to that movement, you’re not only going to feel sick, you also won’t feel presence.
The work that the big VR researchers have done over the last few years seems to indicate the total latency from where the user moves to where the screen reflects that movement. The correct industry term for this is “motion to photon latency” and presence is only possible if that figure is 20 milliseconds or less. That’s how sensitive we are to lag – more than 20 one thousandths of a second and the brain is no longer tricked.
Achieving this level of accuracy has probably been one the greatest technical challenges faced in the drive to make VR something you’d want to use. It’s meant pushing modern electronics to their limit and aggressively minimizing the path that all that data has to take. It’s not about making one thing perform much better, but shaving off milli- and nano-seconds from each step in the chain until it all runs quickly enough.
Not So Persistent
The next obstacle to achieving presence in VR has to do with the actual display technology. The tiny, high-resolution flat panel displays we get in phones are a good start, but they aren’t good enough for the true illusion of persistence. That’s one of the reasons why smartphone-based VR is often so fake-feeling.
Premium VR displays need to have a low “persistence” to prevent afterimages. The pixels in a display need to take less than three milliseconds to turn on and then completely off again. Any slower than this and the picture won’t keep up.
The refresh rate is also very important. This is the speed with which the display can change all the information on it. Most screens these days have a refresh rate of 60Hz, although some high-end computer monitors can go much faster. A cinematic movie only has to change picture about 23 times a second, which is also referred to as “frames per second”. A 60Hz screen can show videos that have 60 complete frames for every second of footage. For proper VR with the illusion of presence, you need at least 90 frames a second. So that means the panel must have a refresh rate of 90Hz.
Persistence is also a factor in motion-to-photon latency, so it’s just better overall to minimize it.
A Fine Little Spot
It doesn’t really matter that you have all this low-latency, fluid, and accurate motion if everything looks like a newspaper through a magnifying glass. When you look out at the real world you don’t see in pixels – you see the world as a crisp, continuous photo-image captured by your retina.
Unfortunately, computer displays don’t work like that and have a grid of microscopic picture elements, or “pixels”. If you hold your phone’s screen really close to your eye you’ll make out those individual little dots, but modern phone screens have such a high resolution that at normal viewing distances those pixels are invisible.
The problem is that in a VR headset the screen is only an inch or two from your eyes, which means the density of the pixels has to be really high to avoid seeing them. Oculus says the prerequisite for presence is 1000×1000 pixels for each eye; this is why we see so many VR headsets with dual 1080p panels, since they’re dense enough and that’s a very common panel resolution. Already, however, new VR HMDs are sporting resolutions of 2048×1080 per eye.
Through the Looking Glass
The final piece of the puzzle is, of course, the optical system. If for some crazy reason you’ve tried, you’ll know that you can’t just tape a phone to the front of your face and then suddenly have a VR experience.
First of all, there’s no way you can actually focus on a screen that’s held that close to your eye. One of the functions of the lenses in a VR HMD is to make you incredibly nearsighted so that the displays are in perfect focus. They’re also meant to bend the picture so that it fills more of your visual field, which is why if you look at the images on the screen without a lens it looks distorted. The lenses bend the light so that it fills not only your central vision but also a significant part of your peripheral vision.
Humans have a field of vision about 270 degrees wide, but if the VR headset can reproduce a field of view at least 90 degrees wide, then presence can be achieved. Most mobile VR headsets achieve this easily since the lenses are a relatively simple component for that level of optical achievement. Most premium VR solutions provide at least 100 degrees and there are products on the market now that can provide 110 degrees or more.
The lenses must also be incredibly precise to prevent presence-spoiling distortion and unwanted focus issues.
You Had To Be There
There you have it – all the key ingredients you need to fool the average human brain into believing it’s somewhere where it’s not. Now, just because we know what the minimum requirements are doesn’t mean that everyone can now pack up and call it a day. There is still a lot of room for improvement when it comes to each of these aspects of VR. That’s not even counting all the things that could still be added on – developments in touch-feedback (haptics), full-body motion tracking, heat and cold simulation, smell reproduction, and so much more. While there is a threshold below which presence is basically not, er, present, that doesn’t mean presence is a binary experience. It’s one of degree and this is only the beginning.