social vr

Social VR: What’s It All For?

We are all very excited about virtual reality. It’s an amazing experience backed by amazing technology. But when the excitement has died down and everyone is used to VR, what are we actually meant to use it for?

Of course, VR gaming is the application everyone thinks of first. It’s also likely to be around for as long as VR. The gaming industry is so big that even relatively niche parts of it can survive with relative ease. At the moment VR gaming may not be mainstream, but it certainly isn’t niche either. The main problem with VR gaming being the main industry supporting VR is the expense inherent to VR gaming systems. Even the low-end of AAA VR is pretty pricey; the Sony PSVR costs about as much as a PS4 console, which means that most only the small fringe of truly hardcore gamers is going to invest in VR. As VR hardware costs go down this might change. As I write this, however, only one in sixty PS4 owners have a PSVR. If we look at PC VR, the vast majority of gaming PC owners don’t have a machine capable of handling VR. Of those, it’s again an even smaller subset who have HMDs.

social vr man on chair

If not gaming, then what? Virtual reality has some excellent educational and training applications. In fact, I personally find the educational potential of VR to be one of its most exciting possibilities. Education is a huge industry, worth billions, but it will be a while before VR devices can scale in a cost-efficient way to broadly apply to learning. For now, science fields and industries that benefit from hands-on experiential training are pouncing on VR, but we are a long way from every student packing an HMD in their kit.

Productivity is another intriguing possibility for virtual reality. One of my favorite Oculus apps is a virtual desktop program I found on Steam. Basically, it puts you in a 3D room with your choice of virtual monitors. Big ones, small ones – there’s quite a bit of flexibility.

For someone like me who doesn’t have to look at the keyboard to type, it provides a focused work space that’s also quiet and, above all, private. The problem here is not so much VR, but its cousin AR. I really believe that physical screens are going the way of the dodo in future, but it will be AR that makes that happen, and not VR.

There is, however, one area of VR that is set to be its most popular segment. Billions have been spent on it already and things haven’t even started yet. Social VR might just be the killer app that VR needs to become mainstream.

What is Social VR?

Just like social media, social VR includes VR experiences that are explicitly designed to facilitate people interacting with each other. This could be virtual chat rooms, parties, community games, and anything else you can think of.

Think of social media such as Facebook and Twitter, and chat programs like Hangouts or Skype – the same uses, but translated into a VR environment. The industry believes strongly in the social appeal of VR versus chatting via text or video. So much so that social media giant Facebook bought out Oculus for between two and three billion dollars, depending on who you ask. That’s a huge endorsement of the technology as a new way for people to get connected. But why do they care so much?

Don’t You Need Somebody To Love?

That’s the way that Jefferson Airplane put it, back in the day. It’s true, we do all need someone to love. Human beings all have a need for social interaction. It’s our monkey troupe ancestry that drives that need for contact and interaction. We like to stick our noses in each other’s business. People are always dynamically forming groups and building intricate social hierarchies.

We communicate on multiple levels at the same time. Spoken language, as complex as it is, only makes up a fraction of the total communication bandwidth of in-person communication. Body language, vocal tone, facial expressions, and other physical aspects of communication all intersect to generate the total channel of information we send over to the other person in the conversation.

In the early days of telecommunication, that was stripped down to the bare minimum. Telegrams were short, terse text messages.

social vr facebook

Tell John STOP
Grandma sick STOP
Come quick STOP

It’s not exactly a flowery speech, but being able to send those (expensive!) words clear across the country in minutes was a revolution. Then came telephones, where you could hear the other person’s voice; hear emotion and feel much closer to them.

Digital communication over the internet also started off as being text-based. With internet relay chat (IRC) we could talk to anyone in the world, as long as it was text. The popularity of text chat also shows up those limits of communication.

It’s very difficult to convey sarcasm in text for, for example. So people have begun to invent new text elements that do the job. For example, you may follow a sarcastic comment with “/s” so that everyone knows you aren’t serious. It’s also why we use emojis, those little happy or sad faces. They started as cheeky repurposing of ASCII symbols, but now with modern smartphones we have elaborate emojis that can even be animated.

Video chat programs like Skype have brought the face and voice together. It’s much more like really sitting across from the person you are speaking to. Just 10 years ago this was a Sci-Fi concept. I still remember how blown away I was by my first video call. Now I Skype with people on a daily basis like it’s nothing.

I Need My Space

None of these communication methods really make it feel as if the other person is sharing the same space with you. A feeling of presence is what’s been missing with our telecommunication methods from the very beginning. No matter how sharp the picture, how clear the audio, or how short the latency, it always feels like there is a large distance between you and everyone else in the conversation. When you hang out on Facebook with your pals, it doesn’t feel as good as sharing a couch and watching a movie together. In social VR applications your friends are there. They may be represented by avatars, but if done well it can feel like you really are hanging out together.

In a world where people experience the paradox of being connected, but feel socially isolated, social VR is set to explode in popularity.

Getting in on the Action

That’s the theory, but how long before we can actually try it? The good news is if you have a smartphone and a mobile HMD you can try social VR right now. Many of these programs don’t even require the HMD. It’s a concession to the many mobile users who may not have VR HMDs yet. You can still traverse the virtual social scene using just your phone.

Here are some of the most prominent social VR apps that are either coming soon in beta or already available for play. In general these are usually free (ad-supported) and, as long as you have a fairly modern phone, should run pretty well.

Facebook Spaces (Beta)

facebook spaces

Facebook Spaces is a key part of the plan that started with the purchase of Oculus. Zuckerberg and friends clearly see the Facebookites of the future lounging around its Spaces virtual world. The other part of the puzzle is their Oculus Go headset – a standalone VR headset announced mid-2017 that costs only $200. It doesn’t need a phone or computer, but just exists as a standalone mobile VR solution. That gives Facebook a platform both in the hardware and software sense.

Spaces lets those without VR gear participate with those who are wearing a headset. You can start a live broadcast in VR and others will be able to watch it on tablets or phones. Conversely, you can receive Facebook Messenger video calls in VR, with the feed appearing as a floating screen. Spaces also lets people who are hanging out together in VR call up and discuss the media on their Facebook account. So you could have a chat about the photos and videos that have been accumulating over the years.

In a stroke of pure genius, you can also create new pictures in the VR world with selfies of your avatars doing various activities in the VR world. Spaces also includes activities where you can create VR media – you can collaboratively draw 3D objects in the shared space and generally mess about with your buddies.

One thing that has to be noted about Facebook Spaces is that it currently only runs on the Oculus Rift and requires the Touch Controllers. So this is not something everyone can just jump into right now.

AltSpace VR

altspaceVR logo

AltSpace VR is probably one of the best-known social VR applications out there. The company went through a rough financial patch in 2017 and went as far as announcing that it would be closing down, but in the nick of time good old Microsoft swooped in with its massive vault of cash and bought them. It’s a good thing too, because it would be a real pity to lose one of the best-put-together social VR apps today.

AltSpace is available on quite a few HMDs, although, sadly, the Google Cardboard is not one of them. If you have Google Daydream, Htc Vive, Oculus Rift, and Gear VR you can hop right into the AltSpace world right now. If you don’t have any of those, then you can still try it out using “2D mode” on your phone. AltSpace is an intuitive and comfortable experience. The spaces are nice to be in and it’s easy to just walk around and chat with people who are hanging around. Altspace is at the top of my list as a recommendation for people to try.

Oculus Rooms

oculus logo

Oculus Rooms is another Gear VR application and is much as you’d expect, based on the name. It provides a social space where you can hang out with other people. It’s possible to do all sorts of shared activities, such as watching films together or playing games. It also acts as a sort of lobby for other multiplayer Gear VR apps, where you can decide what to play together and then launch the software from there.

There’s a fair bit of personalization in Rooms. You can decorate and style your room to your satisfaction before inviting anyone over. That’s quite a cool feature. Apps such as AltSpace just let you hang out in a premade space, so it always feels public. Rooms is designed in such a way that it feels like inviting people to your home. For that reason I think it is quite special. It also helps that it is one of the best looking social VR apps on mobile!

Project Sansar

sansar

Before all this social VR malarkey, there was Second Life. If you have somehow never heard of Second Life, it’s a shared 3D virtual world that people visit using their plain old computers. Second Life is still running to this very day and has become popular as a virtual space to do everything from hang out with people to doing product launches or giving lectures.

The times are moving on, though, and the people behind Second Life, Linden Labs, are ready for a modern stab at social virtual reality. That’s coming in the form of “Sansar”, a new high-end social VR platform for which Linden has huge plans. Sansar is about community-driven social VR creations. It’s meant to be a virtual space where Linden provides the tools, and the public makes stuff with it. They will let you import assets from most of the major 3D modeling packages and straight up sell assets in an online store. Sansar is also going to provide a way for people to monetize their content, much in the same way that people earn a living making content for YouTube.

The Dark Side of Social VR

Social VR looks like it really could be the next big thing that makes VR a permanent fixture in daily life, but that doesn’t mean it will all be good news. “Normal” social media has already been incredibly disruptive to society. Facebook has been implicated in certain types of depression, as one example. It turns out that people are very careful to curate how they portray themselves on social media. They only show the good things that are happening. Consequently it looks as if everyone in your social circle is living a much better life than you. After all, YOU know about all the not-so-nice things that happen in your life. They happen to everyone, but Facebook facilitates the creation of false realities. The end result is that people feel worse about themselves.

There’s also the specter of cyber bullying. Driven by anonymity and the fact that a significant percentage of the population is always going to be made up of, well, douchebags, there’s no reason to think that we won’t see it in VR. People can already feel severely victimized just with text and pictures; imagine what will happen when you can also feel the presence of a cyber bully. It’s not as if social media companies like Facebook are not aware of this issue, but I still haven’t really seen how they’ll address it effectively.

Apart from that, we might also see people develop an unhealthy obsession with social VR. It happens all the time, whether it’s TV, video games, or even books. Some will take it to unhealthy extremes.

Pro-Social Attitudes

Let’s not get mired in negativity, however. The internet might be the poster-child for technological loneliness, but through VR we might be able to bring back a true sense of community.

I remember the times I spent on fan forums when I was a teen. We organized real life meetups whenever we could, but even just a text-based forum was already an amazing way to find your people and spend time with them. A presence-inducing VR system would have transformed that experience and I, for one, am glad that we are the generation that gets to experience this technology first.


playstation vr

VR Positional Audio is a Forgotten Star

For most people, virtual reality is mainly a visual idea. It’s a way to beam realistic pictures to your eyes so that you feel like you're really in the virtual environment where you have been placed. However, as I’ve outlined in various other articles on this site, VR is actually an intricate dance of various factors that work together to craft the illusion of virtual presence.

One of the key factors in creating a truly convincing and immersive VR experience is having audio that syncs up with what your brain expects to hear. It may not seem like a big deal, but you’ve surely noticed that the real world does not sound like a movie soundtrack played over a pair of stereo headphones. Your brain uses the slight time delay between a sound reaching one ear and then the other to accurately and almost instantly figure out where its position is relative to you. That’s why you can tell if a sound is coming from above, behind, or below you.

Of course, your brain has access to a bunch of other sources of information too, such as how the sound changes as you move your head, and visual cues as well. Most of this auditory processing happens on a subconscious level so we aren’t aware of it, but just about anyone can tell that the sound we typically get from a recording played back on a pair of headphones sounds “fake” compared to real-world audio.

Simulating Position and Direction in Audio Playback

One of the ways that have been put into use when it comes to truly simulating where a sound is coming from using a speaker system is to use a multi-speaker surround-sound setup. If you are surrounded by speakers then it’s relatively easy to make it sound like, for example, there’s a bird chirping above and behind your right-hand side. Unfortunately, multi-speaker audio systems are quite expensive and complex. Most home users typically have a system with five, or perhaps seven, speakers. It’s also not terribly practical for VR use since you’ll also hear all the other sounds that are not part of the VR simulation.

Still, when you think about it, you only have two “microphones” – in the form of your ears. So surely there must be a way to simulate the positional audio signal that your brain can interpret and perceive as a real sound coming from a real location in the space around you. Audio engineers have been working on this problem for a long time and it has given rise to a special recording technique known as “binaural” recording.

Not Stereo

Binaural recording is a recording meant for two ears. That may seem like a sort of dumb way to put it, since we use our two ears to listen to everything, after all. What I mean, however, is that a binaural recording is meant to simulate what two ears would have heard had they been present at the time of the recording.

In other words, this is not simply two channels of audio, which is all stereo is. When a stereo track is mixed, all the recording engineer really has control over is how loud and prominent each element of the multitrack recording is in each channel. If a sound is perfectly balanced then you should hear it sort of in the “middle” of your head. The more the sound is pushed to the left or right speaker, in terms of volume balance, the more you’ll hear it on your left or right side. Traditional stereo mixing and recording can really create the right sound to convince you of a sound's origin in a spatial sense.

binaural recording for positional audio

It’s the Head, Dummy

That’s mainly because what you hear coming into your ears is actually a complex acoustic product. For example, you don’t hear sound just as it enters your ear holes. Instead you also hear sound translated from the inside of your skull, where it all mixes together. This is one of the reasons why we often don’t recognize ourselves on a recording. We hear our own voices as a mix of internal and external sound that’s not reproduced on a recording at all.

The shape of our ears also plays an important role. The ear shell is a complex acoustic funnel. If you’ve ever been in an auditorium or an opera house you may have noticed the complex acoustic panels they use to reflect and redirect sound. All the curves and angles of your outer ear do a similar job. This means that your brain expects to hear sounds that have been put through this part of the process when figuring out where a sound is coming from. The problem is that when you are wearing headphones these natural acoustics of your hearing system are partly bypassed. The sound is pumped directly your ear canal, where you hear it as an undiluted stream of sound.

It’s a Dummy Head

Traditional binaural recording aims to recreate the acoustics of the head and ears that your inner ear and brain expect, by literally using a dummy head with microphones where your ears would be. This way the distance between the “ears” is correct and the model ear shells do similar things to the sound as your own ears would.

So if we wanted to make it sound as if you were really standing in the middle of a live band, or a nature scene, you just have to set up the real-world situation and then plonk your recording dummy head in the spot where you want the listener to feel they are when they play back the recording.

There’s just one catch – the positions of the audio sources are all fixed relative to your dummy head recording system. So that’s not much use for us in VR, because we want to move our heads around and have the sound stay where it’s supposed to be, relative to our heads.

The VR Positional Audio Conundrum

The virtual reality world often portrays an environment that is dynamic in nature. When it comes to the audio that you hear, this means you’re listening to not a single recording but many small recordings, and sometimes even completely generated (i.e. “made up”) sounds.

Let’s say there’s a virtual bee flying around your head. Unless the bee takes a pre-recorded path around your head, you’ll need some way of making the positional audio work, without having the luxury of perfectly controlling the position of your head as you listen. In other words, you need to generate or modify the sound in such a way that things in the VR world not only sound as if they are where they seem to be, but that their position in audio space changes correctly as the position of your virtual ears changes.

The Secret Sauce

Different VR experience providers each have their own “secret sauce” algorithm and software developer kit aimed at convincing us the origins of VR sounds really are where they appear to be. Regardless of how they achieve the end result, the basics are bound to be the same. These systems have to take three things into account when generating true spatial sound.

First, they have to simulate the difference in the time the sound takes to reach one ear versus the other. So if a virtual pin drops in the room, the computer has to work out exactly when the sound waves reach which parts of the room. If your ear is occupying that space, then it should hear the correct sound at the right time.

Secondly, they have to take into account that each ear should hear the sound at a slightly different volume. After all, the ear facing the sound gets more sound energy than the other.

The last part of the simulation has to reproduce the “spectral filtering” performed by the outer ear. All this refers to is how the shape of the ear highlights or eliminates parts of the sound. That way the audio sounds the way the brain expects it to after passing into the ear canal.

Bringing it All Together

So in the end a spatial audio system is a simulator that takes recorded sound and then simulates how that audio would travel through a room, interact with various surfaces, and then enter your virtual ears. It’s not hard to imagine what such a system has to do, but it’s a technical act of genius. We pay a lot of attention to the (deservedly) hard work that has gone on in the graphics department, but I think the revolution in true spatial audio deserves just as much praise. All these research and interlocking technological solutions come together in an intricate dance just so you can enjoy an aural experience. Hats off to them, I say.


windows mixed reality interface

What is Windows Mixed Reality?

You’ve probably heard the term “mixed reality” come up in conversation about both VR and AR. As you may know, mixed reality refers to a modern, more sophisticated form of augmented reality that combines a suite of environmental sensors, wearable computer hardware, and sophisticated software. A mixed reality system can include full VR, full AR, or other weird in-between blends, such as a sort of reverse AR where real-world objects are displayed in the VR world.

Microsoft, the same people who make Windows and those named “Bill Gates” incredibly rich, are at the forefront of this new revolution in augmented reality. Their Hololens headset made waves when it was first demoed to the public and is still one of the most impressive mixed reality devices you can buy. Of course, to actually own a Hololens you’ll have to part with a small fortune, since it’s mainly aimed at enterprise clients at this point.

window mixed reality headset

Growing the Family

Microsoft has never really been a company that wants to corner the hardware market. They’re all about creating dominant platforms and then inviting everyone else in. One of the main reasons that Windows is such a success is that Microsoft never tried to keep it away from everyone else. Apple didn’t want their operating system running on anything but an Apple, but Windows ran on everything, which means more people wanted to make software for it. It wasn’t necessarily the best operating system, but it was the most useful one thanks to how widespread it became.

Windows Mixed Reality is Microsoft trying to create the same sort of open platform, but this time for the purpose of creating a standard framework for developers to use for their software products. Microsoft provides guidelines on hardware specifications as well as software that helps to interpret a lot of the complicated aspects of mixed reality. This allows developers to concentrate on creating their unique content instead of having to reinvent the mixed reality wheel every time.

Business as Usual

This sort of standardization is both necessary and common in the tech world. That doesn’t mean that Windows Mixed Reality will be the ultimate winner and become the main standard for the technology, but it makes sense for Microsoft to be the one taking the lead here. Apple is on a similar path with their ARkit developer framework; AR will be built into their future phones and other devices from the start.

Microsoft has won at this game before, though. Back in the early days of 3D accelerators there were a number of graphics APIs that were a way to standardize programming for graphics hardware from different manufacturers. In the bad old days you had to write a piece of software for a specific piece of hardware, but with an API you and the hardware makers just make sure you comply with the standard and, viola! – everything works with everything, at least in theory.

Microsoft’s DirectX won out against OpenGL and Glide in the first big standards war in the graphics world. Today DirectX is still dominant, but it’s being challenged again by new standards and it’s not a foregone conclusion that Microsoft will win again.

wmr mixed

Windows Mixed Reality Hardware

Anyway, that’s getting off the topic at hand. So let’s look at the physical manifestation of Windows Mixed Reality.

So far there have been two third-party manufacturers that are bringing Windows Mixed Reality products to market. Acer is releasing a Developer Edition headset, and Lenovo (which bought out IBM’s laptop division) is releasing the Lenovo Explorer, which can also be bought with motion controllers.

Both of these headsets share some common features. The most obvious feature is the pair of front-facing cameras. These lenses are set very widely apart, giving the whole thing a rather comical look. You’ll also notice that both have a sort of “flip up” front, which means you can easily switch between being hooked or not, without having to take off the whole thing – something that Sony has already done with its PSVR.

Since they are both built according to the same general guidelines, there’s not that much to differentiate the headsets, but the Acer unit is one that developers will be building their experiences on. So let’s look at its specifications as a sort of base standard for these headsets.

Running the Numbers

The Acer HMD has two very high-resolution LCD screens. Each LCD has 1440x1440 pixels, which you’ll notice makes them perfectly square. That’s actually a good approach, since single panel VR HMDs get split into two squares anyway and a lot of the dual 1080p models end up wasting some of that real-estate. This dual-panel setup provides a 95-degree horizontal field of view. This is perfectly in line with premium VR headsets and certainly a lot better than the Hololens, which is based on a fancy but still immature projection system. Just as with the Oculus, the refresh rate of the panel in the Acer HMD is 90 Hz and it is tethered with a four meter cable.

One major difference between this headset and something like the Oculus or Vive is the fact that it uses “inside-out” motion tracking. The camera that tracks motion to provide the full six degrees of freedom are the ones mounted on the headset itself. This is actually an amazing development, because it means that room-scale VR is possible without the need for external sensors.

Priced to Sell

These Windows Mixed Reality headsets are set to sell at around $350 apiece, which is an incredibly low starting price. That’s almost half as much as the launch price of the consumer Oculus Rift. It’s no surprise Oculus has been slashing the price of its HMD recently.

These headsets are quality devices with cutting-edge technology and yet they are being sold for an absolute song. Honestly, these HMDs went on my shopping list the very moment they were announced.

acer-windows vr mixed reality headset

Preparation Required

When the Oculus and Vive originally launched, the minimum computer requirements were quite shocking to a lot of people. Since then the requirements have softened a bit and the GPU technology has become a lot cheaper to boot. The minimum requirement for these Windows headsets are basically the same when it comes to premium VR, but in a very smart move there’s a second lower set of requirements for productivity-type VR and AR.

If you have a computer with an integrated GPU equivalent to the Intel HD Graphics 620, the Windows HMD will clock itself down to 60Hz but still work for Windows itself and the more basic applications. That’s a killer feature for a product that’s supposed to be the real deal mainstream VR solution.

Onto a Winner?

It’s very early days for this new platform, but it looks as if Microsoft has thought of everything that’s holding mainstream AR, VR, and MR back. The only real criticism I have so far, based on the tech demos I’ve seen, is that the inside-out tracking can be a little janky. But that’s with beta developer demos, so we’ll see if it’s an inherent issue.

Other than that, I think they are onto a winner here.