Why Bethesda Just Unveiled a Vision of the Future That Goes Way Beyond Gaming
From a touchscreen, VR sword to console buttons — a cross-device game that has given us a vision of the future way beyond gaming.Read more
Virtual reality is almost the definition of an immersive experience. More than any other medium, it completely envelops us in a generated environment, tracking our motion and mapping it to a realistic visual that convinces the mind that we are in a new, different world.
Unfortunately, while it's remarkable that a computer-generated world can immerse us just as totally as an ocean would, there are some people who find both oceans and virtual reality a little nauseating.
One of the things that surprises first-time videographers is how shaky handheld video footage looks. We move our heads and eyes constantly, and yet our normal experience of vision is stable, smooth and sharp. How can there be such a pronounced difference between these two?
The answer is that our brains fuse sensory information from multiple sources to infer how our bodies are moving, and use this to stabilise our visual experience. You can test this by doing a simple experiment: place your fingertip gently on your lower eyelid, then very carefully jiggle it just a little bit. The image in that eye will jiggle along with your finger.
You may find this experience rather disorienting, and this brings us to the reason for motion sickness: because your brain expects all the signals from your body to align with what you see, a mismatch between the two can make your brain think something has gone wrong. One of the possible causes of this mismatch is poisoning; your body may then take an executive decision to rid you of this poison. You, of course, know you are simply between decks on a ship, but this part of your brain doesn't.
The senses your brain uses to measure your motion through the world fall into roughly three categories: vision, force information from parts of the inner ear that are essentially accelerometers, and proprioception (vision, of course, requires no introduction from me).
The vestibular system is part of the delightfully-named labyrinth of the inner ear. It consists of a series of roughly circular channels filled with liquid. Your brain uses the motion of this liquid to give both a sense of the direction of gravity and changes in the position of your head: roughly, as your head rotates, the liquid doesn't, and so it flows through the channels. This flow is measured by sensors in the channels and used to estimate motion.
This, incidentally, is why you can become dizzy if you spin round or if you drink too many cocktails. Spinning eventually makes the fluid swirl around, too: when you stop spinning, the fluid's motion fools your brain into thinking you're moving the other way. Drinking is subtler, but broadly causes the same effect: alcohol is less dense than water, so the liquid becomes lighter, and this messes with the flow sensors, making you dizzy.
Of course, you need quite a lot of alcohol for this to happen. It's easy to see how the body could interpret this kind of dizziness as poisoning (and in this case, it would be right).
Proprioception is perhaps the most obscure sense of the three. It's a kind of muscular dead reckoning: your brain integrates all your muscle motions to work out where your various bits are in space. This is how it knows to compensate for the motion of your eyes, but proprioception also tracks the motion of your whole body. As such, your brain fuses vision with proprioception quite often, to check that your bits are indeed where you left them.
Incidentally, if you're floating in a swimming pool, your limbs may swirl around without muscle input and surprise you by ending up in other places. This may give you a more intuitive sense of what proprioception does.
Motion sickness usually occurs when you are in a moving vehicle and your brain tries to match your motion relative to the vehicle with the forces you can feel. It observes a mismatch, attributes this to poisoning, and there you are.
Early virtual reality had a rather similar problem. In the 1990s, when these experiences first began to be built, available computing power meant the image could lag perceptibly behind the motion of the user's head. Additionally, frame rates might be slow enough to cause a subliminal feeling of jerkiness as people moved around. This eventually led to the demise of the technology for around twenty years while graphics processing caught up.
However, though VR is now incomparably better than it was then, this is not the only problem that can occur. If you're in a magnificent, immersive world, you're going to want to move around in it. Unfortunately, you're probably actually in quite a small room. This means experience designers have to find ways to fake movement. The appearance of motion without any accompanying forces is the exact inverse of the problem experienced between decks on a ship; the effect on a susceptible viewer is exactly the same.
It's clear that a great VR experience has to deal with this mismatch between what you see and what you feel. At this point, it could seem biology has presented us with an insurmountable problem. Fortunately, physics has a few tricks up its sleeves.
Newton was the first scientist to give us a clear, universal theory of forces and motion. His theory is so effective that despite being superseded by more advanced theories at the beginning of the twentieth century, it was Newton's laws of motion that were used in calculations for the moon landing.
Newton's second law of motion links force to acceleration. Broadly, when you push something, it gets faster or slower, or it changes direction.
However, Newton didn't just give us physics: he also (jointly with Leibniz) gave us calculus. This can be thought of as the study of change, and we will use this concept to understand motion in VR.
Movement is a change of position; the measure of movement is speed (or, more precisely, velocity—which is speed with a direction). Acceleration is a change of velocity, and accelerations tell us about the forces acting on us. If we can find a way to fool the mind into ignoring the lack of acceleration, perhaps we can make VR a more palatable experience.
Let's first consider something that starts still, then moves off at a constant speed. Here is a graph showing its position, velocity and acceleration (thanks to the majesty of Google Sheets):
Position looks nice and smooth. Velocity, though, suddenly steps up halfway through. We can see this as a huge spike on the acceleration graph and a jostle on jerkiness—the rate of change of acceleration.
Someone in a VR experience where the viewpoint moved like this would find it jarring, because of course they would experience no forces at all.
Is there a better solution? Well, let's see the same graphs for a sine curve
You'll notice they all look the same. This is because they are. The rate of change of a sine function is another sine function—but shifted along a little. (This is a really handy property of sine functions).
At least one VR game has used this fact to its advantage: eleVR, made by the excellent Vi Hart. She has a mathematics background, so used this skill to make sure all the motion in their game is as smooth as Stephen Fry in a white tuxedo. Make sure the movement is not too fast, and the lack of force can become barely perceptible.
Other functions can give the same smoothness. Your friendly neighbourhood mathematician can help you choose some. (A few bonus functions are shown in the Google Sheet for this.)
Of course, Newton's other other contribution to the history of physics was as easy as falling off a log. Gravity is the one force we experience constantly. This is exploited by simulator rides on giant hydraulic stilts, which turned up, coincidentally, about the same time as the first wave of VR games, with The Funtastic World of Hanna-Barbera opening in 1990.
Putting people in moving chairs while they experience VR can mimic these rides at a much smaller scale, and is something we have done ourselves for Audi at their Goodwood stand in 2015:
However, another, subtler idea is to use a slight tilt in the visual horizon to give someone the sense that they are leaning against an acceleration—much as you might on a moving train. Combined with the subtler, smoother motion of a sine or similar function, you can fool people into thinking that they are standing on a slowly accelerating platform.
Virtual Reality has an unprecedented capacity to give us a truly immersive, absorbing and realistic experience, where we can explore a virtual world in complete freedom. For this exact reason, there's an understandable temptation to make a VR experience as breathtaking and dramatic as possible, using the same tricks movie directors use to introduce excitement. I hope I've shown you why, counterintuitively, this can actually make a user more aware of the fact that what they are seeing is not reality, because it feels different.
The fact is, movement is an intrinsic part of any VR experience: the VR equipment is measuring motion in much the same way your body does, and presenting you with a visual to match. The best approach is to work with the body's natural movements and senses, and be careful not to work against them.
By thinking of VR as a full-body, physical experience rather than just a feast for the eyes, we can make experiences that truly involve the user without causing this problem.
A Roman maxim, ars est celare artem, tells us that true art is in hiding art: a great artist's technical prowess can draw attention to the subject, rather than to itself. This holds doubly true for virtual reality—and with a little science, true art is well within our grasp.