August 2011 – Pan Studio

Sensory Augmentation: Vision (pt. 1)

Sam Hill

30th August 2011

Blinkered

The above diagram illustrates the full breadth of the electro-magnetic spectrum, from tiny sub-atomic gamma rays to radio waves larger than the earth (there are in fact, no theoretical limits in either direction). That thin technicoloured band of ‘visible light’ is the only bit our human eyes can detect. That’s it. Our visual faculties are blinkered to a 400-800 Terahertz range. And from within these parameters we try as best we can to make sense of our universe.

There is no escaping the fact that our experience of the environment is limited by the capacity of our senses. Our visual, aural, haptic and olfactory systems respond to stimuli – they read “clues” from our environment – from which we piece together a limited interpretation of reality.

So says xkcd:

This limited faculty has suited us fine, to date. But it follows that if we can augment our senses, we can also increase our capacity for experience.

Seeing beyond visible light

Devices do already exist that can process EM sources into data that we can interpret: X-ray machines, UV filters, cargo scanners, black-lights, radar, MRI scanners, night-vision goggles and satellites all exploit EM waves of various frequencies to extend our perceptions. As do infrared thermographic cameras, as made popular by the Predator (1987).

What are the implications of para-light vision?

Let’s for one second ignore a canonical issue with the Predator films – that the aliens sort of had natural thermal vision anyway and pretend they can normally see visible light. Let’s also ignore the technical fact that the shots weren’t captured with a thermal imaging camera (they don’t work well in the rainforest, apparently). Let’s assume instead that we have a boxfresh false-colour infra-red system integrated into a headset, and that human eyes could use it. How effective would it be?

First of all, we’re talking about optical apparatus, something worn passively rather than a tool used actively (such as a camera, or scanner). The design needs special consideration. An x-ray scanner at an airport is an unwieldy piece of kit, but it can feed data to a monitor all day without diminishing the sensory capacity of the airport security staff that use it. They can always look away. If predator vision goggles were in use today, they would be burdened with a problem similar to military-grade “night-vision” goggles.

Predator Vision is not a true sensory augmentation in that it does not *actually* show radiating heat. Instead it piggy-backs off the visible-light capability of the eye and codifies heat emissions into an analogical form that can be made sense of: i.e. false-colour. In order to do so, a whole new competing layer of data must replace or lie above – and so interfere with – any visible light that is already being received.

Predator Vision in the home

For example, let’s task The Predator with a household chore. He must wash the dishes. The predator doesn’t have a dishwasher. There are two perceivable hazards: the first is scolding oneself with hot water, which Predator Vision can detect; the second is cutting oneself on a submerged kitchen knife, which only visible light can identify (assuming the washing up liquid isn’t too bubbly). Infra-red radiation cannot permeate the water’s surface. What is The Predator to do?

He would probably have to toggle between the two – viewing in IR first to get the correct water temperature, then visible light afterwards. But a user-experience specialist will tell you this is not ideal – switching between modes is jarring and inconvenient, and it also means the secondary sense can’t be used in anticipation. A careless Predator in the kitchen might still accidentally burn himself on a forgotten electric cooker ring. The two ideally want to be used in tandem.

What’s the solution?

It’s a tricky one. How can we augment our perception if any attempt to do so is going to compromise what we already have? Trying to relay too much information optically is going to cause too much noise to be decipherable (remember our ultimate goal is to have as much of the EM spectrum perceptible as possible, not just IR). This old TNG clip illustrates the point quite nicely:

Here Geordi claims that he has learned how to “select what I want and disregard the rest”. Given the masking effect of layering information, the ability to “learn” such a skill seems improbable. It seems as likely as, say, someone learning to read all the values from dozens of spreadsheets, overprinted onto one page. However, the idea of ‘selectivity’ is otherwise believable – we already have such a capacity of sorts. Our eyes are not like scanners, nor cameras. We don’t give equal worth to everything we see at once, but rather the brain focuses on what is likely to be salient. This is demonstrable with the following test:

It’s also worth noting the unconscious efforts our optical system makes to enhance visibility. Our irides contract or expand to control the amount of light entering our eyes, and the rod-cells in the retina adjust in low-light conditions to give us that certain degree of night-vision we notice after several minutes in the dark. The lenses of our eyes can be compressed to change their focal length. In other words the eye can calibrate itself autonomously, to an extent, and this should be remembered from a biomimetric perspective.

Option one:

The most immediate answer to para-light vision is a wearable, relatively non-invasive piece of headgear that works through the eye. In order to compensate for an all visible-light output, the headgear would need to work intelligently, with a sympathetic on-board computer. The full scope of this might be difficult to foresee here. Different frequencies of EM radiation might need to be weighted for likely importance – perhaps by default visible light would occupy 60% of total sight, 10% each for IR and UV, and 20% for the remaining wavelengths. A smart system could help make pre-emptive decisions for the viewer on what they might want to know, e.g. maybe only objects radiating heat above 55ºC would be shown to give off infra-red light (our temperature pain threshold). Or maybe different frequencies take over if primary sight is failing. Eye tracking could be used to help the intelligent system make sense of what the viewer is trying to see and respond accordingly. This might fix the toggling-between-modes issue raised earlier.

It’s interesting to wonder what it would mean to perceive radio-bands such as for wi-fi or RFID – obviously, it would be fascinating to observe them in effect, but might their pervasion be over-bearing? Perhaps the data could be presented non-literally, but processed and shown graphically/diagrammatically?

Option Two:

The second, more outlandish option is a cybernetic one. Imagine if new perceptions could be ported directly to the brain, without relying on pre-formed synaptic systems. Completely new senses. Perhaps existing parts of the brain could accept these ported senses. The phenomenon of synesthesia comes to mind, where stimulation of one sensory or cognitive pathway leads to automatic, involuntary experiences in a second sensory or cognitive pathway. Is it possible that in a similar vein the visual cortex could read non-optic information, and would that help us to see several types of information simultaneously but allow us to selectively choose which parts to focus on? If such a segue weren’t possible, would a neural implant bridge the gap?

In Summary

I’ve intentionally only discussed the EM scale here, but of course there are many other forms of data that can be visualised. There might be potential for augmenting vision with sonar, for example, or miscroscopy. Human-centric metadata deserves a whole post in it’s own right.

It’s difficult to predict how the potential for sensory augmentation will change, but whatever opportunities pioneering science unlocks can be followed up with tactical design consideration to make sure applications are appropriately effective and adoptable. It’s an exciting prospect to think that we may be on the threshold of viewing the world in new, never-before seen ways – and with this new vision there will be, inevitably, new points of inspiration and new ways of thinking.

A Brief Musing On Theme Parks

Chris Waggott

29th August 2011

I’m fascinated by theme parks. Not because of the roller coasters or thrill rides, I can take or leave them…

What gets me going is the childish delight from entering imagined spaces, with their caricatured constructs of fibreglass rocks and hessian sacks.

Image taken from http://www.cwoainsider.co.uk

Images taken from http://www.chessingtonbuzz.co.uk and http://themeuk.net

One of my personal favourites is the Transylvanian area of Chesington world of adventure, with large cartoonish Germanic half-timbered buildings, gargoyles, bat motifs and eerie organ music wafting out of cobwebbed windows.

When walking into one of these areas reality becomes suspended, and a fictional scenario is played out around you through every device possible;

The physical constructs of the space and layout; the sounds that are pumped out of small speakers hidden in the rocks; the strange musty smells; they all add together to create a multi-dimensional story that you can interact with. What’s especially interesting is when this narrative bleeds into the rides – this is something I think deserves looking deeper into.

Image taken from http://www.altontowers.com/

The best example I can think of is ‘Hex’ at Alton Towers.

On it’s own, it’s a rather mediocre ride, but incorporating the queue into the experience takes the visitor on a trip through a semi-fabricated history full of witches and cursed trees. The attention to detail is amazing, with film, sound and set dressing all coming together to give the audience a complete narrative experience.

Over the next couple of months I’m intending to look deeper into the subject of theme parks as examples of multi-dimensional narrative constructs and experience, and discuss people using similar ideas in their work such as Brendon Walker and his Thrill Laboratory, but for now I’ll leave you with the radness of the film from Hex:

Apocalust

Sam Hill

14th August 2011

During the start of Riot Week 2011, when many of us were darting wide-eyed between Twitter and rolling news coverage, there was a undeniable feeling of uncertainty. Obviously order was tenuously regained within a matter of days, but because it was difficult to rationalise a reason for the riots starting in the first place, it was even harder to understand when they would stop. Looting and arson seemed to be breaking out in locations arbitrarily. Why exactly were people raiding high streets in Birmingham, Liverpool or Bristol because of the activity going on in Tottenham?

A few people on Facebook and Twitter were quick to make wry allusions toward classic apocalyptic tropes. A comment like “It starts with isolated events…” might earn itself a dozen or so likes and replies within minutes. The implication was, invariably, that these riots were the media misinterpreting the dawn of a zombie/ ‘crazie’ uprising. It was an excuse for some people – the ones with Walking Dead collections and healthy Gamerscores – to fall, misty eyed, into a state of mental preparedness.

It’s hard to calculate precisely what proportion of people have had, at one time, an “undead survival strategy” (how can the property be secured; what immediately available object could be re-appropriated as a weapon; food and water supplies; who needs ‘saving’; routes out of the city etc. etc.) but it must be enough to have warranted the success of Max Brook’s The Zombie Survival Guide, Pegg and Wright’s Shaun of the Dead and Valve’s Left 4 Dead series.

It’s tempting to predict the future of digital culture has only two certainties: zombies and kittens. It’s difficult to imagine what sort of backlash would be significant enough to counter the cultural inertia that both memes possess. Though the undead seem to have been a perennial example since even before Romero, there are other du jour harbingers of the apocalypse too: aliens, viruses, the weather, the environment, meteorites, magnetism, plants, robots, nuclear war etc.

But where is the appeal? How can the apocalyptic fiction industry be so successful? Why indulge in a mind game in which most of your friends, family and colleagues die a painful, traumatic death? Gross schadenfreude? A fetish for unrestrained commercial consumption? The idea of rebuilding society according to one’s own ideals? Subconscious manifestation of genetic competitiveness?

Perhaps it’s nothing more than a lust for adventure and exploration coupled with a resentment for restrictive aspects of western twentieth century living. We have an enormous legacy of exploration, but no frontiers left to investigate*. To quote The Truman Show: “you’re too late, there’s really nothing left to explore”. Global disaster wipes that slate clean – the familiar becomes unknown; the rules are broken; the old order is gone.

How can apocalust be sated? Well, games, films, literature and television drama seem to be doing a good job already – the demand is satisfactorily met. It might be argued that they’re fuelling the desire too, but in general there are more than enough fantasy frontiers out there to explore in fiction, doomsday related or not. But if the illusion no longer works, then there’s always the potential to go travelling. After all, just because everywhere on earth is known by it’s people, it’s not to say the individuals cannot discover new things for themselves.

* – of course, there are tonnes of frontiers left to explore, scientifically: medicine, astronomy, marine ecology etc. But not a lot is open for the every man, and not in a way so literal as, say, the colonisation of the Americas.

Simple Games, Complex Personal Narratives.

Ben Barker

8th August 2011

When a simple rule or a single sentence is enough to explain a game, it can often lead to its dynamics being richer and more fluid. I played one recently where the sentence went like this:

A person hides and all other players seek, then when a seeker finds the hider he joins them in that space until everyone has found it and done the same. Its called sardines and I had never heard of it.

My experience of it was heightened by playing in an old wooden mansion on the French coast in the pitch black of midnight with a sound track of the sea drowning out the informative shuffles of numerous investigations. The idea of being the last seeker in that black house with its multi-roomed basement, sprawling balconies, and creaking staircase leading to a separate annex was simply not an option. In other words, something was at stake.

The games appeal comes in what you might call its personal story arc. Each seeker is in a wrestle between needing the safety of the group to ensure they aren’t left alone at the end (a genuine fear) and the personal victory of finding the target. Within this there is so much room for loyalty and betrayal, honesty and selfishness, all morally acceptable because the game doesn’t say you can’t. In a sense the rules of the game are far more complicated than that one sentence, they are the rules of human morality, reconfigured on the fly in a heightened state of tension.

One moment stood out for me and encapsulated the experience.

We had been seeking for 30 minutes with no success. Then as paths crossed around the house, word started to spread that one seeker hadn’t been seen recently. A friend has become an enemy.

People began to group at the bottom of the staircase to the annex, it’s pitch black, frustration is beginning to show. No one really wants to go back to the dark basement or the shadow splattered garage. Whilst strategy is discussed I subtly rummage in the space under stairs. It’s deep and I feel a hoover, then a rolled up carpet, behind that is a mop, then something bouncy and hairy, under that there’s a forehead, eyes. I trace back over the still head, the shape of a female face, the hair in a bun.

Everything’s changed, I’m one of them. I’m suddenly an enemy among friends, my hand on the key. I have a moment to decide before others think to check the dark triangle under the stairs. Do I want to bring the group of seekers with me, effectively ending the game, or do I want to misdirect, suggest searching elsewhere, say that I know where they’re hidden, then hang back and dive into this cavernous alcove and become part of the select few. I scan the grey silhouettes of the seekers around me. I’ve searched side by side with a couple of them since the beginning. People are starting to fidget, my mind whirrs, decision time:

‘Guys, I’ve got an idea where they are.’

Playmakers

Sam Hill

Playmakers is a film on immersive gaming – a collaborative project between Hide & Seek, NESTA and ThinkPublic.

It features a good range of speakers from the sector and contains some interesting insights. It also demonstrates (in a surprisingly frank way) the ad hoc and experimental nature of immersive game development, illustrating why it’s important to remember the KISS principle when orchestrating events that are designed to be engaging and fun.

Some of the other difficulties and objectives of experiential events are outlined – the need to avoid esoterism, the importance of having objectives and narrative seamlessly work together and the psychology of keeping players immersed. Interesting parallels are drawn throughout to other social activities that “share DNA” – protests, carnivals, parkour and theatre. Obviously, theatre is a biggy. Computer games are slightly conspicuous in their exclusion.

An interesting idea is presented near the beginning of the film [01:50] by Hide & Seek’s Alex Fleetwood. He appears to be describing the four quadrants of their interest. Here it is visualised:

It’s a really nice territory for enquiry. What’s more, Alex’s criteria reveal a robust set of parameters when extrapolated:

Immersive gaming incorporates a dedicated core of researchers and creative thinkers. The industry seems to be gaining plenty of momentum and makes an excellent case study for the development of a broader experience culture. A lot can be learned from a decade of keenly analysed experiential events and the potential within the sector remains huge and continuously changing.

Terror in gaming: Amnesia – The Dark Descent

Sam Hill

4th August 2011

Recently I wrote about how the active consumption of computer games makes them far more immersive than other creative mediums. Amnesia (2010) illustrates this point really nicely.

The game is a “horror survival adventure” created by indie developer Frictional Games and has been critically well received as well as being commercially successful – which is impressive for an independent title.

Viewed in the first-person, the player must solve logic puzzles in the labyrinthine passages of a Prussian castle. Deformed, humanoid monsters haunt the corridors and, in a move that defies convention for the genre, the protagonist is completely unarmed. The only way to survive in the game is to outrun the creatures and hide in the shadows whilst they search for you.

The overall experience is incredibly dark – the encounters are partly random which coupled with the gnawing suspense and powerlessness keep the player constantly on edge.

This is how immersive the gaming experience can be:

It really demonstrates the power of interactive media. With a horror film you can always look away during the gory bits, or shout “don’t go through THAT door” at the protagonist; there are many learned coping strategies for divorcing yourself emotionally from the action. But an interactive title will wait patiently until you walk through that door, and anyone closing their eyes during the action isn’t going to get very far.

The strangest thing about this video, I think, is that this guy keeps on playing even though he seems petrified. I have to be honest, after playing the game for about fifteen minutes, alone, in the dark, I had to pack it in because of wracked nerves. Why is it then, that I decided the best thing to do was to stop, when this guy just powered on? What distinguishes us? Am I more cowardly? Only, I wasn’t screaming or whimpering when I played. There’s something driving him and I’ve missed it. Maybe it’s a greater need to complete the objective, or maybe though he expresses fear more readily, this guy actually has a higher threshold for it.

As for other titles, Alone in the Dark, the 1992 version, is the first brown-trouser title I remember playing. It was necessary to push a wardrobe in front of a window within the first few minutes to avoid being eaten by werewolves. As a child, that shook me up for weeks.

The Dead ‘Event Horizon’ Space titles are often cited as being pretty scary. I found them even more demoralising because on the PC you can’t cheat your way through the fear with F9-F5 quicksave hopping, which really ramps up the jeopardy. The Resident Evil series, Doom 3 and to an extent Bioshock also come to mind as games that really bring the heart into the throat when the ammunition starts to dwindle.

The mind boggles at what terrors the gaming industry might launch in the future.

The Secret Life of Buildings – “Home”

Sam Hill

1st August 2011

An interesting enough quest-u-mentary into improving housing standards in the UK, the first part of The Secret Life of Buildings takes about half an hour to get interesting, from an experiential viewpoint. Unless you’re happy to sit through some fairly contrived experiments to “prove” that 1) lack of sunlight is unhealthy, 2) people are unhappy living in cramped spaces, 3) the golden ratio looks nice.

However from the mid-point onwards the researchers pull up a couple of nice case studies.

The first is a West London drop-in centre for cancer victims, Maggies.

The architects responsible made tactful decisions on materials to give a homely feel that “inspires, relaxes and stimulates” – using inexpensive softwoods, strong colours and polished concrete to create an environment with a warmth that stands apart from the “stark, alienating spaces”, sterility and cheap plastic nature of hospitals and surgeries. The effect this environment has upon the people that use it makes some compelling further arguments for the commonsensical but often forgotten psychology of materials theory outlined by D. Norman^[1] – that the appearance, finish and textures chosen for a designed object or space will affect how people feel whilst using it. This is a very good environment, in other words, to temper a bad experience.

The other relevant case study is the 1920’s Rietveld Schröderhuis, a playful Mondrian-esque structure that avoids fixed corners and has flexible internal walls for re-defining spaces through-out the day (open plan in the morning, enclosed in the evening). The mood of the house can therefore be changed enormously according to the needs of the people within it. An attribute, the programme notes, that most UK housing is lacking enormously.

The series has a further two parts – one relating to work spaces and one to leisure buildings. The third episode, likely to be aired 8pm August 15th should be considered a must-see.

[EDIT:

The third part of the series is largely criticized the corporate-funded, superstar-architect-designed pavilions, galleries and museums that have more going on the outside than the inside. I’d agree that public spaces should show more consideration for the way they’re used, and be designed to be more engaging than passively observed. However, I’d disagree with the implication that buildings can either be impressive but alienating, or “undesigned” and engaging; that we only feel comfortable in spaces is if we can imprint our own identity on them. I see no reason why a structure, or any other piece of design, cannot simultaneously be sensationally stimulating and conducive to engagement.]

[1] Norman, D (2002). The Design of everyday Things. USA: basic books. p9.