Video Games for plants

I have been developing a video game for a houseplant to play.

I found a circuit online that uses Galvanic Skin Response to translate changes in conductance into midi notes. I soldered it together and I have been using it to play video games with my skin using an open-source program called ‘MidiKey2Key’ which translates incoming midi notes into any kind of keyboard button presses. I also use Ableton Live to output the midi notes to a particular octave and not the entire scale (this is also useful with the sonification of these elements). Interestingly this process also works on plants of any sort, although I found it works better on some more than others.

Signals arrive via TENS electrodes (green) and the changes detected in the electrodes are translated and output as midi data (purple)

After ‘training’ the plant (slowly adjusting the signals and assigned button presses, since there is no direct feedback from the game to the plant it can’t ‘learn’ as such) I was able to get it to play Soul Caliber VI, a player vs player fighting game, albeit really badly. It is akin to someone pressing buttons on the controller really rapidly and seemingly at random (a tactic I have too often been accused of abusing…).

Here is a short clip of it fighting in the arcade game mode.

I found some really bizarre outcomes through doing this and because I recorded vast amounts of midi data (on Ableton live) doing these tests I was able to view clear patterns after multiple days. For instance, the plant preferred vertical attacks earlier on in the day and horizontal attacks in the afternoon & evening. An increase in movement/strafing depending on the amount of light in the space. As well as this the frequency of signals being output from the plant would fluctuate based on a variety of elements such as time of day, if I was in the room or not, whether there were other plants that were nearby, and even if those plants were of a similar genus (it seemed to enjoy having ‘friends’ so to speak).

I took the plant online to fight other online players (without telling them they were against a plant) and it lost nearly every single fight ( I pity the people who lost against it). Initially, I thought it (the plant) was ‘bad’ at playing this particular game and tried to find others for it to enjoy, then I realised there should be no ‘bad’ way to play a game, only bad games. Which is exactly when I decided to make a game specifically for this house plant.

It was very difficult for me to imagine what kind of game a plant would want to play as we have no concept of plants playing, furthermore, anything I imagine to be fun or enjoyable is from the perspective of a bipedal human and not that of a house plant so would be inherently wrong.

I posited that if I were motionless and rooted to the same spot for my entire existence that I would dream of being able to move around, just as we, confined by gravity, dream of flying. So that was a start…
Then I began thinking about how I would represent such an entity, how a plant would visualise itself and also perceive the world around it. What would it see? How would it relate to its environment?

Cephlalgia – Gig excerpt (2)

Cephalalgia – Ice cream headache

This is the Second portion of a 3-part performance of visuals & sound, combined generative & composed. Performed at the Sheffield Theatre Delicatessen in support of ‘Jefferson Slim Sheik Experience’ for their ‘Cittasamskara’ show.

One of the few pieces containing voice, albeit barely recognisable. Regarding the cognitive dissonance contained within ice cream headaches.

Relax – Gig Excerpt (1)

This is the first of a 3-part performance of visuals & sound, combined generative & composed. Performed at the Sheffield Theatre Delicatessen in support of ‘Jefferson Slim Sheik Experience’ for their ‘Cittasamskara’ show.

I wanted each section within the performance to have a defined style. Thus this part refers to calming notions of water, nature, birds, forests, etc. I conceived this piece after discovering a ‘relaxation cassette’ made by my father for one of his clients roughly 20 years ago. I listened to the entire recording & constructed complimentary pieces of sound design to go alongside the narration, as well as utilising existing footage of water used in previous projects. Subverting both to form a new, altogether connected experience.

Ideasthesia – Art Installation – May 2015

What do we hear when we listen to sound? What could we see? What would it mean to us? How do we construct this meaning? Through experience? What could we experience? Do we experience the event itself? Or the experience alone?

The installation sought to create a synaesthetic environment through the juxtaposition of colour & sound. Through assigning groups of colour with specific sonic events, a combined aural/visual experience occurs within the audience.

Here’s an interview where I explain it a bit better.

Here’s the tester video I made prior to the installation in order to convey the idea of what it might be like. I was pretty close.