Final project

Project Overview:

Our final project came together as a combination of the ideas we had for our individual projects we did for the SISTA art show earlier this semester. Andrew’s began as a sort of game and experiment in 2D physics using the TUIO protocol to interact with the shapes. Mine was an adaptation of a simple drawing application where the lines would back connect and form a kind of web. It was also designed using the TUIO protocol to allow the users to perform multi-touch drawing. The initial issue trying to bring these two programs together was the difference in how they were being drawn. Since Andrew’s involved moving objects it required the background to be redrawn every frame so that previous frame is erased and the shape would only exist at one point. Mine was the complete opposite since I wanted the lines to persist every frame the background was only redrawn when it hit the reset point. To fix the issue the line drawing code was changed so that it produced a line object that was stored in an array list. Then this list of lines was iterated through every frame and redrawn, maintaining the web. In addition to this the drawing was modified so that instead of being based on TUIOcursor locations, the lines would be drawn by the shapes. Andrew improved upon his initial 2D physics design quite significantly and added a number of features to the shapes with both keyboard and TUIO based UI to interact with the features.

After doing the necessary modifications to my code to work I spent most of my time trying out different computer interaction ideas based around the TUIO protocol. I settled on using the Wiimote since there appeared to be a number of tracker applications and I owned one. In doing my initial research I found that most used the Wiimote’s IR camera to detect an IR light source. This light source would serve as the cursor. Once I got to the actual testing phase this became much harder. The first application I tried was Wiimote Whiteboard due to it being the only one with a Mac build. I was able to get it up and running with little difficulty, the issue arose though that this application did not send TUIO but rather controlled the mouse cursor. Instead of rewriting the code to run off mouse inputs I went with looking for other options. The first I tried was the WiiRemoteJ and Wrj4P5 library. It looked promising however I was never able to get it to run any demo code. I assume it is due to it last being updated in 2009 and did not appear compatible with Processing 1.5. The final application I tried was WiimoteTUIO which unfortunately only has a Windows build. This application did however work the way I wanted. It broadcast the TUIO signal, which was received by our sketch and responded to the IR pen. The only issue that currently exists is what appears to be a rather limited field of view of the Wiimote’s IR camera. This leads to rather poor calibration which makes pressing the interface buttons a bit of a challenge. 

Wiimote Whiteboard 
WiimoteTUIO 
Project code 

Multi-touch with TUIO in Processing

TUIO is an open framework that defines a protocol and API for multi-touch surfaces. The protocol encodes control data from a tracker application and sends it to any client application capable of decoding the protocol. With this there have been many implementations of both tracker and client applications for numerous platforms. Currently the tracker applications that I have worked with are Tonseng, a wrapper for the MacBook's touch pad, and Community Core Visions an OpenFrameworks based multi-touch tracker. I have also used the two Processing implementations of the TUIO client.

The first is TUIOProcessing, the simpler of the two TUIO libraries for Processing. This client generates either a TuioObject or TuioCursor whenever there is a new contact single sent from the tracker application. Objects and Cursor function much in the same way, each has its own unique ID. Additionally, they store current location, direction/magnitude, and path. This provides a lot of freedom to work with touch point information to create interesting application.

The second library is tuioZones, which provides the ability to make predefined zones in an application that respond in different novel ways to TUIO signals. The other thing this library has already implemented is gesture methods. Meaning that is library would be the better choice to create applications with swipe or scaling gestures events. The library also has a method for retrieving the cursor points although it does not look as user friendly as the TUIO point methods or quite as well documented.

Demo of tuioZones


For more information on TUIO
For more on tuioZones

Random Project Ideas

1.  I would like to do something with the Super Formula to create 3D shapes that would transition from one state to another either by user input of some kind or through random number generation.

2. I would like to make some kind of fractal art work either in 2D or 3D.

3. Bleeding the Stone, would be a piece in which a 3D stone block is intractable by a user, through clicks. And after being clicked the stone would begin to drip blood from that point.

4. I would like to revisit a previous work I did in which user text input was converted to sound. In this iteration the audio being used would be based of much simpler such as the keys from a piano. With this it may be more apt to create "music" from texts.

5. Generative paintings: These would be images created through the random movement of curves or lines that leave behind after images of the feature, thus as it continued to run the higher traversed areas would be darker.

6. I would like to expand on the generative poetry assignment. In this version the words contained within the dictionary would all be on a tree, representing leaves. When reading through the poem structure, random leaves matching the part of speech would fall off the tree and into its space as the poem is created.

7. The most ambitious idea I could come up with is to create a digital puppet show. The puppets would be figures generated in processing and their movements would be mapped to the "actors" movements through some form of motion capture such as the Kinect.

For additional ideas view my previous post on showroom ideas.

Artist Statement

Nathan Fitzgibbon is an artist who comes from the cynical internet clad gamer generation. His art however focuses on more his interests in socioeconomic and political injustices within the United States. This became a major influence on him as he grew older and began to pay more attention to what was actually happening beyond his computer screen. This is evident in two of his major recent pieces, Political Capital and TARP. These pieces are both intended to provoke the audience to think about the current course of both the political system as a whole and its interplay with the American economy. He also holds on to his past as a major science nerd with his piece The Real Game of Life. In the end he is trying to teach the viewer something, whether it be political awareness or a little something about evolution.  

Showroom Projects

Political Capital:

This project would explore the way the American political system has been corrupted by money. I would take images of major elected officials and and systematically overlay the image of a hundred dollar bill over the individual. The way I envision this working is the viewer will start off seeing the image of some significant political figure and as time progresses the pixels will being to change, slowly at first, but as the number of changes increases the speed at which they occur also increases. I am still undecided as to the best way to have the changes occur, whether it be very ordered, completely random, or done in a style similar to the computer virus image manipulation.


TARP:

This project would explore the social injustice associated with the 2008 Trouble Asset Relief Program (TARP). I would like to project the corporate offices of the major banks associated with this program in interactive stereoscopic 3D. The buildings would have the scrolling down the sides the CEO's names and their pay each year since 2008, additionally on another side of the building would be the names and mortgage amount of families who have had their homes foreclosed on by said bank. The user would be able to either move around the buildings or move the buildings and be able to scroll up and down in the lists. 

The Real Game of Life:

This project would take John Conway's Game of Life and attempt to move beyond the simple  0 1 notion of life it presents. Instead I would incorporate many of the accepted theories in Evolutionary Biology. Examples of which would be both drift and natural selection. This system would hopefully be able to model population based changes such as fixation. In the end the system should be seeded with a couple of "species" and the end result could see the rise of entirely new "species" that were not initially seeded as well as the absorption and extinction of progenitor "species".

Generative Potery

For this project I chose Edger Allan Poe's "Eleonora" as the source for the words I used in my generative poetry. I chose the grammatical structure of Dylan Thomas' poem "Do not go gentle into that good night." The poems are generated at random with each part of speech presented as a different color and size. The different parts of speech also fall at different velocities.

Git location for program file
View sketch in Open Processing

Abstraction Sketch

When looking for a piece to try to recreate in processing I was not really sure what I was looking for as I cycled through one artist after another on the MoMA website. Then I stumbled across Vaslav Nijinsky whose untitled pieces immediately caught my attention. One of the initial things I found interesting about the artist is that he was much better known as a dancer and choreographer and that his pieces were in a way a graphic representation of the art of dancing.

Vaslav Nijinsky Untitled 2 of 3

The reason his untitled piece 2 of 3 (1918-19) drew me in was the nostalgic feeling that they evoked. Looking at the pieces reminded me of doodling in High School Geometry class. That made these pieces even more impressive since Nijinsky drew them all freehand where as the doodles I would make were with a compass and protractor. And after having attempted to duplicate this piece using the Processing framework it really does make me miss my protractor and compass.

To view full sketch and code