Time for another update

Finally a bit of time to breathe. A lot happening lately, again. Just finished 2 assignments, 1 more to go for now.
This time I'd like to share some of the things I study in Creative Computing at Goldsmiths, University of London.

I am handing in the project for Audio Visual Processing. The task given was to "implement interactive real-time audio
and video systems using the programming tool of your choice (either maxMSPJitter, C++ using provided libraries, or Java/Processing)."

There was plenty of information to sponge in a short amount of time, but very fun. What I've noticed so far is: the simpler, the better. What I mean is, if I use the right numbers by processing video or audio, the results might be a bit 'jittery' and might not convey a theme, that could be easier conveyed with a bit of cheating. The brain does a lot of things for us, and when it comes to video we only need 24 frames per second be fooled into seeing continuos animation, when linked to sound, the images gain value, and feed information (confirmation) back to the sound (source) in a strange but effective loop. I haven't managed to get the right balance between the data processed and the way to map that wisely so it's instantly and easily perceivable.

Well, enough talk for now. Here is a an attempt to modify 3d geometry (a plane) using video(color) and audio(peak amplitudes sampled every 40 milliseconds) as source. For some reason it makes me think it could be a 'quick'n'dirty' technique to achieve an effect similar to Michel Gondry's "Fell in Fell In Love With A Girl Lyrics" video for The White Stripes. The source video is a performance by Les Elephants Bizarres, talented friends from back home.

GridTest - Les Elephants Bizarres - Hello Says The Devil from George Profenza on Vimeo.

The colour channels from the video control the height of each grid square, coresponding to a group of polygons in the 3d plane. The resolution of the 3d plane and source video can be altered in realtime. The inputs can trigger a change all the time, or only when a change in peaks occurs, as with this video. Also the height can be controlled by the audio, not fully demonstrated in the video.
Click here for the MaxMSP/Jitter patches used to generate the videos(really compressed videos).

Here is a simpler attempt to modify a procedural 2d texture that in turn modifies the geometry of a 3d plane using sound as input. This time I used the pfft object narrow 3 ranges of frequencies (low, medium, high), but they're not very cleverly mapped to the texture inputs. The low frequencies control the scale of the procedural map, the peaks in medium frequencies change the procedural map, while the high frequencies alter the weight applied to the maps.

AudioVisual test - Valentin Leonat - Esti Departe from George Profenza on Vimeo.

Music by Valentin Leonat. Be sure to check out more tunes on Valentin's MySpace page.

The method to analyze audio is good enough, but not the method to control the visual. The inputs might work better with a glitchy tune. Here is the Ghost of 3.13 with Orchids and Lilacs

AudioVisual test - The Ghost Of 3.13 - Orchids And Lilacs from George Profenza on Vimeo.

Missing myself

Detail from Nature by John Maeda


Sounds egoistic, and maybe it is. I didn't bother me much, until I've looked back and realized how I've changed. Well, we all change, we interact, we change others in return and so on. I used to draw more, create more, not better, just more. I was annoyed I didn't know enough code to make some of my ideas work. A few random events, like Jared Tarbell throwing one of his laser cuts wood models at me at FOTB'07...



... winning an Arduino at LFPUG, a few dozen coin flips always ending on the same side and I ended up studying Creative Computing at Goldsmiths, University of London. There was a lot of new material to cover (like Java, Processing, OpenGL, MaxMSPJitter, openframeworks, etc), but I've managed to do a decent job and I don't think I did it by my own.

Back in 2007 I just started to tinker with code(as2.0) and I was pretty bad at it, plus the amount of knowledge I had to cover scared me. AS3.0 was out and that was even scarier. When I was at FOTB I had a copy of Essential AS 3.0 which I used to get autographs from most of the great developers there like Keith Peters, Andre Michelle, Brendan Dawes, Erik Natze, Carlos Ulloa, etc.



In my head it was like I had their 'go-ahead'/approval and I'd owe it to them to learn all that. I managed to get a job in London in late April, early May 2008 which meant for a while I commuted from Canterbury, each morning waking at 5, sleeping a few hours a night and had to hand in my projects for Fine Arts and Digital Media at that time. Daily bus journeys meant 4 to 5 hours a day which meant when I wasn't instantly falling asleep after running to catch the bus I would read. The 'good luck charm' worked.

I got scared again when I had to start Creative Computing. Not a problem, I got Robert Hodgin's signature on my copy of Processing: A Programming Handbook
for Visual Designers and Artists. I started to learn Processing and started to play with OpenGL a bit. I still regret I haven't caught James Patterson, although I did put about 10 or 11 forms in 2007 and in 2008 John Davey managed to get him on stage.

This year I am finishing my undergraduate studies. This is a bit unbelievable as this would be the 5th University I'm studying in, but the 1st degree I seem to finish. Now I don't know for sure where I will head, but as usual I 'smell' a direction and keep hitting walls from side to side in order to move forward. Sometimes I find shortcuts, sometimes I get lost for a while, it's all part of learning.

Now fear was creeping in regarding my direction, but I found a bit of light tonight.
I just got back from an amazing talk at the V&A given by John Maeda. I didn't realize how much I went into this whole computing thing and technology, I was thinking more about the geeky details than the ideas. Getting lost into creating tools and the most efficient way to build tools that I ran out of time to actually use them. As Jonathan Harris mentioned, tools are using me, not the other way around.

Thank you John Maeda for reminding me to finish what I've started. Oh, and thank you for the 'go-ahead' as well.



It is always great to meet your heroes in person, as a great man once said.

quick update

I'm trying to write at least a post per month, so here's October, roughly.

First off...I'm year 3 now, the final year in my degree and a lot of work will go into that. I had a few good ideas for my projects, but it so happened that this year it won't be a solo project. I will work with a friend from Canterbury who studies FRTV (Film-Radio-Television) on a movie project. She will take have to deal with the script, props, actors, and what not, I will be doing the soundtrack. Soundtrack ? Yes!

Sound if one the those things that scared me. I work with visual stuff most of the time and its my comfort zone. I've always been scared of playing with sound/music, I never played any instrument, never was good at anything related to it. Now that I understand roughly how sound works in geeky terms (longitudinal pressure waves), all this is like a black hole...a whole new world awaiting for me to get sucked into. That is good, but that goes to prove that there is no support for visual things ( drawing, animation ) in our Creative Computing course, but there is plenty for sound though.

Ever since this project started, things started to fall into place like jigsaw pieces, one at a time, nice and steady. The more I researched sound the more I found confirmations to get rid of my fears. The first books on sound I got my hands/eyes/head on are: "This is your brain on music" and "Musical beginnings".
Aside a lot of interesting research on sound and how it works on your body( It seems the fetus starts to develop his/her auditory system between weeks 8 and 11, later it distinguishes voices, it recognizes the mother voice, it gets accustomed and calmed by rhythms (hear beats), etc. ), things you can read more by yourselves, and hopefully I can blog about them more in time. There was a part that gave me courage:
"There are two widespread myths about musical excellence. The first of these is that high levels of musical accomplishment are necessarily rare. The second is that these high levels of skills are predicated on particular unusual early musical attributes or capacities." - John Sloboda in Musical Beginnings.

I will test this things as I go along, but so far it's been nothing but confidence.
If I think of cymatics and the Jared Ficklin and Craig Swan's sessions at Flash On The Beach...since were all just grouped particles, suddenly stories about magical spells (words that resonate to the right frequencies to change matter) don't seem that far from the truth.

Weird ... :)

Anyway...back to this 'real' world

What I will do is generate a soundtrack on the fly as the movie is being shot. Using computer vision and a few fancy microphone setups as inputs I will generate music that will be in tune with what happens visually on the screen. Since this will be a translation of movement into sound partially...there is a lot of history on that side in Digital Performance. On the other side, post production, there are people developing clever systems to analyze video and annotations to produce soundtracks. We are bit stuck in the middle, and I am not sure where we might go with this. It seems like a fun and interesting thing to explore and we'll see what will happen.

As i said, things fall into place with this project. I need to get the grips with computer vision. I've played with actionscript and webcams a bit, but nothing fancy. openCV has tons of cool things already implemented so I'm pretty sure we'll use that. It so happens that Arturo Castro and Joel Gethin Lewis ( who blew people away with his projects at Flash On The Beach this year ) will teach Computer Vision at Goldsmiths for 4 days.



"Goldsmiths Digital Studios is pleased to announce a Computer
Visionworkshop using
openFrameworks on 2nd, 3rd, 4th, 5th of November 2009 lead by Arturo
Castro(co-founder of
openFrameworks) and Joel Gethin Lewis and
hosted by Goldsmiths College’s MFA Computational Studio
Arts programme.
You will gain a very good insight into what you can control
through video and be able to work on interactive projects involving
motion-tracking, face and body recognition and identification and more.
There will a group for beginners, starting off with some basics in
object-oriented coding and an advanced group who will dive straight into
computer vision. If you’d like to get a sample of openFrameworks in action,
check out Made With openFrameworks"

I've never programmed in C++ before and I haven't used openFrameworks. I've setup OF using the easy to follow instructions on their website. Browsed through a few samples and so far it doesn't look that hard. I'll get used to namespaces probably. I've spotted methods familiar to Processing ( setup, update, draw, etc. ) and it feels confy :)

Hopefully I will document my project using this blog so you will hear more about this.

I haven't ditched Flash, I don't think I will. Lately I've met some brilliant people that use a lot of cool stuff, mostly more powerful than Flash, but they still use Flash and enjoy it. I found that reassuring.


Speaking of people I've met, I will try to tell you a little bit about them.
One of them is Michael Jewell who is a researcher at Goldsmiths. Remember I've mentioned people that develop clever systems that make soundtracks in postproduction ? You can read more about that in his thesis. Aside that he made a few good games for State Of Play. One of them is Moshi Monsters and I couldn't believe the technical details of the project. Some mad actionscript bytecode action going behind the scenes. Michael hasn't updated his blog in a while, but he's got some cool stuff there.

I went to a few geeky workshops lately at Space. There is a guy that I will not mention his name yet. We meet in situations like this (Arduino workshop in this case, Away3D workshop last year, etc. ) and its funny how these things just happen.


Aside OpenProcessing and HasCanvas, there is an actionscript version of a Processing online prototyping tool called SketchPatch. I've met Davide, one of the co-founders of the project at a Chip Tune workshop, nerding away on 8bit noise. The project looks pretty good but there are always things to be added, and I hope I can lend a hand.




Still on the Flash side of things, LazyBoy, a handy extension I wrote for Flash, that got featured in WebDesigner Magazine, is now also on the Adobe Exchange website. If you use the Flash IDE and work with classes and custom layout designs, this might save time setting up.



Since Uni started, there is plenty to do there ( year 3 project, openGL course, Physical Computing course ) so I'm working part time, which means I don't have the time to get involved in the big projects at disturb. I'm missing out on fun, and getting more things that can be handled in the 2.5 days per week I'm there. Somehow lately I've managed to get some fun out of it. I've built a few tools that would do the work for me. Even things that were supposed to be dull ended up being fun which is great. I hope I can get that happening for a long time now. The latest project was a quick karaoke thing, I might not be able to talk more about it as most of the things we do at disturb are secret. I wish I can post some of the tools. I worked with our new designer, Jason Turner. I thought I recognized him from somewhere, he is one of the finalists in the Cut&Paste Digital Design Tournament...that is like Live Coding for Designers :)
New designer, as in extra one, yup we're 8 of us now at disturb and Chris is coming back from Australia...woo hoo! ASS! ASS! I bet he's all saturated with Mumatron love by now...WHUT ?

I keep meaning to post some tutorials never have the time. There's plenty of things in the bag, let's just hope we won't have to wait until Christmas.

Yup, things have gone mad busy again, but it's all good so far.

Until next time,
SMILE! ^_^