hello and greetings from lisbon. things have been busy since we left amesterdam, a couple of weeks ago. i’d like to post the wonderful time at steim. and also some followup on how the workshops relate to my work, and where to now. this post will wander a bit around the wirelessness of it all, about physical wireless controllers and camera invisible interfaces.

steim is the analog resource haven, where a wonderful crew engages all aspects surrounding creation of custom controllers/synths. it has cables, boxes, tables filled with everything useful to electronics all around. and some of the best people handling all of that.

what most caught my attention were wireless controllers being develloped. Michel Waisvisz’s hands are a super example of that. whatt power! in the diy, joysticks, wireless joysticks, and how they can be hacked to almost any kind of analog 5v sensor, how you can boost the signal to provide bigger range using a bit more voltage on the sender, like cell-phone batteries, which is great! joysticks are great ways to pass on data around, and can be very useful for theater/dance, for instance, with camera tracking going on too. they can be invisible triggers in a scene where the camera is not precise enough, or take advantage of a physical situation, like in Joergens’ example, a football wired with accelerometers hacked into a wireless joystick sender.

this gave me a lot of ideas to build wireless controls as instruments that musicians can integrate very well. last week i was programming and performing with parque in festival escrita na paisagem, where i devised the sound->spatializing->lights program on two of three different devices that were played. they were ‘os’ and ‘peça de embalar’ (lullaby piece). the third one is a pendular suspended speaker called ‘pêndulo’.

‘os’ is a piece where 4 musicians play the mirrors suspended, and has microphones attached to each man. these 4(5 with the saxophone) sounds are spatialized in the quad audio setup we had at that piece’s space. there is a graphical video-projection that pinpoints where the sound sources are, and how loud they are. the motion of the spating goes from circular runs to smooth splines traversing space at sound speed, or rapid chance operations.

as interfaces, the macro-parameters were controlled by me in a behringer midi controller and also the computer, but the musicians controlled how fast the sound travelled in the path on some programs, or controlled the light programs and the lights according to each of their sound, on each side of ‘peça de embalar’ through midi pedals, with very very long chords. a wireless joystick can make the same task and even better, you can perform your instrument, and at the same time, control algorithms that use the sound as input and also some accelerometers attached to your hand, for instance. these kinds of ideas spurred at steim. and very likely to develop more as interfaces.

processing live sound is something i have been doing in my installation series ’struct’, where microphones gather site-specific ambiances of galleries or other spaces, and the sounds are continuously recorded into audio buffers that are manipulated by the computer in the first 4 versions, by the sound presence of the user in the 5th version, and by the video presence of people in space in the 6th version ’struct_5′

my interest with controllers now is invisible controllers, mostly camera based tracking, and how one or more people can interact with a digital work. i have also built a different camera approach in 747.3, where a user’s position of arms, and the distance from hand to hand control the heading and the speed of traveling across a digital abstract terrain. here is an image of the object i coded in jitter for this work. one of the main problems/peculiarity of this type of interface is the lack of physical feedback, where the actions that you develop as a user have just a virtual counterpart, and no touch envolved at all. it has its upsides too. quite portable and lightweight. a lot of people enjoyed very much the immersion provided by my piece, and where physically exhausted mostly in the arms. but they relate they forget the real body and immerse themselves in the picture, having really fun runs of 15 to 30 minutes. sometimes it’s also a bit shaky if the conditions are not good. there’s a lot to take in as variable to be controlled in this. the lights have to be very stable, the camera must be blind to visible light, it interferes too much. and finally, the ir lightining must be well made to get good difference images and no shadows interfering. it can also be quite interesting to get different patterns of motion sculpting algorithms of image and sound. like mobs of people or single users specific motion.

future posts will wonder a bit in detail about the topics i mentionned in the works above: creating immersive audio-visual environments, intermedia instruments and camera tracking interaction, all coming around in a new simple work i am developing to be shown as an installation in the lisbon nip workshops.

Sunday morning, final day of the first workshop and we all make it! We all came with our cameras - digital video cameras, webcams, computer cams and used them to begin playing with image data.

Today’s session focused more on computer vision techniques. Zach lead on how Openframeworks can be used to grab information from the camera. In grabbing video data you can then start to compare each frame and its pixel properties and use the data to create a series of real-time effects and interactions.

In grabbing video data you can then start to compare each frame and its pixel properties.

For example:
Set up your video camera to capture in real time an image
Compare incoming images, with previously captured images
Using methods such as foreground and/or motion detection
You compare the images, pixel by pixel
For foreground detection, you write your code, so that it checks the absolute values of the each pixels, checking if they are present or not in each image (e.g, present = true or not present = false)
For motion detection, you write your code, so that it looks at what pixels have changed per frame (e.g, position 1, position 2)
This give us a new binary image, which you can filter to get more useful information - e.g, you reduce the noise and clean up the image by going through the array of pixels and collating the information that you grab
Pixel information that you can work with includes - how big is the pixel, its position, amount of RGB per pixel, gradation, changes over time, shape matching and much, much more - these pixel parameters, allow you the play with it and create new effects