My two week stay at STEIM was to enable me to work on mapping strategies for the Novint Falcon – a haptic USB device – that would help me consider how it might be used in a musical context.
The Falcon allows you to move it’s grip around in three dimensions, but what makes it particularly interesting is that it can also exert fairly powerful forces back on your hand. This makes the mapping layer in the software rather more involved than it is in non-haptic devices as the data generated by the controller can influence both sound and physical forces.
I have chosen to deal with this problem by creating phyisical metaphors for the control – where the controller affects a simulated physical system. I have therefore been primarily concerned with two mapping layers: from the controller to the physical model and from the model to the forces on the controller. The ones I have developed so far are:
– A wheel that can be turned. The momentum of the wheel can be felt as resistance if you push in the opposite direction or as an assisting force if you push in the same direction. With certain settings, once some momentum has been generated you can leave the controller to turn itself due to the forces on it. See the video examples below.
– A 3D grid that locks your hand into discreet cells. The force with which this happens can be varied to make it easier or harder to move between cells. The 3 dimensions of the grid can be also be individually altered (n x m x p). See the video examples below.
– A string model. 4 virtual strings are set up and positioned along the z axis. When they are struck, the falcon exerts a force away from the string bouncing your hand back.
I now hope to develop some instruments based on these models, which will mean grappling with a third mapping layer: from the model to the sound engine. It would also be interesting to try alternative mapping setups, such as analysing the audio produced and using that to determine the forces on the controller.
– 3D grid demo:
– 3D grid demo 2 (with simple audio mapping):
– Winding wheel example:
On another note, I have also begun developing a patch that allows you to ‘feel’ a waveform by running your hand along it, and having your hand displaced vertically by the amplitude of the audio file at that point (this idea was suggested by Michael Young). This can also be expanded by allowing you to scan for different frequencies. For example, if set to 100 Hz it displaces your hand according to the amplitude of the sound file within a certain bandwidth around that frequency. As there are two dimensions to play with – y and z if the waveform is spread over the x-axis – two different frequencies can be scanned at once.
I was also happy to present some of my past work on the Feedback Joypad instrument and give a short performance with it. More info at tommudd.co.uk/projects/joypad.htm
or for more info on me generally: