Live-blogging nime: mobilemuse: integral music control goes mobile

music sensors and emotion
integral musical control involves state and physicl interaction. Performer interacts with other performers, with audience and with the instrument. Sounds from emotional states. Performers normally ingeract by looking and hearing, but these guys have added emotional state.
audiences also communicate in the same way. This guy wants to measure the audience.
temperature, heart rate, respiration eeg and other thing you can’t really attach to an audience.
send measurements to a pattern recognition system. The performer wears sensors.
he’s showing a graph of emotional states where a performer’s state and an audience memberms state track almost exactly.
they actually do attach things to the audience. This turns out to be a pain in the arse. They have now a small sensor thing called a “fuzzball” whnich attaches to a mobile phone.
despite me blogging this from my phone, i find it hugely problematic that this level of technology and economic privilege would be required to even go to a concert….
they monitor lie detector sort of things. The mobile phone demodulates the signals. The phone can plot them. There is a huge mess of liscence issues to connect hardware to the phone, so they encode it to the audio in.
they did a project where a movie’s scenes and order was set by the audience’s state.
the application is open source.

How musicians create augmented musical instruments

augmented instruments are easy for performers of the pre-existing instruments. Musicians themselves have expertise, so let them do design. Come up w a system to let them easily do augmentations.
thr augmentalist was designed collaboratively.
gestures go to instrument, sesnors or both. Sound goes into daw. Processing happens
photo of a slider bar taped to a guitar: quick and easy!
instrument design sessions w 10 pop musicians. Experimenters presented system and updates then a testing session, then instrumentalists played arouns, then instrumentalists made suugestions for changes.
guitarists put tilt measurements on the head. Slider on guitar body. Sensors mapped to typical pedal fx.
an mc stuck thing to a mic and himself. Slider on the mic. Fsr on mic body. Accelerometer on his hand. And movements went to pich shifter.
interesting results: most performers tried to use similar movement, like moving head and body. Only one person kept this. All instruments used tilt.
hundreds of gesture/sound mappings were tried. Most considered successful, but not all were kept. Guitarists tend to develop the same augmentations as each other. But some unusual things were developed also.
musicians start with technology rather than the gesture. Technology is seen as the limitation, so start w it’s limitations.
all musicians believed they could come to master systems.
over time, the musicans make the fx more subtle and musical
can people swap instruments? Yes, they felt each other’s instruments were easy to use.
one guitarist uses the system with his band and they’re gigging w it.
takes up a lot of brain cycles to use extensions. It takes a lot of practice.
every musician had maximum enjoyment at every session.
this kind of user-lead thing can create new avenues for research.

Live blogging nime – ircam assigning gesture to sound

they play a sound and then ask people to represent the sound as gesture and then use that gesture to control a new soud. The sound to gesture is an experimental study, which was a very good idea!
in the existing literature: tapping a beat is a well – known gesture. Body motion to music and more new: mimic instrumental performances (ex air guitar). Sound tracting is sketching a sound?
Gaver says musicl listing has a focus on acoustic properties and everyday listening focuses on cause.
categorisation of sounds involves the sound sources. People would categorise door sounds togther, even if they are very different sonically.
will subjects try to mimic the origins of causal sounds, eg mime slamming a door?
will they trace non causal sounds?
they played kitchen sounds and kitchen sounds convoluted w white noise
track subject’s hand position. Each subject gets some time to work out and practice her gesture, then record it three times. Ask the subject to watch the viseo and narrate it.
the transformed sounds are described more metaphorically. Non transformed sounds describe the object and the action is described, rather than the sound.

Nime blogging- stereotypical transducers

earphones are the most common transducer.
can use doppler frequencies from headphone w a seeperate mic to detect motion. This may be ultrasonic? Need to use bandpass filter to just get relevant frequencies. Also, there needs to stuff in the audio range, so filter that w low pass to make sure it stays where it should.
demo video shows this works when a guy waves his hands around.
now we see a chart of error between movements and detections. Moving away has more error.
the earbuds are meant to be held and not worn, so it actually doesn’t need to handle normal range of sounds too.
to make system wireless, use a portable music player.
people over 18 cannot hear the ultrasonic frequencies. There are hardware limitations with nyquist. Or ability to play 18kHz

Live blogging nime papers – gamelan elektrika

a midi gamelan at mit
the slide shows the url Supercollider.ch
they wanted flixible tuning for the gamelan. Evan ziporyn worked on this project. They got alex rigopolus something about touring with kronos. This is going on too long about name dropping and not enough about how it works. I’m not even yet sure what the heck this is, but media lab sure is cool.
this must be a really long time slot becuase she has not yet talked about a technical issue yet.
low latency is important. I think she accidentally let slip that this uses ableton, thus revealing a technical issue.
mit people are often very pleased with themselves.
urathane bars with piezo sensors for version 1, so they switched to FSR and a different material. Didn’t catch how it senses damping.
reong has 5 sensors per pot anf FSRs for damping. Hit the middle, touch something to damp.
gongs have capacitive disks on side, piezo on the other, to sence strikes and damping
supercollider and ableton on the backend to handle tuning and sample pplaying
samsung laughes at them when approached for sponsorship but they sure showed them! Nime is thus invited to share in gloating of the amazing skillz of mit.
instrument demo fail.
question about why not use normal percussion controller? answer: to play like a regular gamelan.
the monitoring situation is a problem for them. They use 4 speakers to monitor. House speaker to play

Live bloggig NIME papers – electromagnetically sustained rhodes piano

Make a rhodes piano that doesn’t decay. Can start from a strike or from the excitation. The examples sound like an ebow made for the rhodes.
Rhodes has enharmonic overtones, especially on the strike. The pickup gets mostly integer multiples of the fundamental, especially even partials.
the actuator is an electromechanical coil driven by a sinetone generator of the fundamental. The pickup also grabs the actuator, so they remove the phase inverse of it past the pickup. They can also use feedback to drive the tine. This causes out of control feedback, so they sense the output just of the actuator and subtract that out, leaving just the tine, thus getting increasingly rube goldberg.
there are pros and cons of each approach. They measure after touch for control using a pressure sensor below each key. Appropriately, all the dignal processing is analog

Kinect and OSC Human Interface Devices

To make up for the boring title of this post, lets’s start off with a video:

XYZ with Kinect a video by celesteh on Flickr.

This is a sneak preview of the system I wrote to play XYZ by Shelly Knotts. Her score calls for every player to make a drone that’s controllable by x, y, and z parameters of a gestural controller. For my controller, I’m using a kinect.

I’m using a little c++ program based on OpenNi and NITE to find my hand position and then sending out OSC messages with those coordinates. I’ve written a class for OSCHIDs in SuperCollider, which will automatically scale the values for me, based on the largest and smallest inputs it’s seen so far. In an actual performance, I would need to calibrate it by waving my arms around a bit before starting to play.

You can see that I’m selecting myself in a drop down menu as I start using those x, y and z values. If this had been a real performance, other players names would have been there also and there is a mechanism wherein we duel for controls of each other’s sounds!

We’re doing a sneak preview of this piece on campus on wednesday, which I’m not allowed to invite the public to (something about file regulations) but the proper premiere will be at NIME in Oslo, on Tuesday 31st May @ 9.00pm atChateau Neuf (Street address: Slemdalsveien 15). More information about the performance is available via BiLE’s blog.

The SuperCollider Code

I’ve blogged about this earlier, but have since updated WiiOSCClient.sc to be more immediately useful to people working with TouchOSC or OSCeleton or other weird OSC devices. I’ve also generated several helpfiles!
OSCHID allows one to describe single OSC devices and define “slots” for them.
Those are called OscSlots and are meant to be quite a lot like GeneralHIDSlots, except that OSCHIDs and their slots do not call actions while they are calibrating.
The OSC WiiMote class that uses DarWiinRemote OSC is still called WiiOSCClient and, as far as I recall, has not changed its API since I last posted.
Note that except for people using smart devices like iPhones or whatever, OSC HIDs require helper apps to actually talk to the WiiMote or the kinect. Speaking of which…

The Kinect Code

Compiling / Installing

This code is, frankly, a complete mess and this should be considered pre-alpha. I’m only sharing it because I’m hoping somebody knows how to add support to change the tilt or how to package this as a proper Mac Application. And because I like to share. As far as I know, this code should be cross-platform, but I make no promises at all.
First, there are dependencies. You have to install a lot of crap: SensorKinect, OpenNi and NITE. Find instructions here or here.
Then you need to install the OSC library. Everybody normally uses packosc because it’s easy and stuff…. except it was segfaulting for me, so bugger that. Go install libOSC++.
Ok, now you can download my source code: OscHand.zip. (Isn’t that a clever name? Anyway…) Go to your NITE folder and look for a subfolder called Samples. You need to put this into that folder. Then, go to the terminal and get into the directory and type: make. God willing and the floodwaters don’t rise, it should compile and put an executable file into the ../Bin directory.
You need to invoke the program from the terminal, so cd over to Bin and type ./OscHand and it should work.

Using

This program needs an XML file which is lurking a few directories below in ../../Data/Sample-Tracking.xml. If you leave everything where it is in Bin, you don’t need to specify anything, but if you want to move stuff around, you need to provide the path to this XML file as the first argument on the command line.
The program generates some OSC messages which are /hand/x , /hand/y and /hand/z, all of which are followed by a single floating point number. It does not bundle things together because I couldn’t get oscpack to work, so this is what it is. By default, it sends these to port 57120, because that is the port I most want to use. Theoretically, if you give it a -p followed by a number for the second and third arguments, it will set to the port that you want. Because I have not made this as lovely as possible, you MUST specify the XML file path before you specify the port number. (As this is an easy fix, it’s high on my todo list, but it’s not happening this week.)
There are some keyboard options you can do in the window while the program is running. Typing s turns smoothing on or off. Unless you’re doing very small gestures, you probably want smoothing on.
If you want to adjust the tilt, you’re SOL, as I have been unable to solve this problem. If you also download libfreenect, you can write a little program to aim the thing, which you will then have to quit before you can use this program. Which is just awesome. There are some Processing sketches which can also be used for aiming.
You should be able to figure out how to use this in SuperCollider with the classes above, but here’s a wee bit of example code to get you started:




 k = OSCHID.new.spec_((
  ax: OscSlot(realtive, '/hand/x'),
  ay: OscSlot(realtive, '/hand/y'),
  az: OscSlot(realtive, '/hand/z')
  ));

 // wave your arms a bit to calibrate

 k.calibrate = false;

 k.setAction(ax, { |val|  val.value.postln});

And more teaser

You can see the GUIs of a few other BiLE Tools in the video at the top, including the Chat client and a shared stopwatch. There’s also a network API. I’m going to do a big code release in the fall, so stay tuned.

Strategies for using tuba in live solo computer music

I had the idea of live sampling my tuba for an upcoming gig. I’ve had this idea before but never used due to two major factors. The first is the difficulty of controlling a computer and a tuba at the same time. One obvious solution is foot pedals, which I’ve yet to explore and the other idea is a one-handed, freely moving controller such as the wiimote.
The other major issue with doing tuba live-sampling is sound quality. Most dynamic mics (including the SM57, which is the mic I own) make a tuba sound like either bass kazoo or a disturbingly flatulent sound. I did some tests with the zoom H4 positioned inside the bell and it appeared to sound ok, so I was going to do my gig this way and started working on my chops.
Unfortunately, the sound quality turns out not to be consistent. The mic is prone to distortion even when it seems not to be peaking. Low frequencies are especially like to contain distortion or a rattle which seems to be caused by the mic itself vibrating from the tuba.
There are a few possible work arounds. One is to embrace the distortion as an aesthetic choice and possible emphasise it through the use of further distortion fx such as clipping, dropping the bit rate or ring modulation. I did a trial of ring modulating a recorded buffer with another part of the same buffer. This was not successful as it created a sound lurking around the uncanny valley of bad brass sounds, however a more regular waveform may work better.
At the SuperCollider symposium at Wesleyan, I saw a tubist (I seem to recall it was Sam Pluta, but I could be mistaken) deliberately sampling tuba-based rattle. The performer put a cardboard box over the bell of the tuba. Attached to the box was a piezo buzzer in a plastic encasing. The composer put a ball bearing inside the plastic enclosure and attached it to the cardboard box. The vibration of the tuba shook the box which rattled the bearing. The piezo element recorded the bearing’s rattle, which roughly followed the amplitude of the tuba, along with other factors. I thought this was a very interesting way to record a sound caused by the tuba rather than the tuba itself.
Similarly, one could use the tuba signal for feature extraction, recognising that errors in miccing the tuba will be correlated with errors in the feature extraction. Two obvious thing to attempt to extract are pitch and amplitude, the latter being somewhat more error-resistant. I’ve described before an algorithm for time-domain frequency detection for tuba. As this method relies on RMS, it also calculates amplitude. Other interesting features may be findable via FFT-based analysis such as onset detection or spectral centroid, etc using the MLCD UGens. These features could be used to control the playing of pre-prepared sounds or live software synthesis. I have not yet experimented with this method.
Of course, a very obvious solution is to buy a better microphone. It may also be that the poor sound quality stemmed from my speakers, which are a bit small for low frequencies. The advantage of exploring other approaches include cost (although a tuba is not usually cheap either) and that cheaper solutions are often more durable or at least I’d be more willing to take cheaper gear to bar gigs (see previous note about tuba cost). As I have an interest in playing in bars and making my music accessible through ‘gigability,’ a bar-ready solution is most appealing.
Finally, the last obvious solution is to not interact with the tuba’s sounds at all, thus creating a piece for tuba and tape. This has less that can go wrong, but it looses quit a lot of spontaneity and requires a great deal of advance preparation. A related possibility is that the tubist control real-time processes via the wiimote or other controller. This would also require a great deal of advanced preparation – making the wiimote into it’s own instrument requires the performer to learn to play it and the tuba at the same time, which is rather a lot to ask, especially for an avant guarde tubist who is already dealing with more performance parameters (such as voice, etc) than a typical tubist. This approach also abandons the dream of a computer-extended tuba and loses whatever possibilities for integration exist with more interactive methods. However, a controller that can somehow be integrated into the act of tuba playing may work quite well. This could include sensors mounted directly on the horn such that, for example, squeezing something in a convenient location, extra buttons near valves, etc.
I’m bummed that I won’t be playing tuba on thursday, but I will have something that’s 20 minutes long and involves tuba by September

WiiOSCClient.sc

Because there are problems with the wiimote support in SuperCollider, I wrote a class for talking to Darwiin OSC. This class has the same methods as the official wiimote classes, so, should those ever get fixed, you can just switch to them with minimal impact on your code.
Because this class takes an OSC stream from a controller and treats it like input from a joystick, this code may potentially be useful to people using TouchOSC on their iPhones.
There is no helpfile, but there is some usage information at the bottom of the file:


 // First, you create a new instance of WiiOSCClient, 
 // which starts in calibration mode
 
 
 w = WiiOSCClient.new;

 // If you have not already done so, open up DarwiinRemote OSC and get it talking to your wii.
 // Then go to preferences of that application and set the OSC port to the language port
 // Of SuperCollider.  You will see a message in the post window telling you what port
 // that is .... or you will see a lot of min and max messages, which lets you know it's
 // already callibrating
 
 // move your wiimote about as if you were playing it.  It will scale it's output accordingly
 
 
 // now that you're done callibrating, turn callibration mode off
 
 w.calibrate = false;
 
 // The WiiOSCClient is set up to behave very much like a HID client and is furthermore
 // designed for drop-in-place compatibility if anybody ever sorts out the WiiMote code
 // that SuperCollider pretends to support.
 
 // To get at a particular aspect of the data, you set an action per slot:
 
 w.setAction(ax, {|val|
  
  val.value; // is the scaled data from ax - the X axis of the accelerometre.
  // It should be between 0-1, scaled according to how you waved your arms during
  // the callibration period
 });
 
 
 
 // You can use a WiiRamp to provide some lag
 (
  r = WiiRamp (20, 200, 15);
 
  w.setAction(ax, {|val|
   var scaled, lagged;
  
   scaled = ((val.value * 2) - 1).abs;
   lagged = r.next(scaled);
  
   // now do somehting with lagged
  });
 )

Calibration

this class is self-calibrating. It scales the wiimote input against the largest and smallest numbers that it’s seen thus far. While calibration is set to true, it does not call any of its action methods, as it assumes the calibrated numbers are bogus. After to set calibration to false, it does start calling the actions, but it still changes the scale if it sees a bigger or smaller number than previously.

WiiRamp

The WiiRamp class attempts to deal with the oddness of using accelerometers, but it does not just do a differentiation, as that would be too easy. The accelerometers give you major peaks and valleys, all centred around a middle, so just using the raw values often is a bit boring. In the example, you see that we scale the incoming data first: ((val.value * 2) – 1) changes the data range from 0 to 1 into -1 to 1. The puts the centre on 0. Then, because we care more about the height of peaks and depth of valleys than we care about whether they’re positive or negative, we take the absolute value, moving the scale back to 0 to 1.
When you shake your wiimote, the ramp keeps track of your largest gesture. It takes N steps to reach that max (updating if a larger max is found before it gets there), then holds at the number for M steps and then scoots back down towards the current input level. You can change those rates with upslope, hold and downslope.

OscSlot

This class is the one that might be useful to iPhone users. It creates an OSCResponderNode and then calls an action function when it gets something. It also optionally sends data to a Bus and has JIT support with a .kr method. It is modelled after some of the HID code. It also supports callibration. How to deploy it with TouchOSC is an exercise left to the reader.
http://www.berkeleynoise.com/celesteh/code/WiiOSCClient.sc