LiveBlogging: Modality – modal control in SuperCollider

by many people

Modality is a loose collaboration to make a toolkit to hook up controllers to SC.  Does mapping, including some complex stuff and some on-the-fly stuff.

Marije spoke a bit of how they began collaborating

Concept – support many devices over many protocols. Make a common interface. Easily remap.

Devices

They currently support MIDI and HID. the common interface is MKtl. Provides a system to process the data. They have templates. Templates for common ways of processing. Same interface for MKtl and MDispatch. (they may move to FRP (I don’t know what that is))

Ktl quark is out of date.

(I think I might be interested in contributing to this project – or at least provide templates for stuff)

Different protocol have different transport mechanisms. Things very by OS. Different controllers have different semantics.

A general solution is not trivial.

Scaling is different on different OSes. Names of devices may have variations. MIDI has some device name issues.  real MIDI (non-usb) will not report their names, but use MIDI ports.  Similar issues will arise with OSC or SerialPort. 

The device description index is an identity dictionary. It’s got some NanoKontrol stuff in it. I am definitely interested in this…

They’ve got some templates, but it’s still a bit vapourware.

For every button or input on your device, they define what it is, where it is, etc.  This is good stuff.  You can also set the I/O type.

Device descriptions have names, specifications, platform differences, hierarchical naming (for use in pattern-matching). You can programmatically fill in the description

nanoKontrol, Gamepad, DanceMat, a bunch of things.

Events and signals

Functional reactive processing. Events, data flow, change propogation. FRP – functional reactive programming

These are functions without sideFX until you get to the output phase.

In the FP Quark – functional programming Quark.

Events are encoded in an event stream.  Event Source with a do method adds a side effect.  When somethng happens (is “fired”), do the do.  Only event sources can be fired.

the network starts with an event source. 

Signals are similar but have state? You can ask for the value and change it.

To create the network use combinators.

inject has state internally.

Dynamic Event Switching limits and event depending on a selector.  this is kind of like the gate thing in max.

With Modality, every control has an elements, every element has a singal and a source. Controls have keys.

You can combine values, attach stuff to knob changes. Easy to attach event streams to functions.

this is complex to describe, but works intuitively in practice.  You can do deltas, accumulators, etc.

Closing remarks

this is on github, but it not yet released.  depends on the FP quark.

Needs gui replacements.  Needs a backend for OSC devices.

Needs some hackin in the SC source.

Questions

  • Would you be interested in doing the descriptors in JSON, so it can be used by non-SC guys? Yeah, why not.  This is a good plan, even.

WiiOSCClient.sc

Because there are problems with the wiimote support in SuperCollider, I wrote a class for talking to Darwiin OSC. This class has the same methods as the official wiimote classes, so, should those ever get fixed, you can just switch to them with minimal impact on your code.
Because this class takes an OSC stream from a controller and treats it like input from a joystick, this code may potentially be useful to people using TouchOSC on their iPhones.
There is no helpfile, but there is some usage information at the bottom of the file:


 // First, you create a new instance of WiiOSCClient, 
 // which starts in calibration mode
 
 
 w = WiiOSCClient.new;

 // If you have not already done so, open up DarwiinRemote OSC and get it talking to your wii.
 // Then go to preferences of that application and set the OSC port to the language port
 // Of SuperCollider.  You will see a message in the post window telling you what port
 // that is .... or you will see a lot of min and max messages, which lets you know it's
 // already callibrating
 
 // move your wiimote about as if you were playing it.  It will scale it's output accordingly
 
 
 // now that you're done callibrating, turn callibration mode off
 
 w.calibrate = false;
 
 // The WiiOSCClient is set up to behave very much like a HID client and is furthermore
 // designed for drop-in-place compatibility if anybody ever sorts out the WiiMote code
 // that SuperCollider pretends to support.
 
 // To get at a particular aspect of the data, you set an action per slot:
 
 w.setAction(ax, {|val|
  
  val.value; // is the scaled data from ax - the X axis of the accelerometre.
  // It should be between 0-1, scaled according to how you waved your arms during
  // the callibration period
 });
 
 
 
 // You can use a WiiRamp to provide some lag
 (
  r = WiiRamp (20, 200, 15);
 
  w.setAction(ax, {|val|
   var scaled, lagged;
  
   scaled = ((val.value * 2) - 1).abs;
   lagged = r.next(scaled);
  
   // now do somehting with lagged
  });
 )

Calibration

this class is self-calibrating. It scales the wiimote input against the largest and smallest numbers that it’s seen thus far. While calibration is set to true, it does not call any of its action methods, as it assumes the calibrated numbers are bogus. After to set calibration to false, it does start calling the actions, but it still changes the scale if it sees a bigger or smaller number than previously.

WiiRamp

The WiiRamp class attempts to deal with the oddness of using accelerometers, but it does not just do a differentiation, as that would be too easy. The accelerometers give you major peaks and valleys, all centred around a middle, so just using the raw values often is a bit boring. In the example, you see that we scale the incoming data first: ((val.value * 2) – 1) changes the data range from 0 to 1 into -1 to 1. The puts the centre on 0. Then, because we care more about the height of peaks and depth of valleys than we care about whether they’re positive or negative, we take the absolute value, moving the scale back to 0 to 1.
When you shake your wiimote, the ramp keeps track of your largest gesture. It takes N steps to reach that max (updating if a larger max is found before it gets there), then holds at the number for M steps and then scoots back down towards the current input level. You can change those rates with upslope, hold and downslope.

OscSlot

This class is the one that might be useful to iPhone users. It creates an OSCResponderNode and then calls an action function when it gets something. It also optionally sends data to a Bus and has JIT support with a .kr method. It is modelled after some of the HID code. It also supports callibration. How to deploy it with TouchOSC is an exercise left to the reader.
http://www.berkeleynoise.com/celesteh/code/WiiOSCClient.sc

HIDden Options

I’m trying to plug a new joystick into SuperCollider. I got a Logitech Attack 3 joystick which I want to do a short act with, maybe next week at a drag king bar. But I can’t get SuperCollider to talk to it.

The newer version of SC broke all of my joystick code. That’s fine, except I can’t get the newer stuff to work. When I try running the examples under GeneralHID, it can see my joystick and knows about all the buttons and the XYZ stuff, but it doesn’t seem to notice when I push one of those buttons or wiggle the stick. I tried the joystick briefly with JunXion, so I know it works, but SC just isn’t getting data from it.
I wonder if there’s some sort of trick or secret to this? I had to switch my audio stuff to an aggregate device to read in and out. Is there something like that for HIDs? Some secret magic?

HID, SuperCollider and whatnot

My HID classes are now in version 1.0 and I’m not going to change them again without good reason. The change is that the call back action passes the changed element back as the first parameter, followed by all the other parameters. It’s redundant, but it matches how the subclasses work. This should be usable for HID applications. a helpfile will be forthcoming.
I am trying to write a Biquad filter UGen and also to compile a FFTW UGen that my friend wrote. Jam fails when I try to compile, on both files. I think there is a step missing in the howto document on Ugen writing. My code is simple and the syntax looks ok. I’ve done every step (as far as I know) in the howto. bah. I could use a better error message than “jam failed.” sheesh. I like XCode, but it’s got nothing on the Boreland compilers I used to use back in the 90’s as far as usability and usefulness of error messages.
Workaround: Take an existing target and duplicate it and use that for your target. Rename it and replace the files in it with your own. Ctrl-click on the target to bring up a menu with the option to duplicate.
Tags: ,

HID Update

Download a new version of my HID classes.
So I think some of my original features were overzealous, so some of them have been removed. For instance, no more specifying a ControlSpec for an HIDElementExt. If you want to use a ControlSpec, you should attach the HIDElementExt (or HIDElement Group) to a CV by means of the action. Remember that CVs are really useful objects. See their help file for more information. For example of how to attach one:

element.action_({arg vendorID, productID, locID, cookie, value;

 var scaled, mycv;
     
 scaled = element.scale(value);
         
 mycv = cvDictionary.at(element.usage.asSymbol);
 mycv.input_(scaled);
});

Note the name and order of arguments, as this is a change. In this example, cvDictionary is an IdentityDictionary where CVs are stored according to the element.usage as a key. scale is a message you can pass to an element in which a value is scaled according to the min and max to be a number between 0 and 1. You can still set the min and max of an element based on what you’ve discovered to be the range of your HID, regardless of what lies the device may whisper in your ear about it’s ranges.
The HIDDeviceExt.configure message was not really needed, so it’s gone.
Here is some code to automatically generate a GUI for every HID that you have attached. Note that you will need the Conductor class for it, which is part of the Wesleyan nightly build. This GUI will give you one line for every reported piece of the device. You can then try moving everything around to see what their actual ranges are (if they are smaller than reported) and the correlation between reported names and the elements.

(

 var elems, names, cv, conductor;

 HIDDeviceServiceExt.buildDeviceList;
 
 HIDDeviceServiceExt.devices.do({ arg dev;
 
  elems = [];
  names = [];

  dev.queueDevice;

  dev.elements.do ({ arg elm;
  
   cv = CV.new.sp(0, elm.min, elm.max, 0, 'linear');
   elems = elems.add(cv);
   names = names.add(elm.usage.asSymbol);
   elm.action_({ arg vendorID, productID, locID, cookie, value;
     
    var scaled, mycv;
     
    scaled = elm.scale(value);
         
    mycv = conductor.at(elm.usage.asSymbol);
    mycv.input_(scaled);
   });

  });
 
  conductor = Conductor.make({ arg cond; 
  
   elems.do({ arg item, index;
   
    cond.put(names[index], item);
   });
   
   cond.argList = cond.argList ++ elems;
   cond.argNames = cond.argNames ++ names;
   cond.valueItems = cond.valueItems ++ names;
   cond.guiItems = cond.valueItems;
  });
  
  conductor.show;
  
 });
 
 HIDDeviceServiceExt.runEventLoop;

)

Then stop it with:

HIDDeviceServiceExt.stopEventLoop;

But what if the returned values are outside of the range? And what are the cookies? You should then generate a little report to correlate everything.

(
 HIDDeviceServiceExt.devices.do({arg dev;
  [dev.manufacturer, dev.product, dev.vendorID, dev.productID, dev.locID].postln;
  dev.elements.do({arg ele;
   [ele.type, ele.usage, ele.cookie, ele.min, ele.max, ele.minFound, ele.maxFound].postln;
  });
 });
)

You can use the minFound and maxFound data to calibrate the device.
This is my current code to configure my (crappy) joystick:

HIDDeviceServiceExt.buildDeviceList;
HIDDeviceServiceExt.queueDeviceByName('USB Joystick STD');

(

 var thisHID, simpleButtons, buttons, stick;
 
 thisHID = HIDDeviceServiceExt.deviceDict.at('USB Joystick STD');
 simpleButtons = HIDElementGroup.new;
 buttons = HIDElementGroup.new;
 stick = HIDElementGroup.new;
  
 thisHID.deviceSpec = IdentityDictionary[
   // buttons
   trig->7, left->9, right->10, down->8, hat->11,
   // stick
   x->12, y->13,
   wheel->14,
   throttle->15
  ]; 
  
  
 simpleButtons.add(thisHID.get(trig));
 simpleButtons.add(thisHID.get(left));
 simpleButtons.add(thisHID.get(right));
 simpleButtons.add(thisHID.get(down));
  
 buttons.add(simpleButtons);
 buttons.add(thisHID.get(hat));
  
 stick.add(thisHID.get(x));
 stick.add(thisHID.get(y));
  
 // make sure each element is only lsted once in the nodes list, or else it
 // will run it's action once for each time a value arrives for it
  
 thisHID.nodes = [ buttons, stick, thisHID.get(wheel), thisHID.get(throttle)];
  
 // When I was testing my joystick, I noticed that it lied a bit about ranges
  
 thisHID.get(hat).min = -1;  // it gives back -12 for center, but -1 is close enough
  
 // none of the analog inputs ever went below 60
  
  
 thisHID.get(wheel).min = 60;
 thisHID.get(throttle).min = 60;
 // can set all stick elements at once
 stick.min = 60;
)

Setting the min and the max for each element is used for the scale message. If you don’t want to bother scaling the values to be between 0 and 1 (or some other range) and you set the threshold to 0, then you can skip setting the min and the max.
Here is a silly example of using the x and y axis of my joystick to control the frequency and amplitude of a Pbind in a GUI. Note that it would work almost identically without the GUI.

(

 HIDDeviceServiceExt.runEventLoop;

 Conductor.make({arg cond, freq, amp;
 
  var joystick;
 
  freq.spec_(freq);
  amp.spec_(amp);
  
  joystick = HIDDeviceServiceExt.deviceDict.at('USB Joystick STD');
  
  joystick.get(y).action_({ arg vendorID, productID, locID, cookie, value;
  
   freq.input_(joystick.get(y).scale(value));
  });
  
  joystick.get(x).action_({ arg vendorID, productID, locID, cookie, value;
  
   amp.input_(joystick.get(x).scale(value));
  });
  
  cond.name_("silly demo");
  
  cond.pattern_( Pbind (
  
   freq, freq,
   amp, amp
  ));
  
 }).show;
  
)

Don’t forget to stop the loop when you’re done:

HIDDeviceServiceExt.stopEventLoop;

As you may have guessed from this EventLoop business, HIDs are polled. Their timing, therefore, may not be as fast as you’d like for gaming. They may be more suited to controlling Pbinds like in the silly example, rather than Synth objects. You will need to experiment on your own system to see what will work for you.
The HIDElementExt all have a threshold of change below which they will not call their actions. This defaults to 5%, which is a good value for my very crappy joystick. You can set your own threshold. It’s a floating point number representing a percent change, so you can put it between 0 and 1. Again, you may want to experiment with this. Most HIDs have a little bit of noise and flutter where they report tiny actions where none occurred. Also, do not buy a 5€ joystick from a sketchy shop. It’s range will suck and it will will flutter like crazy and it will be slow and you will end up buying a more expensive joystick anyway.

joystick.threshold_(0.01); // 1% 

Tags: , ,

HID, Joysticks and SuperCollider

So, game controllers and joysticks and the like are called Human Interface Devices (HID). There is some HID stuff built into SuperCollider, but it seemed sort of half implemented. They let you specify things, but not do much with it. So I wrote an extension yesterday, which maybe I should submit as something official or a replacement, although I didn’t bother checking first if anybody else had already done anything. No internet access at home sucks.

To use it, download HID.sc and put it in ~/Library/Application Support/SuperCollider/Extensions. Plug in your joystick or controller. Start SuperCollider.
First, you want to know some specifications for your device, so execute:


HIDDeviceServiceExt.buildDeviceList;

  
(
HIDDeviceServiceExt.devices.do({arg dev;
  [dev.manufacturer, dev.product, dev.vendorID, dev.productID, dev.locID].postln;
  dev.elements.do({arg ele;
   [ele.type, ele.usage, ele.cookie, ele.min, ele.max].postln;
  });
});
)

I got back:

  [ , USB Joystick STD, 1539, 26742, 454033408 ]
  [ Collection, Joystick, 1, 0, 0 ]
  [ Collection, Pointer, 2, 0, 0 ]
  [ Button Input, Button #5, 3, 0, 1 ]
  [ Button Input, Button #6, 4, 0, 1 ]
  [ Button Input, Button #7, 5, 0, 1 ]
  [ Button Input, Button #8, 6, 0, 1 ]
  [ Button Input, Button #1, 7, 0, 1 ]
  [ Button Input, Button #2, 8, 0, 1 ]
  [ Button Input, Button #3, 9, 0, 1 ]
  [ Button Input, Button #4, 10, 0, 1 ]
  [ Miscellaneous Input, Hatswitch, 11, 0, 3 ]
  [ Miscellaneous Input, X-Axis, 12, 0, 127 ]
  [ Miscellaneous Input, Y-Axis, 13, 0, 127 ]
  [ Miscellaneous Input, Z-Rotation, 14, 0, 127 ]
  [ Miscellaneous Input, Slider, 15, 0, 127 ]
  [ a HIDDevice ]

Ok, so I know the name of the device and something about all the buttons and ranges it claims to have. (Some of them are lies!) So, I want to assign it to a variable. I use the name.

j = HIDDeviceServiceExt.deviceDict.at('USB Joystick STD');

Ok, I can set the action for the whole joystick to tell me what it’s doing, or I can use the cookie to set the action for each individual element. I want to print the results. In either case, the action would look like:

x.action = { arg value, cookie, vendorID, productID, locID, val;
 [ value, cookie, vendorID, productID, locID, val].postln;
};

You need to queue the device and start the polling loop before the action will execute.

HIDDeviceServiceExt.queueDeviceByName('USB Joystick STD');
HIDDeviceServiceExt.runEventLoop;

It will start printing a bunch of stuff as you move the controls. When you want it to stop, you can stop the loop.

 HIDDeviceServiceExt.stopEventLoop; 

Ok, so what is all that data. The first is the output of the device scaled to be between 0-1 according to the min and max range it reported by the specification. The next is very important. It’s the cookie. The cookie is the ID number for each element. You can use that to figure out which button and widget is which. Once you know this information, you can come up with an IdentityDictionary to refer to every element by a name that you want. Mine looks like:

  j.deviceSpec = IdentityDictionary[
   // buttons
   a->7, left->9, right->10, down->8, hat->11,
   // stick
   x->12, y->13,
   wheel->14,
   throttle->15
  ]; 

The last argument in our action function was the raw data actually produced by the device. You can use all your printed data to figure out actual ranges of highs and lows, or you can set the action to an empty function and then restart the loop. Push each control as far in every direction that it will go. Then stop the loop. Each element will recall the minimum and maximum numbers actually recorded.

(
j.elements.do({arg ele;
   [ele.type, ele.usage, ele.cookie, ele.min, ele.max, ele.minFound, ele.maxFound].postln;
  });
)

This gives us back a table like the one we saw earlier but with two more numbers, the actual min and the actual max. This should improve scaling quite a bit, if you set the elements min and max to those values.   ele.min = ele.minFound;   Speaking of scaling, there are times when you want non-linear or within a different range, so you can set a ControlSpec for every element. Using my identity dictionary:

j.get(throttle).spec = ControlSpec(-1, 1, step: 0, default: 0);

Every element can have an action, as can the device as a whole and the HIDService as a whole. You can also create HIDElementGroup ‘s which contain one or more elements and have an action. For example, you could have an action specific for buttons. (Note that if you have an action for an individual button, one for a button group and one for the device, three actions will be evaluated when push the button.) Currently, an element can only belong to one group, although that might change if somebody gives me a good reason.
This is a lot of stuff to configure, so luckily HIDDeviceExt has a config method that looks a lot like make in the conductor class. HIDDeviceExt.config takes a function as an argument. The function you write has as many arguments as you want. The first refers to the HIDDeviceExt that you are configuring. The subsequent ones refer to HIDElementGroup ‘s. The HIDDeviceExt will create the groups for you and you can manipulate them in your function. You can also set min and max ranges for an entire group and apply a ControlSpec to an entire group. My config is below.

 j.config ({arg thisHID, simpleButtons, buttons, stick;
  
  thisHID.deviceSpec = IdentityDictionary[
   // buttons
   a->7, left->9, right->10, down->8, hat->11,
   // stick
   x->12, y->13,
   wheel->14,
   throttle->15
  ]; 
  
  
  simpleButtons.add(thisHID.get(a));
  simpleButtons.add(thisHID.get(left));
  simpleButtons.add(thisHID.get(right));
  simpleButtons.add(thisHID.get(down));
  
  buttons.add(simpleButtons);
  buttons.add(thisHID.get(hat));
  
  stick.add(thisHID.get(x));
  stick.add(thisHID.get(y));
  
  // make sure each element is only lsted once in the nodes list, or else it
  // will run it's action once for each time a value arrives for it
  
  thisHID.nodes = [ buttons, stick, thisHID.get(wheel), thisHID.get(throttle)];
  
  // When I was testing my joystick, I noticed that it lied a bit about ranges
  
  thisHID.get(hat).min = -1;  // it gives back -12 for center, but -1 is close enough
  
  // none of the analog inputs ever went below 60
  
  
  thisHID.get(wheel).min = 60;
  thisHID.get(throttle).min = 60;
  // can set all stick elements at once
  stick.min = 60;
      
  // ok, now set some ControlSpecs
  
  thisHID.get(hat).spec = ControlSpec(0, 4, setp:1, default: 0);
  // set all binary buttons at once
  simpleButtons.spec = ControlSpec(0, 1, step: 1, default: 1);
  
  // all stick elements at once
  stick.spec = ControlSpec(-1, 1, step: 0, default: 0);
  thisHID.get(wheel).spec = ControlSpec(-1, 1, step: 0, default: 0);
  thisHID.get(throttle).spec = ControlSpec(-1, 1, step: 0, default: 0);
  
  thisHID.action = {arg value, cookie, vendorID, productID, locID, val;
  
   [value, cookie, vendorID, productID, locID, val].postln;
  };
   
 });

I wrote all of this yesterday, so this is very demo/pre-release. If something doesn’t work, let me know and I’ll fix it. If people have interest, I’ll post better versions in the future.
Tags: , ,

Heh heh. She said “Joystick”

Is it me, or is everything about game devices loaded with innuendo? “Joy stick?” I mean, really. I went out yesterday and attempted to purchase one called the “thrust master.” Sublimation much?
No, I have not discovered the joys of gaming. (Well, I briefly became interested in a Soduku widget, but it got old fast. Closely related to minesweeper but without the smiley face or the implied violence. bah.) If I gamed, I would not do anything else. It’s difficult for me to compartmentalize.
Anyway, I found a bargain joystick with 2 degrees of A->D conversion for the stick and another degree for the throttle. I have plans to use it with my tuba (or, rather, my formerly bubblewrapped sousaphone). If the stick proves too awkward / stupid, then different variable resistors, such as ribbons can be put in place of where the stick went.
Right now, though, I am giggling like a middle schooler at game device terminology. If I remove the plastic of the handle and replace it with a cyberskin packer, would this be over-the-top for performance? What if it was crotch-mounted?
(For those of you wondering what a “cyberskin packer” is: It’s a limp, squishy prosthetic penis that one might use for stuffing their trousers. These are used by Drag Kings, FTMs, cisgender males with injuries, and people who know how to get a party started. And very serious gamers, of course.)
Tag: