My initial plan has been to use Ableton live and Resolume arena together as I wanted to have the viewer actually remix on the fly. Initially looking at it I thought it would be great to trigger video and sections of songs. I attempted to put this together and have found that the Leap Motion only sends through CC messages (a continuous stream of data which has a value between 1-127) rather than piano note data (a single button press if you will, usually used to trigger samples or video clips). There may be ways to get around this, but due to time and ease, I’ve decided to stick to a pre arranged sequence and manipulate the video and audio with effects and plugins.
Attempting to put this all together I’ve encountered some issues. I originally exported my video at 1080p. I wanted it to be as sharp as possible. I had to convert my video into DVX format which is a proprietary format from Resolume and apparently allows it to play better. Previously I have found that a JPG sequence has worked well playing back in Ableton live’s video window. Avoiding H.264 as, although great for streaming, is terrible for programs like Resolume or Ableton. I’ve found though that the 1080p video is a little jerky, so may try it at 720p.
Setting up the controls for both Resolume and Ableton has been tricky as I’ve had to go via Geco, Leap Motion’s Midi controller software. It does work well, but there are so many options open to you, that you don’t know how people are going to use it, especially as they have no proper training and it’s just “wave your hands over this thing”. I’ve considering using a small picture which shows what to do, without stopping people from experimenting. Presently I have various controls mapped to movements, a distortion on the video matching a distortion on the audio. This is proving very tricky though as I’m manipulating several streams of video and audio.
All this said, I think I’ve figured out the problem. the controller would jump channel numbers if I mapped to another effect. I can solve this by mapping to Resolume’s control panel and then matching that to a global effect rather than a specific effect.