Using the Leap Motion device to input 3D data, a MidiFighter Classic to provide button input, and Touch Designer to drive the visuals.

The top row of the MidiFighter records input when held down, and the buttons in the row just below play them back when held down. Everything is momentary for now.

I’m really digging the combo of the MidiFighter and the Leap – they have complementary strengths. Still have much work to do like smarter handling for the switch between live and recorded data, and enabling toggle vs momentary recordings.

Music is ‘This Beat’ by Alex Beroza – Licensed CC-BY-NC

As the old story goes, there never seems to be time to put together a proper reel.
So here is a sample of what my work is like now.

These projects are all on Vimeo, so browse through my videos for more on each one.

1. FiddlyBits – Cinder dev + design, Performance
2. The Trip – Cinema4D Animation, Quartz Composer PreViz
3. FiddlyBits iOS – openFrameworks dev + design, Performance
4. Beyond Right Now – openFrameworks dev + design, Animation
5. Lighting Experiment – TouchDesigner production using Leap Motion
6. Three-Oh-Five – TouchDesigner production with Ableton Live integration
7. Bridge Invaders – Android/AIR Development + Design
8. Countdown – After Effects Animation and Illustration
9. Cydrop – AIR dev + design
10. Touch Me – Flash Animation, AIR software dev, Ableton Live integration
11. Ho-Lee-Cow – Isadora dev, Ableton Live integration, AV Performance
12. Push & Rise – After Effects Animation
13. You Should Smell The Air – After Effects Animation
14. The Glitch Mob – openFrameworks dev, Live Visuals Performance
15. Deadmau5 – Playback System Design, Live Visuals Performance
16. AV Collaboration – Flash Animation, AIR software dev, Ableton Live integration
17. Make Space Ship – openFrameworks iOS dev + design, Live Art

The Problem

The app is meant to be used in landscape orientations, so I set:

ofxiPhoneSetOrientation(OF_ORIENTATION_90_RIGHT);

As I’d seen in the openFrameworks examples, and then handled orientation in testApp using deviceOrientationChanged. However, this was causing two different issues when the app was launched in something other than Landscape Right.

When the App launches, I’ve got a Space Ship (the character in the middle of the pictures below) as well as Start and Tutorial buttons. These are made with UIKit in an .xib file called TitleScreen, and overlaid on the live starfield, which is the GLView created by openFrameworks. This was accomplished by adding the TitleScreen to the GLView, like this:

titleScreen = [[TitleScreen alloc] initWithNibName:@"MakeSpaceShip" bundle:nil];

// now move the view to the GLView to allow touches, removed from window automatically
[ofxiPhoneGetGLView() addSubview:titleScreen.view];

Under some circumstances, I’d get the correct TitleScreen orientation, but the starfield would be set up with a portrait orientation, like in this picture:

You can see that the stars do not fill up the full screen – their placement is random within the current screen bounds (using ofGetWidth and ofGetHeight), so they were being initialized at 768×1024 instead of 1024×768. This fixed itself once they started moving around, but I found that unacceptable.

Another circumstance would find the GLView set up correctly, but the TitleScreen would be rotated 90 degrees out of place after it was added to the GLView, like this:

Terrible! I found Damian’s blog post about orientation and tried his method. It worked, but whenever I touched a part of my TitleScreen that was transparent, the touches would not make it through to the openFrameworks view below. These touches worked fine in the original method, so I knew there had to be another way around.

The Solution

I took some of what worked from Damian’s post, and combined it with the method that allowed touches to get through. Here’s how it goes:

There are three classes at work here.

• TitleScreen is a UIViewController which contains the UIView for all my UIKit elements, which are present as GUI throughout the app.
• TestApp is the openFrameworks application.
• AppSettings is a C++ file which is injected into all my Manager classes (I’ve got a Manager for the Starfield, the Triangles, the Tutorial, etc.) and holds Application State Settings. It also acts as the communication channel between my TitleScreen and my app. It’s instantiated in the TestApp, and then I inject a pointer to it into every Manager Class when I create them.

I don’t call ofxIphoneSetOrientation in my TestApp setup method at all. here’s what I do:

   // create and initialize titleScreen
    titleScreen = [[TitleScreen alloc] initWithNibName:@"MakeSpaceShip" bundle:nil];

   // inject settings
    titleScreen.settings = settings;
    
    // adding the titlescreen to the main window sets the orientation properly
    [ofxiPhoneGetUIWindow() addSubview:titleScreen.view];

    // now move the view to the GLView to allow touches, removed from window automatically
    [ofxiPhoneGetGLView() addSubview:titleScreen.view];

Previously, I had been adding the TitleScreen to the GLView, which may or may not have already been rotated, and calls to all three methods of getting device orientation gave unpredictable results. Now, I’m adding the TitleScreen directly to the UIWindow at the top of the View hierarchy, so it orients itself correctly. However, leaving it here will cancel touch events going through to the oF app, so once it’s been oriented, I add it to the GLView, which automatically removes it from the Window, since UIViews can only have one SuperView.

There’s just one further piece missing after this, and it’s a method to rotate the GLView. openFrameworks provides a convenience method for this in TestApp called deviceOrientationChanged. However, I couldn’t get this to work right any way I tried. So instead, I added this to the TitleScreen implementation:

- (void) willRotateToInterfaceOrientation:(UIInterfaceOrientation)toInterfaceOrientation duration:(NSTimeInterval)duration {
    settings->setOrientation(toInterfaceOrientation);
    // snap to new rotation
    [UIView setAnimationsEnabled:NO];
}

So once the TitleScreen decides that it’s ready to move to a new orientation, it tells the AppSettings class (via a pointer called settings) to run the setOrientation method using the new orientation. Here’s what that method looks like:

    void setOrientation(int newOrientation) {
        switch (newOrientation) {
            case OF_ORIENTATION_90_LEFT:
                ofxiPhoneSetOrientation(OF_ORIENTATION_90_RIGHT);
                break;
                
            case OF_ORIENTATION_90_RIGHT:
                ofxiPhoneSetOrientation(OF_ORIENTATION_90_LEFT);
                break;
        }
    }

I’m using the ofxIphoneSetOrientation method, and flipping left and right since they seem to report backwards to what I expect. And with that, my app launches with correct orientation no matter which way the iPad is facing (Portrait, Upside-Down, Landscape-Left, Landscape-Right, sitting flat on the desk in Upside-Down mode, etc.), and correctly snaps to the right orientation if you flip it around. The final correct intro screen:

Woohoo!

I’ve learned a LOT through making this Application, so I’ll be documenting my experience here as time allows. I’ve got a new theme in the wings, too (this one dates back several years), and this blog will eventually be merged into the main mmmlabs.com site, which was not easily accomplished with the current theme there.

Hope this helps someone out!

Join us tonight at theSlate in NW Portland and try out our two digital drawing installations while you munch on some Koi Fusion fantasticness.

Free
Thursday, December 9th
6pm-9pm
2001 NW 19th Ave #104

I’m proud to announce the beta release of MMMLabs’ first Android Application – Bridge Invaders!
It’s a lo-res Drawing and Animation application Check out the video below for a taste:

So you draw little 5-by-5 animations frame-by-frame, and then you can play them back, and tap in the tempo.
You can save and load – to manage animations beyond that, you can do it directly – they’re all saved as XML files in SDCARD/BridgeInvaders

Please let me know if you have any problems or feature suggestions aside from the following:

• Feature: Copy/Paste Frames
• Feature: Change Mirroring Modes
• Feature: Delete Saved Files from GUI
• Issue: Previous/Next Frame Buttons are laggy
• Issue: Large animations take a LONG time to load

Geeky Details

Bridge Invaders is developed using AIR for Android, which enables Actionscript 3 applications to run on the Android platform (AS3 is commonly known as the language that powers Flash). This application was originally developed as an installation for the PDX Bridge Festival as a large-scale interactive video project. Thanks to WKE for sponsoring some of the development, FDT for their awesome dev tools, Robotlegs for keeping the code manageable, the Hype Framework for the shortcuts, and to Jared Tarbell and Invader for the inspiration.

Direct Download

If you cannot find it in the market, it means it probably will not work on your phone.

HOWEVER – if you want to install it anyway just to try, you can download it here: Bridge Invaders for Android

I’m currently developing a system for playback and control of live visuals using Ableton Live and Flash – more on the details of this project soon. One of the core needs is to have controls in one window and playback  in another, which is piped out to a projector.

So upon starting my Storyteller AIR application, I load controls into my main window, and spawn a secondary Utility window which holds all the movies for playback. These are all added to a single view object I call the PlayBox. The problem arises when the window is resized or full-screened. AIR doesn’t like using anything besides StageScaleMode.NO_SCALE, and Adobe’s documentation discourages using any other scaling mode.

Fighting the System

What I really want is StageScaleMode.SHOW_ALL, so that the whole of each movie will be shown, no matter the aspect ratio of the window, with black bars at the top or sides as necessary. Unfortunately, using this scale mode resulted in HUGE movies, of which I could see only a corner. I thought I’d hit the jackpot when I found this Adobe documentation which spelled out what was going on and how to fix it:

“…When you create a NativeWindow object, AIR chooses an arbitrary relationship between the window size and the scale factor of 72:1. Thus, if your window is 72×72 pixels, a 10×10 rectangle added to the window is drawn the correct size of 10×10 pixels. However, if the window is 144×144 pixels, then a 10×10 pixel rectangle is scaled to 20×20 pixels. If you insist on using a 
scaleMode

other than 
noScale

for a window stage, you can compensate by setting the scale factor of any display objects in the window to the ratio of 72 pixels to the current width and height of the stage. For example, the following code calculates the required scale factor for a display object named 
client

:

if(newWindow.stage.scaleMode != StageScaleMode.NO_SCALE){
client.scaleX = 72/newWindow.stage.stageWidth;
client.scaleY = 72/newWindow.stage.stageHeight;
}

Unfortunately, this never worked quite right. I applied this formula to my main PlayBox view object before adding any loaded movies to it, but it was never quite tall or wide enough.

Giving In

So I scrapped that idea (after, of course, hours of trying to get this work way too late at night when my brain was fried and wouldn’t give it). The new plan was to use NO_SCALE, and instead listen for window resizing and adjust the PlayBox size and position to match.

It was still not as straightforward as I thought. The idea was simple enough – listen for NativeWindowBoundsEvent.RESIZE, take a look at the new afterBounds or stageWidth, scale to match, and center. The problem was in centering the PlayBox. The formula should be:

view.x = (rect.width * 0.5) - (view.width * 0.5);

Where view is the PlayBox, and rect are the new bounds. I swear this was working earlier in development. The issue is that the view’s width does not directly correlate to anything I could see. If I were to take the bottom edge of the window and pull it straight down (making it taller), the view’s width would fluctuate in a Sine-like pattern. This means that I couldn’t do any calculations involving the width of the view.

The Answer

Ultimately, I devised a solution that works, though its reliance on constants isn’t ideal. I store two numbers for the original width and height of the window and stage – 640 and 480, in this case. These go in a Preferences model I’ll call prefs. On resize, I generate a newScale variable like this:

var newScale : Number = window.stage.stageHeight / prefs.playHeight;

Then I can apply it to the PlayBox:

view.scaleX = newScale;
view.scaleY = newScale;
view.x = (view.stage.stageWidth - (prefs.playWidth *  newScale)) * 0.5;

By taking the original playWidth variable and multiplying it by the new scale, I get a reliable width for the PlayBox. Then I subtract the width of the PlayBox from the width of the stage, which gives me the total number of ‘extra pixels’ I have. Divide this by two (aka multiply by 0.5) and we’ve got the number of pixels that should be to the left of the PlayBox in order to center it. Phew!

Throwing Beam for Fun and Profit
http://panelpicker.sxsw.com/ideas/view/7548

Motion Paintings, Visual Storytelling and Deaf Film
http://panelpicker.sxsw.com/ideas/view/7886

Beyond 3D: How Interactive Screenings Will Save Cinema
http://panelpicker.sxsw.com/ideas/view/7020

DIY Motion Tracking for Public Spaces
http://panelpicker.sxsw.com/ideas/view/7003

People as Peripherals: The Future of Gesture Interface
http://panelpicker.sxsw.com/ideas/view/5716

Developing Futuristic Multi-Touch / Multi-Screen Applications
http://panelpicker.sxsw.com/ideas/view/6154

Interactive Narratives: Creating the Future of Literature
http://panelpicker.sxsw.com/ideas/view/6462

3D Interfaces: A Primer on Stereoscopic Interaction Design
http://panelpicker.sxsw.com/ideas/view/5934

Data Visualization: Sci-Fi Interfaces in the Real World
http://panelpicker.sxsw.com/ideas/view/7358

Toss the Projector: Redefining the Presenter/Audience Dynamic
http://panelpicker.sxsw.com/ideas/view/6389

Hybridizing Interaction with Motion Design: User Interfaces
http://panelpicker.sxsw.com/ideas/view/5568

The Future of Touch User Interface Design
http://panelpicker.sxsw.com/ideas/view/5984

Fun With The Lights Off: Interactivity Without Graphics
http://panelpicker.sxsw.com/ideas/view/7208

Getting Touchy: Going Beyond the Keyboard and Mouse
http://panelpicker.sxsw.com/ideas/view/6662

Creating Interactive Art Through Social Media Collaboration
http://panelpicker.sxsw.com/ideas/view/7239

The Music of Interaction Design
http://panelpicker.sxsw.com/ideas/view/6643

Realtime Interactivity for Music, Broadcast and Art
http://panelpicker.sxsw.com/ideas/view/5400

Realtime Mixing In Video Games
http://panelpicker.sxsw.com/ideas/view/6648