July 30, 2011

Syncing your real-time graphics right

Note: this article is based on the talk “GNU Rocket Science” held by Erik Faye-Lund at The Gathering 2011 as well as the personal experiences of the author.

For me, as a musician and designer, the most exciting part of the visual aspects of making a demo is undoubtedly the direction, editing and synchronization with the music. The problem is of course that I’m not a coder. Enter GNU Rocket – the general purpose, ad-hoc tool that gives non-coders an intuitive interface to play around, in a very powerful way.

What is it?

GNU Rocket is what we have given the nickname “sync tracker” – that is because all of this is commonly referred to as “syncing” in the demoscene. For anyone with a background in electronic music on the Atari, Amiga or PC using “trackers”, the concept should be very familiar. Instead of having a set of horizontal channels, like working in Logic, Pro Tools or the likes, you work vertically. Each column is a channel, and each channel consists of a series of horisontal “steps”.

Each step can contain a note or an effect, and when you press play, the tracker traces from top to bottom, playing each note or effect when it encounters it.

This is what the GNU Rocket interface looks like:

Instead of notes or samples, each column represents a controllable variable in the graphics engine — and that’s pretty much all there is to it. If you understand this concept then you can safely move on. If you don’t, you can watch this somewhat dull video where someone uses Renoise to make “music”. When he starts inputing samples — just imagine that’s a keyframe in a video. That’s right – that’s how it works.

The tech

GNU Rocket consists of two elements: an API and an editor. The API is non-intrusive, allowing coders to be coders, and GNU Rocket tries very hard to stay out of their way. The API is versatile, engine agnostic and anti-semantical by nature – the tool doesn’t care what you edit, everything is just floating-point variables. The downside of this is that you can’t do high level editing, but that can also be seen as a positive, because it keeps complex decision-making out of the way.

Now to the juicy center: the client is a network server, and the API library is a network client. That’s it. Very simple, but very powerful. This is a typical workflow:

  • The client connects to the server and tells it what variable it uses
  • The editor sends the data for each variable it has recorded to the client
  • While both ends are running, any change done in the editor is instantly communicated to the client
  • In addition to that, some control data are also sent, including play position (“current row”) and run data (“play/pause”)
  • The control data can be expanded to incude more elements in the future

The API, explained

When using the API, there are really only a few elements you need to keep track of. “sync” is the path-prefix, and the actual file-name is [prefix]_[track-name]. “localhost” and SYNC_DEFAULT_PORT is the host and port to it.

struct sync_device *rocket;
rocket = sync_create_device("sync");
if (sync_connect("localhost",
/* Handle error */

You need to implement hooks to your music-player to use GNU Rocket. It ships with example implementations for BASS by Ian Luck (an excellent library – when in doubt, use it). “rbp” is “rows per beat”, a value of 8 is usually sufficient, and “sync_update” does the network updating.

static void pause(void *d, int flag)
/* player specific */
static void set_row(void *d, int row)
/* player specific */
static int is_playing(void *d)
/* player specific */
static struct sync_cb cb = {
pause, set_row, is_playing
/* ... */
while (1) {
  double time = /* player specific */
  double beat = time * (bpm / 60);
  double row = beat * rpb;
  int irow = (int)floor(row);
  sync_update(rocket, irow, &cb, ...)
  /* ... */

“sync_track” is a handle to a variable in the track data. “sync_get_track” can only fail when loading from disk, if a track has not been (correctly) stored. The “row”-variable depends on your music-player, but is a function of the playback time, your BPM and “row rate” (i.e how many rows to have per beat in the music).

const sync_track *red_track;
red_track = sync_get_track(rocket, "red");
/* ... */
while (1) {
  double row = /* ... */
  sync_update(rocket, ...);
  /* ... */
  float red;
  red = sync_get_val(red_track, row);
  /* ... */

Now you should have a working editor-executable – congratulations! At this point I should mention that while the editor GUI is compiled for Windows, it should be very easy to port to OSX or Linux. The GNU Rocket system itself has been used in productions ranging from full-size demos, down to 64k and 4k intros. It has also been used in demos for Flash and Nintendo GameBoy Advance.

Using the editor

Now that we have all of the tech-talk out of the way, it’s time to actually use the editor to get some work done. The overall principles are frighteningly simple:

  1. Start the editor
  2. Start the demo executable (available tracks should appear in the editor)
  3. Edit tracks
  4. Stop the demo, modify source code, add tracks
  5. Restart demo
  6. Changes and new tracks should appear
  7. The demo should start up at the same position as you left it

..and when the time comes to release your masterpiece to the general public:

  1. File -> Remote Export
  2. Build release version

Hotkeys and tricks

While using the editor, these are your available hotkeys:

Hotkey Function
Up/Down/Left/Right Move cursor
PgUp/PgDn Move cursor 16 rows up/down
Home/End Move cursor to begining/end
Ctrl+Left/Right Move track
Enter Enter key-frame value
PgUp/PgDn Move cursor 16 rows up/down
Del Delete key-frame
i Switch interpolation mode
Space Pause/Resume demo
Shift+Up/Down/Left/Right Select
Ctrl+C Copy
Ctrl-V Paste
Ctrl+Z Undo
Shift+Ctrl+Z Redo
Ctrl+B Bias key-frame(s)
Shift+Ctrl+Up/Down Quick bias by +/- 0.1
Ctrl+Up/Down Quick-bias by +/- 1
Ctrl+PgUp/PgDn Quick-bias by +/- 10
Shift+Ctrl+PgUp/PgDn Quick bias by +/- 100

Bulding for release

When making a release version, you need to export your tracks to external data-files. With the demo and editor connected, just use File -> Remote Export in the editor. You’ll also need to define a pre-processor variable called SYNC_PLAYER in your demo source. Guard calls to “sync_connect” and “sync_update” with an #ifndef on this, because these functions do not exist when in playback-mode.

The hooks (“sync_cb” etc.) can also be removed if you want, and link to the player-version of the library (this is just the normal version built with SYNC_PLAYER set). Keep in mind that this differs if you’re using the .NET bindings.

That’s it – you’re now done with the implementation. I will write a follow-up post in the near future on how to use the tool in a good way, but for now: get integrating — download GNU Rocket here and use it in your own projects.

About the author, Bent Stamnes

Bent is a technologist, demoscener, musician and a real-time graphics evangelist. He has spoken at several high-profile events and companies such as FMX, FITC, Assembly, the Bay Area Visual Effects Society, Pixar, ILM, Adobe and NVIDIA. He arranges the Solskogen demoscene event in Norway and lives just outside of Oslo with his wife and daughter. Sometimes, when the cold Norwegian winter rages on, he wishes he lived in California.

24 Comments Post a comment
  1. avatar
    Aug 22 2011

    Keyboards has even hacked the system to make it suitable for 4k’s.. was used in their intros Texas (http://pouet.net/prod.php?which=51448) and 4096: Electric Space Odyssey (http://pouet.net/prod.php?which=57313). Fantastic workflow.

  2. avatar
    Aug 22 2011

    I advise just compiling those few files along with your project instead of linking to a separate lib. It’s cute but — and mind you I haven’t used it really beyond integrating and testing — what to do when timelines get long and parameters get numerous? I can imagine that editing them will become a bit of a drag. Unless using a code-driven solution to split them up into slices or categories or what have you. Any experiences anyone wants to share? :)

  3. avatar
    Aug 22 2011

    niels: actually, I’ve used this on some pretty complex demos and I’ve never had any issues with keeping track. It’s surprisingly easy. I believe Kusma added support for bookmarks and collapse/expand of tracks, but I haven’t needed it so far.

  4. avatar
    Aug 22 2011

    As an effect coder rather than a framework coder, I give this tool a thumb up :) Being able to hand an un-synced demo over to someone who knows and loves the rocket is great fun.

    Bent, did Kusma implement collapsing? I think that would be a really nice addition to the tool.

  5. avatar
    Aug 22 2011

    Nope, I didn’t implement collapsing yet. I did implement bookmarks, but that feature has not yet been merged to the main-branch. Mostly because I’m lazy.

  6. avatar
    Ebbe Berge Smith
    Aug 23 2011

    This is one of my favorite applications on my computer! It has made me grin like a monkey several times (and hopefully lots of times to come). Everyone who makes demos/intros should try this tool. Tweaking graphics is suddenly a _very_ fun thing to do!

    @Kusma: Looking forward to that merge. Collapsing would also rock! Name your price in beer! ;)

    @Niels: As soon as you figure out how to use the parameters in an efficient way, its not a problem at all to keep track, as Gloom stated earlier. I had some problems in the very start though, but it’s a very progressive learning curve!

  7. avatar
    Aug 23 2011

    I still have big problems to synch my effect on a track made with V2, I didn’t find any good solution for the moment :(

    Synching was easier when I used minifmod with xm modules ;)

  8. avatar
    Aug 23 2011

    I was primarily thinking about creating tracks related to specific events in the music (e.g. simply “snare”). That would create some fine building blocks. Then perhaps custom tracks per part/scene for global timing and stuff you’d like to tweak with nonstandard values and different interpolation modes (I can imagine things getting freaky when you break out of that [0, 1] range :)).

  9. avatar
    Aug 23 2011

    I’m still looking forward to implement my own syncing tool – the same way I ultimately enjoyed the switch from trackers to “pro” DAWs, I can imagine I would imagine using a DAW-like interface better than a tracker-like one (64th triplet note edits <3)

  10. avatar
    Aug 23 2011

    Anyway a follow-up on clever usage would be great.

  11. avatar
    Aug 23 2011

    niels: I’ll definitely write a follow-up piece. In terms of the musical aspect, I have the benefit of being the musician, so I know exactly where the snare is without any sort of computational method involved. I guess that’s a major advantage.

  12. A follow-up piece with clever tricks is certainly useful, but I think such an article is more on Bent’s turf than mine. I do have some tricks to contribute, though ;)

  13. avatar
    Aug 23 2011

    I the dead project I work on a decade ago, we used a non-registered effect letter in FastTracker 2 to sync music on effect. The use of that effect (“Z” or something like that) trigger a specific effect.

    It was really nice to use :D

  14. avatar
    Aug 23 2011

    sounds familiar rez, I can’t remember exactly either but I’ve dabbled with that too :)

  15. avatar
    Aug 23 2011

    Rez, in fact those non-registered FT2 effects were what inspired this idea in the first place. It’s basically lots of Zxx columns with real-time demo control, interpolation, and no reloading the tune and demo at each change.

    Cool writeup, Bent!

  16. Gloom: I find it a bit ironic that you chose to use Scyphozoa for the screenshot; we didn’t use GNU Rocket for Scyphozoa, we used Skrebbel’s Synctracker2 ;)

  17. avatar
    Aug 25 2011

    Kusma: that is true, but really – that’s just splitting hairs. Also: it was the one high-res screenshot I had on my Mac at the time ;)

  18. avatar
    Aug 26 2011

    First i must congratulate with the launch of this site. I have been looking/waiting for such a site for some time. Hope it lives long and prospers.

    Interesting article on GNU Rocket. I’ve never heard about this utility before now (well i did see it mentioned in one of the contributions to the Assembly 2011 demo compo).

    I worked on some scene-related projects in the 90’s, but as a graphican with below average programming knowledge i was mostly limited to deliver a picture, font, color palette and (later) 3d models. I did on a few occasions get to do synching and timing and some movement direction on effects, but it required me to travel by train (and maybe boat) to get to the coder, and then have him adjust things manually in asm. Not very effective workflow. Alot has happened since then when it comes to demomaking, and i’m sort of at square one now, but i’m motivated to get creative again.

  19. avatar
    Ebbe Berge Smith
    Aug 26 2011

    @andre: make a post in the forum with a “portfolio” if you can’t force someone you know to code for you. Alot of groups are in desperate need of proper graphicians* , fyi :)

  20. avatar
    Aug 26 2011

    I hear that.

  21. avatar
    Aug 26 2011

    @andre: I second Ebbe. Graphicians are sorely needed, so show people what you can do, and let the coders walk over fire to woo you :) I, for one, happen to know of a certain Norwegian demo group who is planning a demo for this fall. ;)

  22. avatar
    Oct 25 2011

    Very exciting stuff! I’ve been working on a similar system for creating concert visuals – http://meds.mmmlabs.com – I’ll download GNU Rocket and check it out, maybe there’s a way we can help each other…

  23. avatar
    Oct 28 2011

    momodmonster: Cool approach – I like it. :) You’ll find that while GNU Rocket _can_ be used to simply sync to specific events, the real magic happens (IMHO) when you start syncing stuff like scene-changes or subtle effects as well — things that have no natural “trigger” in OSC-data, but can be felt.

    One example (here goes one of my secret tricks :) I can give is with bass. If you stand in front of a big PA-system and some heavy bass plays, your natural instinct will be to close your eyes, or at least, that they contract a bit. In the past, I’ve played with this by syncing a slight black screenfade to places in the music (or ambience) where there are a lot of low frequencies. The effect is almost never noticable, but it really helps to build the effect you want.

Trackbacks & Pingbacks

  1. Making the Machine – part 1 | Displayhack

Leave a Comment