August 19, 2011

Node Code: Quartz Composer Overview (Part 1 Of Infinity)

by George Toledo


This is going to be an ongoing column, concerned mostly with execution graph technology and Quartz Composer. I originally was going to write a piece about Quartz Composer, and attempt to cover basics and some uber-cool projects done with it. The reality is that the subject is way too broad to cover in one article.

Getting started

Since I’ve become involved with Quartz Composer (also known as “QC”), I’ve read countless posts to Apple’s Quartz Composer Developer list or Kineme where users complain about a lack of documentation. I don’t think that criticism is entirely valid; I think it’s a matter of history, and this article is going to go into that a bit. What follows will be a bit of broad overview of some issues surrounding Quartz Composer, while future articles will be more in depth and certainly have a great amount of info that one can use in many domains, not just Quartz Composer, though there will be a focus on implementing in QC.

Back to the common complaint of lack of documentation; a cursory browsing of the internet will reveal countless pages that attempt to explain Quartz Composer, they just don’t explain the problem the user wants to solve. Users often disregard Apple’s documentation as a starting point as well. I suggest Apple Quartz Composer Programming Guide as a very good piece of reference material.

Apple has done a great deal of documenting of Quartz Composer, provided tons of example projects that show how to work with Quartz Composer graphs (qtz files), how to write plugins for Quartz Composer, or integrate it into applications that use the technology. Doing a search through Apple’s Developer code will reveal numerous examples and articles, all of which I would recommend that anyone interested in Quartz Composer examine. I think these are the absolute best examples and documentation of Quartz Composer in existence.

The documentation workaround

That said, there is a very major reason why people tend to immediately ignore Apple’s documentation, and it’s somewhat valid.

With Apple’s standard API, one can indeed expand the modules that Quartz Composer loads into it’s environment by writing plugins that one can construct graphs with, but it also places numerous embargoes on that QCPlugin API. There is a large subculture – or perhaps the dominant culture – of QC users that utilize what has become known as “the SkankySDK” to circumvent this. For example, Apple may prevent plugins from loading inside of certain environments like the Web Browser, or from utilizing built in QC data types like “QCMesh” (because of proprietary reasons or simply a lack of updating the spec. Speculation is probably unhealthy!)

Before Apple had actually published a plugin API, users were eager to extend Quartz Composer’s functionality. QC was based on Pixelshox, which was also an extendible technology, and number of users had noted that Apple had also designed Quartz Composer to load “GFPlugins”. By backwards engineering this API, this soon became a way for users to extend Quartz Composer’s functionality. By the time that Apple had actually introduced a plugin API, several developers were already using the “skankySDK”, and numerous plugins that conformed to this standard “non-standard” had been released. Interestingly, Apple continues to create GFPlugins for use in their own projects, choosing not to eat their own dog food. The infamous tape where Apple calls the reverse engineered SDK the “skanky” SDK while introducing their hobbled SDK is entertaining enough though.

What was most problematic is that through use of GFPlugin/skankySDK one could create new data types inside of Quartz Composer, and leverage control over the entire system. QCPlugin uses a wrapper that precludes one not only creating new data types, but also creating certain types of macros, or even utilizing certain built in data types.

While it is far from useless, and one can create a great deal of things in that box, the fact that QCPlugin is limited, but accepted by Apple, has a significant impact on Quartz Composer development and culture. If one wishes to create new modules that are allowed in apps to be sold on Apple’s app store, one must write plugins in that box. If you wish to write modules in a way that leverages complete control, you use the skankySDK, fully knowing that this isn’t viewed as an awesome thing by Apple, and also, that they do not support.

With a little help from your friends

Over the years, Kineme codified the non-standard plugin API/skankySDK into an actual framework, which is available at their site. Kineme’s forum is also the largest community of Quartz Composer users, and a great place to get feedback and help, as well as an excellent source of add on modules. Another very noteworthy developer in the QC community is Anton Marini, a.ka. Vade. His v002 plugins are considered to be some of the most useful plugins available for Quartz Composer, and he also shows how far boundaries can be pushed even when using Apple’s standard QCPlugin API (and he also has a forum for support).

It may seem quirky to lay all of that out on the front end, before even getting too far into “what Quartz Composer is”, but I never said I wasn’t quirky. I think it’s important to understand this when delving into Quartz Composer, and good to understand it before getting deep into interacting and programming with the system.

The inner workings of QC

Quartz Composer is multifaceted. It presents itself in the form of a framework for OS X, as a technology, and as an application to leverage the technology. A good way to think of Quartz Composer is that one is creating a file, and that by linking together modules (patches) with cables (noodles), one can send information from one module to another. Anyone that is familiar with node based editing systems may also think of these as nodes, and of Quartz Composer as a node graph, and that’s an accurate description. Others may think of this as a decision graph, or an execution graph. There are strong parallels with non-programmer paradigms, especially signal flow. Musos that have chained together multiple effects boxes, or modular synths have a good running start to understanding the method of interaction with QC. Users typically work with the Quartz Composer.app, which is included with Apple’s Developer Tools, to create and manipulate Quartz Composer graphs.

In Quartz Composer, a method is used that is called “lazy evaluation”. QC graphs tend to lay out from right to left on the Quartz Composer editor application. When one connects the output of a QC Video Patch, to the image input of a QC Sphere, execution happens thusly: The QC Sphere is a “Consumer” patch. First, a check happens to see if it is enabled. Then, all input port values are surveyed.

If there is a value at the port, it’s used. If there is a connection to another patch upstream, like the Video Patch, that patch then executes. If the Video Patch output was connected to nothing, it would never execute at all. If there was a switcher in between (known as a multiplexer in QC parlance), and connections existed, but the Video Patch was not currently chosen, the Video patch would still not execute. If that Video Patch has an input port, connected to another patch’s output, then it will trigger the next patch in the line to execute. This is likely an important distinction to make to people familiar with graph environments that use a Push model (like Max or VVVV).

The fact that one can program plugins, whether skanky or standard API, that there is a defined evaluation mechanism, that one can incorporate any of Apple’s frameworks fairly easily, use past work modularly, or write GLSL shaders, OpenCL kernels, JavaScript programs in the Quartz Composer editor while live compiling and seeing the output results render in the QC Viewer while working makes QC a tool for some projects that is indispensable.

I’m excited to delve into that more. In coming installments, we will learn about creating graphs, prototyping shaders and CL kernels, extending functionality with the standard and skanky SDK’s, integrating Quartz Composer into standalone apps, as well as broader issues in coding and how they interplay with this technology.

 

avatar
About the author, George Toledo

Life's been good to me so far.

7 Comments Post a comment
  1. avatar
    Aug 19 2011

    I’m looking forward to following this series George :)

  2. avatar
    cyberointernet
    Aug 24 2011

    A good opening article. looking forward to reading your further posts :-)

  3. avatar
    jonathan.muth
    Sep 8 2011

    Very good article, indeed. I just got started with Quartz Composer and it is this kind of writing that encourages me to dig deeper. :)

    Looking forward to the posts to come as well. :)

  4. avatar
    leon-geyer
    Oct 16 2011

    Great! I would also be one of the people who complains about lack of documentation: for users who are new in programming and for whom the detailed instructions in the guides aren’t near enough to practice – so we have no idea how to visualize what we read or search what we think, and we build info out of chunks. There’s like a gap between rude basics and deep expert, or a overviwe as starting point. Great you write this, looking forward. Thank you!

  5. avatar
    George Toledo
    Oct 22 2011

    Leon, what are you trying to use Quartz Composer for, for example? Would you say you’re new to coding, in general, and trying to use QC as an entry point?

  6. avatar
    George Toledo
    Nov 4 2011

    As a little note, I’ve added many Quartz Composition files, some plugins, and some patches, to the Dropbox widget @ http://www.georgetoledo.com . The next article in the series will be coming out soon, and after that one (that lays a little more groundwork), we’re going to dive deep into programming original patches.

  7. avatar
    leon-geyer
    Jan 8 2012

    Sorry, didn’t read this a while. I was absolutely outside coding, never thought of it, until I found QC. I worked with video. I am still a total newbie to coding, but since then I started to work visuals through generation. So I am not using QC as an entry point to coding, but coding as a media to visuals – in other words, my interest and fascination is not the coding itself (like its internal logic, which I know can be fascinating to many and a reason by itself), but the visual world to produce. Well, it can’t be speared, of course, but I hope you understand the point – my mind is visual. That fore, your help this way to understand the instrument QC is for me so productive.

Leave a Comment