How to Render the Mandelbrot Set in the Joyent Cloud With Node.js

First of all: Apologies for not posting for a long time. The reason? I was having too much fun with node.js and the Joyent Cloud :).

What started as a small experimental hack turned quickly into an exciting new pet project involving the good old Mandelbrot Set, as a web service, running in the Joyent cloud, programmed in node.js.

But first things first: Let’s take a look at node.js as a language and programming model, at the Joyent Cloud and how it relates to Solaris and finally some details on how the picture you see was rendered inside the Joyent Cloud, including an interactive Mandelbrot Set explorer you can play with now, written as a web app.

I think the first time I heard about node.js was from Bryan Cantrill in this post where he explained the story about Joyent and a node.js programming contest and how they added DTrace probes to Node to enable some analytics for their contestants’ machines.

What Is Node.js?

node.js (or short: Node) is both a programming language and a runtime system. The programming language is V8 JavaScript with a few extensions, the more important part is the runtime system and the programming model: Node is server-side, event-driven and non-blocking. Which makes it ideal for writing highly concurrent, scalable web applications, without caring about processes, locks, threads, etc.

Check out the nodejs.org homepage for details and please do spend an entertaining hour watching Ryan’s introduction to node.js. Don’t worry, I’ll wait.

Node is lower level than most web application frameworks: HTTP and sockets are first-class citizens in Node, and “Hello world” in Node is actually a web server. Other frameworks like Java, PHP etc. typically sit on top of an existing web server (though it is possible, but not common, to implement web servers in these languages as well). Node’s intimacy with networking protocols make it interesting from a minimalistic point of view: Develop the web server with the web app in sync, and only implement the pieces that are really necessary, leaving out the rest.

You can also implement pure socket based protocols with node.js, which makes Node interesting to game developers, for chat servers, messaging in general, etc.

There’s a rapidly growing library of Node extensions called Modules, including a package manager called npm that makes installation of modules very easy. Most low level web server programming needs can be addressed easily with the connect framework, higher level web apps will enjoy working with the Express framework.

Getting started with node.js is easy: Download the software, build it using the easy to follow instructions and start playing with it. The node.js wiki is a great resource to start digging in, and Mozilla.org’s JavaScript Reference will help you polish up your JavaScript skills in a breeze.

Why Node is Cool

Here’s why I think node.js is so interesting:

• It’s event driven. This means that a node.js program is not just a sequence of stuff to do. It gives you the ability to attach pieces of code to events: Someone connects to your server? Fire up this code. A task has finished? Go run this other code. Got a new packet coming in? Go do these three things with it. Timer’s up? Here’s what to do. And so on. Just like JavaScript in a browser is based on browser events (you know: onload, onclick etc.), node.js is based on server events.

• This allows for symmetric web app development: If you have some JavaScript skills that you developed by writing AJAX apps on the client, you’ll be familiar with node.js on the server very easily. If you want to develop a new web app, you can now do both the server and the browser sides at once, with the same set of skills, libraries, models, etc. Millions of JavaScript hackers are becoming server side hackers overnight, thanks to Node.

• node.js removes the complexity of dealing with concurrency, locking etc. by ignoring these concepts: If everything is driven by little pieces of code that run independently, triggered by events, there’s a complete disconnect between them. This is a good thing: No need to watch out for the other guys, no need to do any locking or other complicated stuff. Just write small pieces of code that do their job little by little and let the database, the OS or the node.js runtime deal with the messy concurrency stuff. Eventually, there will be a way to fire off parallel threads, but it will be batch-oriented and message based, through the web workers API.

• It’s easy. I managed to put together a Mandelbrot Set rendering web service in about an hour. From scratch. And I’m not a real programmer.

• It’s new. There’s a lot of innovation around Node. It’s a completely new world, which makes it fun and exciting, too.

Joyent and node.js

As you may know, Joyent (no link, page no longer exists) is a startup company that delivers cloud software and cloud hosting services. Joyent is probably the most innovative company in terms of cloud computing right now, and they’re big supporters of Solaris. In fact, their own “SmartOS” operating system for the cloud is based on OpenSolaris. Cool people including sysadmin god Ben Rockwood, DTrace gods Bryan Cantrill and Brendan Gregg (Buy his DTrace book now*, it’s really good!), and of course node.js inventor Ryan Dahl work there.

IMHO, Joyent is where the future of Clouds is made.

Anyway, the Joyent Software Stack (no link, page no longer exists) uses node.js at the top to deliver a powerful, asynchronous, open source PaaS platform.

Getting Started with Your Own Joyent SmartMachine for Node.js for Free

Of course, you can play with node.js on your laptop, but it’s more fun when you use it inside a real cloud. Thankfully, this is really easy (and free) with Joyent’s SmartMachine for node.js service:

• Go to https://nodejs.org/ (yay, a German domain!) and sign up for a Joyent account there.

• Read the Getting Started With a Node SmartMachine (no link, page no longer exists) guide, including how to apply for a provisioning code so you can provision your first Joyent SmartMachine.

• Upload your SSH key, set up your Git repository and off you go!

The beauty of a Joyent SmartMachine is that it’s actually a Solaris Container: You get all the flexibility and power of being king on a Solaris system, with all of the security, reliability and efficiency of a Solaris system, but without the hassle or without annoying your other buddies running on the same physical machine.

The node.js service running inside a SmartMachine is nicely integrated with Solaris: It’s configured as an SMF service and the pre-supplied commands node-service-start, node-service-stop, node-service-log etc. make great use of it. Even better: As soon as you push your Git repository into your Joyent SmartMachine, the node.js service is automatically restarted through SMF to immediately update your service. And if anything fails, it will automatically revert back to the previous version. How cool is that?

But the real killer here is the integration of node.js, DTrace and the Joyent Cloud Analytics (no link, page no longer exists) feature. As your node.js app gathers more and more users, you can use Cloud Analytics to diagnose, monitor and better understand the dynamics of your service. Watch the video with Bryan Cantrill and Brendan Gregg on Cloud Analytics (no link, joyeur.com no longer exists) to get an overview of how it works.

Joyent gives you a simple package manager with a large amount of pre-configured packages called pkgin (no link, page no longer exists). npm, the Node Package Manager is pre-installed, but some extra packages are recommended for a smoother install of some npm modules.

Introducing the Constant Thinking Mandelbrot Rendering Web Service

So I started playing with Node in a Joyent SmartMachine. And it was fun! After a few weeks of trying stuff out, optimizing and tinkering I can now present you with my first real web app, the Constant Thinking Mandelbrot Set Web Service (no link, this page is no longer available.)!

Most of you already know that the Mandelbrot set is a beautiful mathematical object that can be generated using a simple formula over and over again over the coordinates of a complex number plane. I think I wrote my first Mandelbrot set program at the age of 12 or so, when these things became fashionable in the home computer crowd.

Now here’s how Mandelbrot 2.0 works:

• On this page (no link, this page is no longer available.), click anywhere within the image to set a new center for the image computation.

• Click on “Zoom” to zoom into the set, and on “Zoom out” to back out. The math says there’s infinite detail hidden in the set. The limits of floating-point calculations will put an end to this but as soon as I start seeing precision errors, I’ll work on a more precise implementation.

• The amount of detail is limited by the maximum number of iterations. If after a few rounds of zooming into the image the black areas start to look boring, hit the “more details” button and you should see more cool stuff. This simply doubles the maximum number of iterations for the computation.

• There’s some anti-aliasing, too: If you select one of the anti-aliasing options, the image will be generated at a larger resolution (3x3 or 5x5), then shrinked into the target resolution using one of two filtering methods (simple Gaussian or the more sophisticated Mitchell-Netravali filter). In theory, this should make the image smoother. In practice, the infinite detail of the set counters that a bit. Still worth a try.

• Finally, there’s some timing displayed and you have a choice of different optimization algorithms. The fastest algorithm is chosen by default, but if stuff looks funky, send me the parameters that were displayed and how you got there, and go with one of the lower optimization levels to get a decent picture.

Feel free to play with the Mandelbrot service (no link, this page is no longer available.) and place a comment below. Or send me a message if you find a bug or have any other suggestions.

Experiences, Gotchas and Further Plans

My first impression of node.js: Boy is this fast! Rendering a 512x512 pixel Mandelbrot set picture takes less than a second, including color mapping and generating the PNG image from the raw pixel data, even without any optimization. This is possible because under the covers, node.js uses Google’s V8 JavaScript engine which has some pretty advanced optimization techniques.

Like any new platform, it takes some tinkering until you figure out things and find your ways around it, but thanks to the Node Wiki, the Mozilla JavaScript Reference and the Joyent Node Wiki (no link, page no longer exists), most answers are just a few clicks away.

Since Node is so new, there are a couple of minor annoyances: There’s no easy way to do parallelism (it’s not encouraged yet, until the web workers API is built in), the amount of available modules is rapidly changing or incomplete, and sometimes the best documentation is to read some example source code. All of these are just symptoms of a good thing: A new, innovative and rapidly evolving world.

Bonus: Source Code and Some Mandelbrot Set Geekiness

Most of the time I spent writing this web service was actually spent on optimizing the algorithm for computing the Mandelbrot set. For the standard image you see in the beginning of the Mandelbrot web service page, the speedup is more than 2x, so it was worth the effort. Here’s a crash course in Mandelbrot set optimization:

• The two main big bulbs that the Mandelbrot set is made of are actually well understood (see the Wikipedia article on the Mandelbrot set). This means one can use a simple formula to detemine whether a point is inside one of the two bulbs, then decide whether to skip or continue the standard, lengthy iteration formula.

• Exploiting connectedness: The Mandelbrot set is connected. This means that there are no “islands”. This also means that if all points of the circumference of any area are inside the set (= black), the whole area is. To exploit this, an algorithm can subdivide the set into rectangular blocks, walk the blocks’ edges and try to find blocks with circumferences inside the set through binary (or any other convenient) subdivision. For such blocks, further iterations can be skipped and the whole block can be marked as inside the set.

• Major areas of the Mandelbrot set are well-known: This means that we don’t need to test for bulbs if we’re rendering the piece that’s above them, so we can skip the check. We can also skip the check for bulb 1 if we’re left or right of it, etc. This optimization therefore computes the intersection of different areas with predefined optimization settings and the areas that are actually to be rendered, then uses the right optimizations for the right pieces of the image.

• Exploiting symmetry: The Mandelbrot set is symmetrical along the real axis. So if the area to be rendered crosses that axis, up to half of the image can be recycled through mirroring.

The highest optimization setting (which is also the default) makes use of all of the above and should be pretty good for most cases. If you have any more cool and easy to implement optimization ideas, let me know!

That said, the only thing left for me to do in this article is to give you the source :). Like any good Node citizen, I’ve set up an account on Github and uploaded the current version of the source:

Check out the Mandelbrot set generator in node.js source here.

Of course the usual disclaimers apply: Use at your own risk, don’t sue me, forgive any code uglinesses, let me know if you find bugs, send me your suggestions and generally have fun with it.

Conclusion and Further Plans

Many thanks to the fine people at Joyent for bringing node.js to the world and giving it such a cool place to live! Thanks to their great support people who helped me find the magic incantation for getting a provisioning code. I’m looking forward to seeing Node grow and continue playing with it :).

Actually, the fun has just started: Right now, the Mandelbrot set rendering web service ist pretty basic, and it doesn’t use many of the cool Node features. Over time, I’d like to add some more:

• Caching should be easy to do in Node, and I already have an idea on how to exploit it.

• Parallelism is the next big thing: Put together a stack of batch jobs that make up the total rendering process, then fire them at some worker nodes, then collect the pieces together. Should be fun to implement while learning something new.

• Integration with C: While Node is pretty fast already, it would be fun to implement the basic number-crunching in C so I can learn how the integration of C works with Node.

• While we’re at it, I’d like to increase rendering precision through implementing the computations in a purely integer manner. You know, floating-point is evil.

• The user interface on the web browser side could use some brushups, perhaps with some jQuery magic.

• Colormaps should be configurable.

• And so on. You know, some projects are never finished…

Your Turn

Did you try out Node already? What are your experiences? Any cool libraries, hacks, tips to share? Do you like the Mandelbrot set web service, or do you think this is too 80’s? Let me know in the comments!

• Affiliate link: You buy great stuff that’s good for you, I get a small commission to help finance this blog, we both win!