Having gone through the onerous task of selecting what type of laptop I wanted at Red Badger (13″ MacBook Pro with 16gb RAM and 512gb SSD) and waiting impatiently for it to be delivered, it was time to get it ready to use.

My previous personal Mac, a late 2010 11″ Air, has done sterling service – even just about managing when asked to run a couple of virtual machines concurrently alongside a Node.js environment. But it was starting to creak a bit and the poor thing’s SSD was chock full of all the random crap that had been installed to support various projects. So as well as the stuff I regularly use there were also leftover bits and pieces of Java, Python and who knows what else.

At Red Badger we’ve largely adopted Vagrant and Chef as a great way of provisioning development environments without imposing on the personal space of your own laptop. Clone a repo from Github, run ‘vagrant up’ on the project directory, the Vagrant script installs and configures a VM for you, ‘vagrant ssh’ and there you are with a fresh, clean dev environment that can be reconfigured, reloaded and chucked away as often as you like. Joe is going to be writing more about this process soon.

So I decided, I would challenge myself to install the bare minimum of tooling to get up and running, with all the stuff specific to our current project residing in a virtual machine. This should also, in theory, make a nice solid base for general front-end development to current practices. And hopefully leave 400gb -odd of disc space to fill up with bad music and animated cat gifs.

The clock was ticking…

1. Google Chrome – goes without saying really (3 mins)
2. Node.js – nice and easy these days with a pre-packaged installer available for Mac, Windows and Linux (5 mins)
3. Useful Node global modules – First trip into the terminal to ‘sudo npm install -g …’ for coffee-script, grunt, express, less, nodemon and jamjs. This is by no means an exhaustive list but gives me all the Node-based essentials to build and run projects. (5 mins)
4. Xcode Command Line Tools – Rather than the enormous heft of all of Xcode, the command line tools are a relatively slimline 122mb and give you the bits you need (as a web rather than OS developer) to compile the source code of stuff like wget. An Apple developer account is needed for this so I can’t give a direct link, unfortunately. (15 mins)
5. Git – Last time I did this I’m sure there was a fair amount of messing around but there is now an installer package for OS X (5 mins)
6. Virtual Box – This is where we start getting in to the real time saving stuff. Virtual Box is a free VM platform which can be scripted, essential for the next part (10 mins)
7. Vagrant – This is the key part, as explained in the introduction. Once Vagrant is up and running, a pre-configured dev environment can be there in just a few minutes (3 mins)
8. Github ssh key config – Back to the command line. To be able to clone our project from its Github repo I had to remember how to do this, and the documentation on the Github site doesn’t have the best signposting. (5 mins)
9. Clone repo and post-install tasks – Get the code down, then run ‘npm install’ and ‘jam install’ in the project directory to fetch server and client dependencies. Run ‘grunt && grunt watch’ to build CoffeeScript, Less and Jade source into application code and then watch for file changes. (2 mins)
10. Vagrant up – With the repo cloned and ready, the ‘vagrant up’ command downloads the specified VM image and then boots it and installs the relevant software. In this case, Node and MongoDB. At a touch over 300mb to download (albeit a one-off if you use the same image on subsequent projects) this is the longest part of the whole process (20 mins).
11. Vagrant ssh – Log into the new virtual machine, change to the /vagrant directory which mounts all your locally held app code, run ‘node app/app.js’. Hit Vagrant’s IP address in Chrome (see step 1) and you’re there (1 min).

So there we go. By my reckoning 74 mins from logging in to the laptop to being able to get back to productivity. And because the VM environment is identical to where I was on my old Mac I could literally carry on from my last Git commit. I did those steps above sequentially to time them, but at the same time was installing the Dropbox client for project assets and Sublime Text for something to code in.

Obviously on a new project you would have the added overhead of configuring and testing your Vagrant scripts, but this only has to be done once and then when your team scales everyone can get to parity in a few minutes rather than the hours it might once have taken.

Vagrant really is one of those tools you wish you’d known about (or that had existed) a long time ago and really feels like a step change for quick, painless and agile development. This is also the first project I’ve really used Grunt on, and its role in automation and making your coding life easier has already been widely celebrated. Hence the title of this post, because this feels like the first step in a general refresh of the way I do things and hopefully I’ll be writing more about this journey soon.

Q: What do our 3 favorite open source projects (node, redis and nginx) have in common?  Apart from being uber-cool?

A: They are all single threaded.  

But aren’t they all really fast and highly scalable?  Yep.  So how does that work?

Nginx, redis and node are all event-based.  They have an event loop that will listen for an event saying that an asynchronous operation (IO) has completed and then execute the callback that was registered when the async operation started.  Rinse, then repeat.  It never waits for anything, which means that the single thread can go hell-for-leather just running code.  Which makes it really fast.

In days gone by, when we were Microsoft slaves, we used to wrestle with multithreading as a way of dividing up work.  In web apps every request started a new thread.  We’d also use the Task Parallel Library (TPL) which was not an easy abstraction.  And combine that with some event processing library like Reactive Extensions (Rx).  Now you’re asking for a lot of trouble.  The new await keyword in C# helps out alot, but either way you have to think about thread safety all the time.  And all kinds of locking strategies to deal with concurrent access to the same data.  And even with all that, it isn’t as fast.

The difference between the two worlds lies in the way that pieces of work are orchestrated.

Event-based applications divide work up using callbacks, an event loop and a queue.  The unit of work, or task, is a callback.  Simple.  Only one callback is ever executing at a time.  There are no locking issues.  You can write code like you’re the only kid on the block.  You decide when you’re done and then effectively yield control to someone else.  Everyone is really polite so it just works.

Thread-based applications essentially divide work up in hardware.  Because each piece of work has its own thread, and will block if it needs to (like when it’s waiting for IO), the CPU will suspend that thread and start running another that is waiting.  Every time that happens there is quite a hefty context  switch, including moving about 2MB of data around.  In effect the hardware decides when to yield control and you don’t get much of a say.

Who’d have thought that a single thread, dealing with everything, could be faster than multiple threads each dealing with just one thing?  Well, on a single core, that may be true.  On multiple cores it actually may also be true.  That’s because you’ve probably got nginx and node and redis all running on the same machine – simplistically, on a quad core, that’s one core each and still one left over

But isn’t writing synchronous code for a multithreaded environment a lot easier than writing asynchronous code for a single threaded environment?  Well, maybe, a little.  But some great patterns have emerged within the node community that really help.  

The simplest continuation-passing style (CPS) is the callback.  Which actually is not at all hard when you get used to it.  And it happens to be a great way to encapsulate and really easy to modularise.  The pattern for async functions is that the last argument is always the callback, and the pattern for callbacks is that errors are always the first argument (with results after that).  This standardisation makes composition really easy.

There are a ton of npm modules that can often help reduce complexity.  The best, in my opinion, is still Caolan’s async.  It’s still the most popular and follows the node conventions.  And there are also a few CPS compilers that allow you to code in a more synchronous style.  I wouldn’t have recommended these in the past, but there are a few, such as tamejs and Iced CoffeeScript, that use an “await, defer” pattern that is quite nice.  We’re using CoffeeScript more and more these days, and this “icing” is very tempting (seeing as we’re compiling anyway), but we haven’t strayed that way yet.

We’ve been writing big apps in node since October 2011 and have learnt a lot about how to separate concerns and modularise our code.  It’s a lot different to the object-oriented class-based separation we were used to, but after your head is reprogrammed to use a functional style it becomes second nature and actually much easier to structure.  Caolan’s post on programming style for node sums it up nicely.  If you hear anyone say that node is no good for big projects, tell them that all you have to do is follow a few simple rules and then it becomes perfect.  And fast.

Over the past couple of months, a small team at Red Badger has been working on a number of HTML5 prototypes for an interesting client. Speed of development and easy iteration have been essential so we’ve taken the opportunity to try out a new technology stack which has given what we were looking for and is exciting the whole business.

Maybe a demanding prototype schedule isn’t the ideal place to chuck away everything you’re used to and start afresh, but actually a lot of the front-end development has built on tools and themes we’ve worked with throughout 2011 and we’ve found that the speed and ease of using Node has more than compensated for the learning curve. So, what have we been using?

Server

Node – Underpinning everything we’ve been doing in our prototyping project; Node is fast, event-driven and built on Javascript. Its been fascinating for myself, as a primarily front end developer, and Stuart with an ASP.net background to see how our respective specialisms are converging on a single set of tooling. The module loading system, NPM, which is similar to Ruby’s Gem ecosystem, also makes it incredibly easy to pick up and play with the many extensions that are out there – and to create your own too.

Express – A development framework for Node, giving RESTful routing and content negotiation. After working with Open Rasta in .net MVC on a previous project, the ease of setting up applications using Express has been a delight.

Jade – With Express providing the application routing and view rendering, we next add Jade for templating. A HAML-style syntax, offering us simplicity and brevity but compiling to really well-formed HTML and easy to use with HTML5 data-* attributes for Knockout (which we’ll come to later). In fact having used Jade for a while now I’m not sure I want to go back to writing “proper” HTML.

Step / Async – Node’s asychronicity takes some getting used to after having worked with linear control flows for such a long time. Imagine AJAX callbacks as the fundamental way a language works – you can’t rely on other parts of your application providing data at the moment you need them, so you need to be able to create queues for parallel and serial execution. Step and Async are two modules we’ve tried out for this, both have their benefits but Async seems to be slightly in the lead for what we’re doing.

Now – The other great benefit of Node is its ability to serve real-time applications, and the Now module takes this to almost magical levels. Existing as a namespace on both the client and server simultaneously, you can call client methods from the server (and vice-versa) to push data instantly as a general or targetted broadcast. Seeing this in action has really convinced us that Node is something to get excited about, and has wowed the client too!

Client

Knockout – We used Knockout on other projects throughout 2011, but with Node’s inherent ability to supply real-time data its really coming to its own. It’s a Javascript library implementing the MVVM (Model-View-View Model) pattern, which should be familiar to Silverlight developers, and makes rich UIs a breeze to update. The new 2.0 version, released during the Christmas break, removes the reliance on jQuery for its default templating which really opens up its flexibility.

Underscore – Described a ‘utility-belt library’ for Javascript, Underscore is another tool that is compimentary to the likes of jQuery and adds a whole raft of functional programming methods to objects and arrays (among other things). It also runs as a Node module so we can make use of it on both client and server, great for code consistency.

Ender – In fact with Knockout and Underscore at work, it was beginning to feel like jQuery had been relegated to just a “ready” utility, DOM selection and effects. That’s a lot of weight for something we weren’t using very much so as an alternative we’re trying out Ender, which allows you to compile your own library from smaller modules – such as the lightweight DOM selector Qwery. And it all installs and builds in a similar approach to NPM.

LESS – A CSS pre-compiler, and another tool we used during 2011, but as a native Node module its integration is now much easier. If you’re developing in a Node environment you can use it to watch for LESS file changes and compile locally (we also use LESS.app on OS X), and then deploy the LESS and have the server startup create a complled and minifed version in production. Alongside Jade and Coffeescript its beginning to feel like compilation from more efficient syntaxes down to browser-readable files is becoming a key element to web development.

The whole picture

As well as this Javascript-oriented development we’ve also been trying out MongoDB and Redis for data storage as part of the stack, with equally encouraging results. And to make project compatibilty and pair programming between our Windows, Mac and Ubuntu users easier we’ve given JetBrains’ Webstorm (and PHPStorm) IDE a thorough test drive – given it has all that familiar Reshaper goodness from their Visual Studio tools its looking like a great combination so far.

It might seem that, with all these Javascript-heavy technologies in HTML5 documents, older browers won’t get a look in. As we’re prototyping on this project perhaps its not important, but actually support is pretty good. Every view Node creates is sent as rendered HTML, just like any other web server, and in fact due to its speed we’re finding that we can make sites less “AJAX-y” than we might otherwise – which of course is better overall for accessibility and discoverabilty. On the client side, Knockout is compatible back down to IE6 and even the ‘magic’ of Now is mobile compatible, with beta support for older IE versions. Of course we want to move away from those legacy browsers as much as any other developer, but if it’s a client requirement this stack can still provide it.

So as a result of these experiences we’re investigating using Node and its related technologies more widely at Red Badger during 2012, its already looking like its going to be an exciting year!