Lessons learnt from my first Node.js project

Over the last couple of months my wife and I have been working on a little web game called findthatrobot.com. It’s a small, building it for the fun of it project, and gave me a chance to stretch my newly acquired Node.js muscle.

I’ve played around with some small Node.js projects over the last year or so, mostly following tutorials and the such, and this was my first complete project. I’m going to share with you what worked for me and what didn’t, in hopes that anyone else out there might learn something to make their lives easier, or perhaps be inspired to try Node.js for the first time.

Getting Started and Development Environment

The first thing I needed to get set up was the development environment. We’d already decided on using a free Amazon EC2 Micro instance as our hosting, and I’d played around with running Cloud9 IDE locally on a Virtual Box running Ubuntu, so setting up Cloud9 IDE on EC2 seemed like the most sensible first step forward.  For those who don’t know, Cloud9 IDE is a browser based IDE, which might seem strange at first, though it doesn’t take long to see what a powerful tool it is, and how much of a pleasure it is to use. I really can’t recommend Cloud9 IDE enough.

In the end I set up two instances listening on different ports, since the Cloud9IDE in NPM is a limit version of what’s on c9.io, and restricted it to one user at a time. I used GIT to sync the changes between the two, and to deploy to the production site and backup locally.

Frameworks and Middleware

When I started, I didn’t have much of an idea of what the final structure of the app would be. I’ve had a fair bit of experience in building MVC web apps, so I figured that’d be a good place to start.

After trying a couple of different frameworks and doing a bit of research on what was popular, I discovered Express, which takes Connect, a middleware framework, and expands it into a reasonably complete web app framework. I come from an ASP.net background, so the word “middleware” didn’t mean a lot to me a first. Simply put, a middleware component is a function that acts on request and / or response. Connect “connects” these functions in a pipeline. An example middleware component might be parsing a file in the request, or compressing a response.

It took me a while to pick the bits and pieces of middleware I needed, though most of what I wanted came built in to Express or Connect. I used a couple of components built by others, and built a few myself. I found it really easy to create my own, perhaps too easy, and got carried away re-inventing the wheel a couple of times, though it was all good learning.

Here’s the additional bits of middleware I used:

• A custom logger for errors and request logging. I loved the inbuilt formatting options in Connect.logger, but I wanted to output the data into a rolling log. I found Winston was good for the job, as well as giving me more logging capabilities elsewhere. Winston wasn’t connect ready to begin with, though it didn’t take much coaxing to make the Connect.logger use it as its output.
• I had quite a bit of trouble finding a gzip component. I started with connect-gzip but the static encoding didn’t add an expires header. I switched to gzippo which did, but didn’t have compression for dynamic files. So I ended up using both. I’m not entirely sure of how well either component performs, but they both seemed to work well enough for my purposes.
• I added connect-form to deal with uploading files in the admin screens.
• I wrote some custom tracing code that utilized Winston’s tracing capabilities and the inbuilt connect profiler. It also added custom timing logic to to each middleware component. After writing all this profiling logic, the only thing I discovered was that everything was running well.

Coding Style and Standards

There’s a couple of practices that I picked up early on, most of which I learnt from Stella Laurenzo’s post on Bulletproof Node.js Coding.

The habit of returning on the last statement is a technique I think every Node.js coder should use. The code becomes much easier to follow since callbacks often obscure the end of the function, and it stops you accidentally executing code after a call to an async function.

Using named function declarations instead of inline functions vastly improves maintainability. It makes following the flow of async callbacks much more straight forward to read, and makes the code a little more self-documenting with the callbacks given names.

Data access was something I struggled with and I think, given time, I would have rewritten much of my current solution. I used redis as my sole data store because I was more concerned with performance than data reliability. This meant that the act of reading and writing data was simply serializing and deserializing JSON strings to and from redis. I kept a set of model objects which I serialized directly into JSON, and each model object had an ‘update’ method that took the deserialized data and copied it into the instance. The goal of this was to make sure the fields on the object where always consistent, and gave the opportunity to deal with different versions of the data.

I think the approach was generally okay, but in its current state there’s a lot of duplication. If I were to rewrite it, I’d attempt to abstract much of the serializing, deserializing and object update logic. I’d also attempt to centralize the actual calls to redis, as at the moment there’s a lot of code that’d need to change if I wanted to change data providers.

The last thing I’d like to mention is using a Block to centralize error handling, as mentioned Stella Laurenzo’s blog post. This pattern fits very nicely into the Express framework and can be a lifesaver, because any exception that bubbles up to the core Node.js event loop will terminate your Node.js instance. The code in Express itself already has error handling built in, but if a callback comes from somewhere else (such as redis) the exception will cause problems. The Block pattern simplifies the error handling, reducing the amount of code you need to write. This sort of error handling is a requirement for any Node.js app.

Final Thoughts

If you’re coming from a Windows / .NET development background there’s a steep, but short, learning curve to start programming in Node.js, but it’s well worth it. JavaScript offers a very different way to thinking about application programming from C# and VB.net, it feels more free-form and dynamic and gives you more of an opportunity mould your application into a shape that you feel comfortable with.

Node.js is still a very new environment, tools are still being built, patterns and best practices are being forged, much will change in times to come and it all makes Node.js a fun tool to work with. It has great potential as a tool for building websites and so much more, and I hope that you get a chance to use it on your next project.

// Phil