Now, this isn't necessarily a problem when it comes to building tools. In fact, I think it's a great demonstration of flatiron's flexibility. Each project can use exactly the libraries and organization that make sense for their applications, even for apps as different as haibu and jitsu. It can be confusing for newcomers, though, that are used to frameworks that enforce stronger conventions, or those that are trying to figure out how to get from "hello world" to something more profound.

I thought what would be helpful for flatiron newcomers is an in-depth tutorial, which iterates on a "hello world" flatiron server (like those in the project's examples) into a relatively complex webservice that uses some of flatiron's more complex functionality.

In this tutorial, I'm going to create a webservice for browserify, a popular client-side code framework for node.js. All of the code is on github, and every section of this tutorial corresponds to a code commit. In early parts of this tutorial I will show all the code, but as it gets longer I will typically explain the code with links to the relevant files/commits on github.

Flatiron HTTP Hello World

To get started, let's write a simple "hello world" app using flatiron, in ./app.js:

// Require flatiron and grab the app object.
var flatiron = require('flatiron'),
    app = flatiron.app;

// Use the http plugin. This makes flatiron act as an http server with a
// router on `app.router`.
app.use(flatiron.plugins.http);

// Route handler for http GET on the root path
app.router.get('/', function () {
  // The request and response objects are attached to `this`.
  var req = this.req,
      res = this.res;

  // Flatiron comes with a logging object already attached!
  app.log.info('Saying hello!');

  // Handle the response as you would normally.
  res.writeHead(200, { 'content-type': 'text/plain'});
  res.end('hello!');
});

// Start the server!
app.start(8080, function (err) {
  if (err) {
    // This would be a server initialization error. If we have one of these,
    // the server is probably not going to work.
    throw err;
  }

  // Log the listening address/port of the app.
  var addr = app.server.address();
  app.log.info('Listening on http://' + addr.address + ':' + addr.port);
});

Writing a basic flatiron server is pretty simple. First, we require flatiron and grab the 'app' object. Then, we assign a route handler to '/' which responds to request with the text, 'hello!'. This server also takes advantage of flatiron's built-in logging to print information to the screen.

Here's the accompanying ./package.json:

{
  "name": "flatiron-helloworld",
  "version": "0.0.0",
  "dependencies": {
    "flatiron": "0.1.16",
    "union": "0.3.0"
  }
}

The most important thing to notice here is that union is also a dependency. Union is what's called a "peer dependency". This means that flatiron doesn't itself list union as a dependency, but the http plugin expects to be able to require it. This is because flatiron has uses outside of http servers where it doesn't make sense to install union.

Luckily, if you forget, flatiron will return a reasonable error message:

$ node app.js 
flatiron.plugins.http requires the `union` module from npm
install using `npm install union`.
Trace: 
    at Object.<anonymous> (/home/josh/dev/nodejitsu/flatiron-tutorial/hello/node_modules/flatiron/lib/flatiron/plugins/http.js:25:11)
    at Module._compile (module.js:441:26)
    at Object..js (module.js:459:10)
    at Module.load (module.js:348:31)
    at Function._load (module.js:308:12)
    at Module.require (module.js:354:17)
    at require (module.js:370:17)
    at Object.http (/home/josh/dev/nodejitsu/flatiron-tutorial/hello/node_modules/flatiron/node_modules/broadway/node_modules/utile/lib/index.js:261:14)
    at Object.<anonymous> (/home/josh/dev/nodejitsu/flatiron-tutorial/hello/app.js:7:25)
    at Module._compile (module.js:441:26)

Once flatiron and union are both installed, of course, our app works as expected:

$ node app.js 
info: Listening on http://0.0.0.0:8080

Now we can make a request to our server and receive a kind message:

$ curl localhost:8080
hello!

Furthermore, we can see evidence of our request in the server's logs:

$ node app.js 
info: Listening on http://0.0.0.0:8080
info: Saying hello!

"Hello World" With A Twist: Configuration and POST

Let's make our "Hello World" app slightly more sophisticated:

var path = require('path');

var flatiron = require('flatiron'),
    app = flatiron.app;

// Set up app.config to use ./config.json to get and set configuration settings.
app.config.file({ file: path.join(__dirname, 'config.json') });

app.use(flatiron.plugins.http);

// This router syntax allows you to define multiple handlers for one path based
// on http method.
app.router.path('/', function () {

  // This is the same functionality as previously.
  this.get(function () {
    this.res.writeHead(200, { 'content-type': 'text/plain' });
    this.res.end('hello!');
  });

  // Now, when you post a body to the server, it will reply with a JSON
  // representation of the same body.
  this.post(function () {
    this.res.json(200, this.req.body);
  });
});

// Now we're using app.config to set the port, with a default of 8080.
app.start(app.config.get('port') || 8080, function (err) {
  if (err) {
    throw err;
  }

  var addr = app.server.address();
  app.log.info('Listening on http://' + addr.address + ':' + addr.port);
});

The accompanying package.json has a few minor changes as well:

{
  "name": "browserify-cdn",
  "version": "0.0.0",
  "scripts": {
    "start": "node ./app.js"
  }
  "dependencies": {
    "flatiron": "0.1.16",
    "union": "0.3.0"
  }
}

Besides a name change, our app also now has a "scripts.start" field. This means two things:

• 'npm start' will start your app.
• 'jitsu deploy' will successfully start your app.

Note that this app also has configuration now, using app.config! We're using this to load configuration settings from ./config.json:

{
  "port": "3600"
}

So, if you run this server and GET /, the response is the same as before:

$ curl localhost:3600
hello!

However, if we POST some data:

$ curl -X POST -d 'foo=foo&bar=bar' localhost:3600
{"foo":"foo","bar":"bar"}

We can also post JSON if we remember to set the content-type header properly:

$ curl -X POST -d '{ "foo": "foo", "bar": "bar" }' -H 'Content-Type: application/json' localhost:3600
{"foo":"foo","bar":"bar"}

This all Just Works is because flatiron has built-in tools for buffering and parsing request bodies, making the development of JSON APIs dead easy.

Now we're ready to do something a bit more interesting.

Basic Browserifying Server & Separation of Concerns 101

Now we're going to make our app browserify posted code snippets. Once it's done, the interface should work like so:

$ curl -X POST -d 'var traverse = require("traverse");' localhost:3600
var require = function (file, cwd) {
    var resolved = require.resolve(file, cwd || '/');
    var mod = require.modules[resolved];
    if (!mod) throw new Error(
        'Failed to resolve module ' + file + ', tried ' + resolved
    );
    var res = mod._cached ? mod._cached : mod();
    return res;
}

require.paths = [];
require.modules = {};
require.extensions = [".js",".coffee"];

require._core = {
    'assert': true,
    'events': true,
    'fs': true,

**snip**

I didn't include the entire bundle here, but the general idea is this: You post browserify-able javascript code to the service, and it returns the browserified bundle.

By this point, our app is getting big enough that we want to break functionality up into different pieces. The way I approach this is like so:

1. Write a library (or a few) that implement the basic pieces of functionality you need.
2. Call that library from ./app.js, only handling http details in the router.

The idea here is to separate the concerns so that there isn't routing logic in the bundler or bundling logic in the router. In larger apps, this approach gets broken down further into large-scale design patterns like MVC. In this case our app is pretty small though, so simply separating the bundling into its own library will suffice.

The bundler, which lives in ./bundler.js for this iteration, is a constructor that inherits from EventEmitter2:

var EventEmitter2 = require('eventemitter2').EventEmitter2,
    browserify = require('browserify'),
    detective = require('detective'),
    npm = require('npm'),
    crypto = require('crypto'),
    util = require('utile');

// I wrote the Bundler as a constructor with prototype methods.
// I find that it's a good fit for stateful problems.
var Bundler = module.exports = function (opts) {

  // This lets you create a new Bundler without the 'new' keyword.
  if (!(this instanceof Bundler)) {
    return new Bundler;
  }

  var bundler = this;

  // Set the bundler's persistent options here. Also handle defaults.
  bundler.options = opts || {};
  bundler.options.npm = bundler.options.npm || {};

  // Overrides falsy values such as `undefined`
  if (bundler.options.cache !== false) {
    bundler.options.cache = true;
  }

  // Bundler inherits from an EE2 with wildcards and the :: delimiter.
  EventEmitter2.call(this, {
    wildcard: true,
    delimiter: '::',
    maxListeners: 0
  })

  // Bundler requires a loaded npm in order to work properly.
  // The rest of the code in the constructor sets this up and emits events
  // to signal when it's ready.
  bundler.ready = false;

  npm.load({}, function (err) {
    if (err) {
      // What would be really cool is if I checked to see if there was an error
      // event listener or not.
      bundler.emit('error', err);
    }

    bundler.ready = true;
    bundler.emit('bundler::ready');
  });
};

util.inherits(Bundler, EventEmitter2);

The bulk of the constructor handles setting default options for our object and configuring the EventEmitter2 aspects of the object. The interesting piece here calls npm.load on the npm singleton object, emits events when it succeeds or fails, and sets a this.ready flag. This allows us to create our object and then listen for when we may use it.

Next, we'll define the prototype method that actually bundles our code:

// The 'bundle' method is what actually attempts to bundle your project.
Bundler.prototype.bundle = function (src, cb) {
  var bundler = this,
      modules = detective(src);

  // We used 'detective' to get a list of needed modules, so that we can make
  // sure they're installed before trying to browserify.
  npm.commands.install(modules, function (err) {
    if (err) {
      cb(err);
    }

    var bundle;

    // Attempt to browserify the passed-in javascript source
    try {
      bundle = browserify(this.options)
        .addEntry('index.js', { body: src || '' })
        .bundle();

      // Hit the callback with our complete bundle object.
      cb(null, {
        src: src,
        md5: crypto.createHash('md5').update(src).digest('base64'),
        bundle: bundle
      });
    }
    catch (err) {
      cb(err);
    }
  });
};

This method handles making sure that required modules are installed, and then attempts to browserify the code. It then calls back with an object representing the code bundle.

Finally, let's add .attach and .init methods, which will let our new module act as a broadway plugin:

// Bundler can be used as a broadway plugin.
Bundler.attach = function (opts) {
  this.bundler = new Bundler(opts);
}

// The major win we get from making Bundler attachable is that we can integrate
// its initialization step with our app.
Bundler.init = function (done) {
  var npm = 'npm'.red.inverse;

  if (this.bundler.ready) {
    console.log('init: %s is ready.', npm);
    return done();
  } else {
    console.log('init: Waiting for %s.', npm);
    this.bundler.once('bundler::ready', function () {
      console.log('init: %s is ready.', npm);
      return done();
    });
  }
}

Now that the bundler library is done, all we have to do is integrate it with our ./app.js.

First, let's talk about how the bundler was integrated into the app. After requiring it, we use bundler like so:

// This is a new library used for running browserify bundle jobs for us.
var bundler = require('./bundler');


// Note that 'bundler' is written to work as a flatiron plugin.
app.use(flatiron.plugins.http);
app.use(bundler);

This creates a property called app.bundler, and adds an initialization step so that when we call app.start npm is also loaded.

Later, we use app.bundler in our POST handler like so:

this.post(function () {

  // If the request body doesn't have the property we expect, it's assumed
  // to be raw javascript. Note that the raw unparsed body is buffered into
  // req.chunks (as a Buffer).
  var req = this.req,
      res = this.res,
      js = req.body.js ? req.body.js : req.chunks.toString();

  // 'app.bundler' was created when we attached 'bundler'.
  app.bundler.bundle(js, function (err, data) {
    if (err) {
      return res.json(500, {
        success: false,
        reason: err.message || 'unknown'
      });
    }
    res.writeHead(200, { 'content-type': 'text/javascript' });
    res.end(data.bundle);
  });
});

Note that most of the code here is about figuring out what the user wants to browserify and how to respond in return, and that all the bundling takes place in a single call.

Finally, here's the ./package.json for completeness:

{
  "name": "browserify-cdn",
  "version": "0.0.0",
  "scripts": {
    "start": "node ./app.js"
  },
  "dependencies": {
    "flatiron": "0.1.16",
    "union": "0.3.0",
    "utile": "0.0.10",
    "eventemitter2": "0.4.x",
    "browserify": "1.10.6",
    "detective": "0.1.x",
    "npm": "1.1.x"
  }
}

The only important changes are in the dependencies field, since we now have a few more of them!

Finally, we can run our service with npm start, and when a user runs the curl command from earlier (curl -X POST -d 'var traverse = require("traverse");' localhost:3600) they will receive a bundle and our logs will look like so:

$ npm start
npm info it worked if it ends with ok
npm info using npm@1.1.14
npm info using node@v0.6.14
npm info prestart browserify-cdn@0.0.0
npm info start browserify-cdn@0.0.0

> browserify-cdn@0.0.0 start /home/josh/dev/nodejitsu/flatiron-tutorial
> node ./app.js

init: Waiting for npm.
init: npm is ready.
info: Browserify-CDN Listening on http://0.0.0.0:3600
npm http GET https://registry.npmjs.org/traverse
npm http 304 https://registry.npmjs.org/traverse
npm info into /home/josh/dev/nodejitsu/flatiron-tutorial traverse@0.6.1
npm info installOne traverse@0.6.1
npm info unbuild /home/josh/dev/nodejitsu/flatiron-tutorial/node_modules/traverse
npm info preuninstall traverse@0.6.1
npm info uninstall traverse@0.6.1
npm info postuninstall traverse@0.6.1
npm info preinstall traverse@0.6.1
npm info build /home/josh/dev/nodejitsu/flatiron-tutorial/node_modules/traverse
npm info linkStuff traverse@0.6.1
npm info install traverse@0.6.1
npm info postinstall traverse@0.6.1
traverse@0.6.1 ./node_modules/traverse

Congratulations, you now have a working browserify webservice!

Add Caching with Resourceful

So, our browserify service works, but there's a problem: Our browserify webservice has to bundle the project every time, even if it's done it before! We can minimize this with caching.

To facilitate this, I reorganized the project a little bit:

$ ls
app.js  config.json  lib  node_modules  package.json  post
$ ls -R ./lib
./lib:
bundler  common.js

./lib/bundler:
bundler.js  cache.js  index.js

As you can see, I moved bundler.js into ./lib/bundler and added a cache.js and an index.js.

You'll also notice that I removed config.json from the git repository. This is because it now includes credentials to a couchdb. Mine looks like this:

{
  "port": "3600",
  "couch": {
    "https": true,
    "uri": "**********.iriscouch.com/browserify",
    "port": "6984",
    "auth": {
      "username": "**********",
      "password": "**********"
    }
  }
}

In order to add caching, the approach I took was to create a new constructor which inherits from Bundler, called CachingBundler, in ./lib/bundler/cache.js. This new constructor contains a new prototype method, .get, which checks our cache and only calls .bundle if required. This sort of inheritance pattern is appropriate for cases where your new object is "an x that also does y"---for example, Bundler is an EE2 that also bundles, and CachingBundler is a Bundler that also caches.

Our new library starts with a long list of requires:

var Bundler = require('./bundler'),
    common = require('../common'),
    resourceful = require('resourceful'),
    errs = require('errs'),
    colors = require('colors'),
    utile = require('utile');

Bundler and common are internal libraries, while resourceful, errs and utile are all flatiron components. This caching layer uses resourceful to interact with a storage backend (in this case, couchdb). Resourceful is an "Object Document Mapper," or ODM, which means that it internally maps javascript objects and prototypes with documents in a document store like couch. This saves developers the step of wrapping their couchdb http calls in a custom abstraction, and is typically used to put the "model" in MVC.

Here's what creating a resource with resourceful looks like:

// Cached bundle resources.
var CachedBundle = resourceful.define('cachedBundle', function () {
  this.string('md5');
  this.string('source');
  this.string('bundle');
});

This resource is quite simple. It's called a 'cachedBundle', and is expected to at least have three strings (md5, source and bundle).

Now, we can create our constructor:

// A version of the bundler that caches. Use `.get(text)`.
var CachingBundler = module.exports = function (opts) {
  // This lets you create a new caching Bundler without the 'new' keyword.
  if (!(this instanceof CachingBundler)) {
    return new CachingBundler;
  }

  opts = opts || {};
  Bundler.call(this, opts);
  // Set up resourceful to use couchdb.
  resourceful.use('couchdb', opts);
}

utile.inherits(CachingBundler, Bundler);

Generally, this constructor just repeats the actions of the original Bundler constructor and then inherits from it. However, it also sets up resourceful to use our couchdb, using the options passed into it.

The meat of our new functionality is in the get prototype method:

CachingBundler.prototype.get = function (src, cb) {
  var self = this,
      md5 = common.md5(src);

  // This logging function simply adds some extra text to the message and
  // then emits it with the event 'log::lvl' (ex: 'log::info').
  function log (lvl, msg) {
    var restOf = [].slice.call(arguments, 2),
        data = utile.format.apply(
          null,
          [ '(md5 `%s`) '.cyan + msg, md5 ].concat(restOf)
        );

    self.emit('log::' + lvl, data);
  }

  log('info', 'Trying to find bundle...');

  // Check to see if the file already exists in the cache, based on md5 hash of
  // the source text. This logic accounts for the possibility of hash collisions
  // and multiple versions of the same bundle.
  CachedBundle.find({ md5: md5 }, function (err, bundles) {
    if (err) {
      return cb(realError(err));
    }

    var matches;

    if (!bundles.length) {
      // No cached bundles found. We should bundle.
      log('warn', 'Not found.');
      return bundle(src);
    }

    // Log the document id's for found bundles. If there is more than one
    // bundle, it means there was either an md5 hash collision or multiple
    // documents for the same bundle exist.
    log('info', 'Found bundle(s): %j', bundles.map(function (doc) {
      return doc._id;
    }));

    // Try to find one that has the same text.
    matches = bundles.filter(function (bundle) {
      return bundle.source == src;
    });

    // Any answer other than 1 means an actual md5 hash collison.
    switch (matches.length) {
      case 1:
        log('info', 'Match found.');
        finish(null, matches[0]);
    break;
      case 0:
        log('info', 'Hash collision with existing document(s) %j', matches);
        bundle(src);
        break;
      default:
        log('Multiple documents describing this document exist.');
        log('Showing the first one.');
        finish(null, matches[0]);
        break;
    }
  });

  // This function is called when we need to bundle. It addition to logging, it
  // also attempts to cache the resulting bundle.
  function bundle (src) {
    log('info', 'Bundling.');
    self.bundle(src, function (err, doc) {
      if (err) {
        return cb(err);
      }

      // Here, we create a new CachedBundle, which maps to our data store.
      var bundle = new CachedBundle(doc);

      // This is how you save resources with resourceful.
      log('info', 'Caching bundle for next time...')
      CachedBundle.save(bundle, function (err, bundle) {
        if (err) {
          log('error', 'Error while caching bundle');
        }
        return finish(realError(err), bundle);
      });

    });
  }

  // This is pretty much the last step, whether the doc came from the cache
  // or from a bundling.
  function finish (err, bundle) {
    if (err) {
      return cb(err);
    }

    log('info', 'Returning bundle.');
    cb(null, bundle);
  }

};

Most of the logic you can see involves grabbing the document, and making sure it's the right one, and logging the behavior using the EventEmitter2 api (remember, Bundler inherited from EventEmitter2). Interaction with our datastore is simplified to the creation of a resource type, a call to .find and a call to .save.

Like Bundler, our caching bundler also has .attach and .init methods for integration with flatiron:

// Attach the caching bundler, passing in our config options for the couch.
// Also emits logging events to the app.
CachingBundler.attach = function (options) {
  var app = this;

  this.bundler = new CachingBundler(utile.mixin(
    options,
    this.config.get('couch')
  ));

  // Little known fact about flatiron: The logging plugin will log messages
  // emitted on the `log` namespace of the core app, which is an instance of
  // EventEmitter2.
  app.bundler.on('log::**', function (msg) {
    app.emit(this.event, msg);
  });

};

CachingBundler.init = Bundler.init;

The init method is exactly the same. The new behaviors are in our attach method. One of these behaviors is to pass our app's couchdb configuration to the object constructor. The other behavior is to re-emit logging events on the app object, whose log plugin will handle them.

I also created a ./lib/bundler/index.js which looks like so:

module.exports = require('./cache');

and allows us to simply require our new caching bundler in ./app.js like so:

var bundler = require('./lib/bundler');

Now, let's run our app and do another curl -X POST -d 'var traverse = require("traverse");' localhost:3600:

$ npm start
npm info it worked if it ends with ok
npm info using npm@1.1.14
npm info using node@v0.6.14
npm info prestart browserify-cdn@0.0.0-3
npm info start browserify-cdn@0.0.0-3

> browserify-cdn@0.0.0-3 start /home/josh/dev/nodejitsu/flatiron-tutorial
> node ./app.js

init: Waiting for npm.
init: npm is ready.
info: Browserify-CDN Listening on http://0.0.0.0:3600
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Trying to find bundle... event=log::info
warn: (md5 `RxLf0LntDlue3quYVoaVtw==`) Not found. event=log::warn
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Bundling. event=log::info
npm http GET https://registry.npmjs.org/traverse
npm http 304 https://registry.npmjs.org/traverse
npm info into /home/josh/dev/nodejitsu/flatiron-tutorial traverse@0.6.1
npm info installOne traverse@0.6.1
npm info unbuild /home/josh/dev/nodejitsu/flatiron-tutorial/node_modules/traverse
npm info preuninstall traverse@0.6.1
npm info uninstall traverse@0.6.1
npm info postuninstall traverse@0.6.1
npm info preinstall traverse@0.6.1
npm info build /home/josh/dev/nodejitsu/flatiron-tutorial/node_modules/traverse
npm info linkStuff traverse@0.6.1
npm info install traverse@0.6.1
npm info postinstall traverse@0.6.1
traverse@0.6.1 ./node_modules/traverse
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Caching bundle for next time... event=log::info
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Returning bundle. event=log::info

Naturally, this request wasn't cached so our app had to bundle it. If you check your couch with futon, you'll see that our document has been stored by resourceful. So, if we curl our api again:

info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Caching bundle for next time... event=log::info
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Returning bundle. event=log::info
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Trying to find bundle... event=log::info
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Found bundle(s): ["f4328d0a958746dfa8b45856a900134e"] event=log::info
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Match found. event=log::info
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Returning bundle. event=log::info

Since we've already bundled that particular source snippet, we may simply return it! Now we're cooking.

What Now?

In this quick tutorial, I showed you how I built the foundation for the new browserify-cdn project, which is hosted at http://browserify.nodejitsu.com, by starting with a "hello world" flatiron example and then iterating on it until we had a fully-working api server. Along the way, we learned a little about flatiron core's capabilities and many of its supporting libraries, as well as touching on concepts for organizing medium-sized apps.

The master branch has a number of tweaks and improvements, the most noticeable of which is serving an html landing page and static content using my static server middleware and adding more colors to the text output. Of course, now that you have your own browserify-cdn, you can change it however you want!

If you do, don't forget to send me a pull request.

We are also co-hosting two events in April: The Node Philly Conference and the NYC MongoDB Hackaton and speaking at three more!

Here's our schedule: be sure to say hi and hangout.

JSConf (&NotConf)

It's JSConf! Enough said! Pre conference will be notconf and it is looking amazing!

Web-5 Conference (Béziers, April 4-6)

Maciej and Charlie will be attending the Web-5 conference in Béziers, France. The conference revolves around Web Technologies and has some great speakers. Great cheese and great weather are also givens. You can register for the conference here.

Dyncon (Stockholm, April 21-22)

Dyncon is an fantastic event. Charlie spoke last year and had a blast. Nuno will be doing the honors for Nodejitsu this year, and we expect great talks from a great selection of speakers.

Node Philly (Philadelphia, April 23)

Node Philly (node.ph) is the first ever node.js conference in Philly. Charlie and Paolo are attending, as well as some great minds from the Node.js community like James Halliday and Mikeal Rogers. The event is part of a larger initiative called Philly Tech Week. Tickets are available for a nominal fee.

MongoDB Hackaton NYC (NYC, April 27)

We are co-hosting the MongoDB hackaton with 10gen. Charlie will be on location to help you out with your applications, and get you some free Nodejitsu accounts. This event is free and will be in the 10gen NYC headquarters.

Now, after three years of toiling away on Github, we have now expanded our engineering team to over fourteen strong and every Nodejitsu developer is an active contributor to the open-source community.

• How to become an open-source developer?

• How do you get recruited as an open-source developer?

#

How to become an open-source developer?

Becoming an "open-source developer" is actually quite simple. You really only need to do one thing: contribute.

Contributing can be done on several levels of participation. One of the easiest ways to contribute to a project is to use it, find an issue, and report the problem. If you've found an actual problem that can be resolved, hopefully one of the maintainers will respond by fixing the issue or providing guidance on how to resolve the issue.

There are also several other useful things you can do such as: unit-tests, benchmark suites, documentation, multi-platform support, third-party integrations, and new functionality.

How do you get recruited as an open-source developer?

If you are contributing ( as described in the previous paragraph ), you've already done the hardest part.

Good work always speaks for itself.

If you have worked on projects which are significant to companies who are hiring developers, they probably already know your name. It's much more likely a company will hire a developer who has already made an improvement to a piece of their codebase versus someone who has little knowledge of any company projects. At Nodejitsu, we primarily hire developers who have already contributed to major company supported projects like: jitsu, haibu, flatiron, and hook.io.

How to recruit open-source developers?

Recruiting open-source developers is not difficult, but it can be very difficult for certain companies.

First off, don't cold email open-source developers. If an open-source developer has posted their email in a public place ( like on Github ), they probably put it up for technical support emails, not recruiter spam.

Sponsoring hackathons with open-source incentives is always a great idea.

Allow company time for open-source projects.

This isn't about setting an arbitrary percentage of time like 10% or 20% of hours. If an employee is working on a project in their free time that could help improve open-source software in a significant way, you should encourage them to work on it. Hopefully, the projects they are working on in their free time align with the goals of your company.

Use algorithmic recruitment to find developers.

http://www.githits.me/ is fun.

http://coderwall.com/leaderboard is also good.

http://develop.github.com/ is great.

The data is out there. Do some googling and get creative.

Upcoming jobs at Nodejitsu

At Nodejitsu, our developers are top-class and their open-source contributions stand on their own. Our developers are constantly spammed for job offers and choose to work for us because we try to let them do the work they would be doing anyway.

Over the next few months, we will be aggressively expanding our development team and hiring new people who can step up in any of the following roles.

• Support Engineering
• Development Operations
• Front-End Development
• User-Experience
• Design and Graphic Art

We are looking for people who are self-motivated and are always willing to take the extra step needed to get the job done. We are looking for people who can not only provide results, but can also proactively take the initiative to get results.

Location: On-Site in San Francisco or New York City, Telecommute

• Compensation is competitive and based on experience
• Health insurance (including dental and vision)
• Equity opportunities available for senior roles

If you are interested in working for Nodejitsu, send an email to hiring@nodejitsu.com or better yet...

Get a pull request merged on Github.

Now get coding.

asynchronous functions

an asynchronous function is one that usually returns before having finished its job and takes a function it should call when the job is done. that action is the callback, a function to be executed at that point in time.

// async
var fs = require('fs');
// callback
function logContents(err,contents) { console.log(contents); }
var contents = fs.readFile('a.txt', 'utf8', logContents); 
console.log('1');
// outputs 1, then the contents of the file when the file is read

vs.

// sync
var fs = require('fs');
var contents = fs.readFileSync('a.txt', 'utf8'); 
console.log(contents);
console.log('1');
// outputs the contents of the file, then 1.

iterating in javascript with asynchronous functions

the problem

when someone is starting with node i always recommend they watch pedro teixeira's video nodetut on asynchronoys iteration patterns

in this nodetuts video pedro shows how to iterate asynchronously in node.js. here is the naive approach code he shows which doesn't work:

// simulating an asynchronous call, e.g. getting something from a database
var asyncFn = function(data, callback) {
  var timeout = Math.ceil(Math.random() * 3000);
  // run callback after timeout miliseconds
  setTimeout(function() { callback(null, data); }, timeout);
};

var insertAll = function(coll, callback) {
  coll.forEach(function (nr) {
    asyncFn(nr, function(err,data) {
      console.log('‣', data);
    });
  });
  callback();
};

insertAll([1, 2], function() {
  console.log('done'); });

this will not work as you probably think it does:

done
‣ 1
‣ 2

the reason is that the insertAll function returns before the async function calls the callback so done gets logged before the function is complete.

solving the problem with a module

if you think implementing your own asynchronous iteration function is too much hassle there's plenty of modules out there that solve this problem. my favorite is async.

in async the solution for this problem would be something like:

async.forEach([1, 2], asyncFn, function(err, results){
   console.log('done');
});

to iterate is human, to recurse is divine

when looking to the solution to this problem without the help of a module it struct me as obvious how much all these algorithms work like pattern matching.

• if the queue exists run some generic processing the queue function
• if the queue is empty run the callback

what is pattern matching?

pattern matching is a recursive way to develop your programs where you define patterns for the program to identify and match those patterns with actions that should be executed.

let's consider the very well understood map function that takes a collection and transforms each element by applying a function:

[1,2,3].map(function duplicate(x) { return x*2; });

if you solved this problem using pattern matching the execution of this function would be:

transformed_map = [];
map:
  l = [1,2,3]
    is l an empty array?
      no.
    is l anything?
      yes.
      head = 1
      tail = [2,3]
      transformed_map.shift(duplicate(1));
      map([2,3])
transformed_map = [2];
map:
  l = [2,3]
    is l an empty array?
      no.
    is l anything?
      yes.
      head = 2
      tail = [3]
      transformed_map.shift(duplicate(2));
      map([3])
transformed_map = [2,4];
map:
  l = [3]
    is l an empty array?
      no.
    is l anything?
      yes.
      head = 3
      tail = []
      transformed_map.shift(duplicate(3));
      map([])
transformed_map = [2,4,6];
map:
  l = []
    is l an empty array?
      yes.
      return transformed_map
[2,4,6]

solving the problem with pattern

i decided try to implement pattern matching in javascript as my js1k entry called functional love.

the rules were simple: implement functions that allow you to do pattern matching in javascript without changing the language itself.

this is what i came up to for the node tuts episode:

insert_all([], function done() { console.log('done'); });
insert_all(_, function all(l) {
  insert_element(l.shift(), function (err, elem) {
    console.log('‣ ', elem);
    insert_all(l);
  });
});

• if the array is empty, log the message done.
• if its not empty, insert the element and when the callback is complete recursively call the insert_all function with the rest of the list.

you can now call this function like this:

insert_all([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);

how it works

you probably noticed i use the same syntax to declare the insert all function and to call the function itself. what happens is the first time you call pattern it sets the arity of the function. is this case the first pattern has two arguments, meaning i should expect a function with one argument. whenever i call insert_all with one argument i'm calling the function.

when you registered the first two patterns i created a fact table like:

now when you call the insert_all function we need to iterate this fact table and see which rule applies. then call that function. simple. if the function wants to recurse it simply need to invoke the insert_all function.

mad science corner

ok enough with the easy stuff. let's use pattern to implement a function that reverses an javascript array. asynchronously of course:

reverse([5,4,3,2,1], function (list) { console.error(list); });

we can do this using the reduce function that takes a list l and an accumulator z and returns something that was constructed by applying a function f to each element of the list x and the accumulator z.

// pretending asynchronous behavior
// image this is some kind of echo server
function echo(z, x, cb) {
  setTimeout(function() {
    z.unshift(x); cb(z);
  }, Math.ceil(Math.random() * 100)); 
}

function reverse (l, cb) { reduce(echo, [], l, function(z) { cb(z); }); }

now all we need is to implement the reduce function with pattern:

var _;
reduce(_, _, [], _, function (f, z, l, cb) { cb(z); });
reduce(_, _, _, _, function (f, z, l, cb) {
  f(z, l.shift(), function (new_z) {
    reduce(f, new_z, l, cb);
  });
});

get the full code for this example here.

js1k

pattern is my js1k submission. it was a lot of fun to write. if you are wondering if you should submit an entry yourself you totally should.

Flatiron is an adaptable decoupled framework for both node.js and the browser. An adaptable framework enables users to build multiple types of applications. In Flatiron we are focusing on:

• Web applications (http and websockets)
• Command line (CLI) applications
• Front-end applications

This article will focus on the tools at your disposal when using Flatiron to build CLI applications as well as how many existing Flatiron CLI applications are structured.

• Getting started
• To the code!
• Configuring your Command-line Application
• Composable Command-line applications
• Flatiron Command-line apps in the wild

Getting started

Writing a Command-line application with Flatiron starts out like writing every other flatiron application:

  $ flatiron create simple-app cli
  info:   Creating application simple-app
  info:   Using cli scaffold.
  info:   Creating directory config
  info:   Creating directory lib
  info:   Creating directory commands
  info:   Creating directory test
  info:   Writing package.json
  info:   Writing app.js
  info:   Writing lib/index.js
  info:   Application simple-app is now ready

Lets examine the directory structure created by running flatiron create simple-app cli:

  simple-app/
    app.js
    config/
    lib/
      index.js
      commands/
    package.json
    test/

With this simple scaffold the application has already been setup with:

• Commands lazy-loaded from the file or directory specified by options.source.
• app.cmd: Helper function for creating CLI routes.
• app.argv: Parsed CLI arguments using optimist.
• app.config: Hierarchical configuration sourced from: environment variables, CLI arguments and a JSON file in that order.
• app.log: Multi-transport logging using winston.
• app.alias: Helper function to alias shallow commands (e.g. app config list to app conf).
• app.inspect: Robust object inspection and formatting using cliff.
• app.prompt: CLI prompt using the Flatiron prompt module.

There are other options which make writing CLI applications quick and easy. Leave the tough stuff to us.

To the code!

Lets examine each line of the code generated by flatiron create simple-app cli:

var flatiron = require('flatiron'),
    path = require('path'),
    app = flatiron.app;

app.config.file({ file: path.join(__dirname, 'config', 'config.json') });

app.use(flatiron.plugins.cli, {
  source: path.join(__dirname, 'lib', 'commands'),
  usage: 'Empty Flatiron Application, please fill out commands'
});

app.start();

Setup up Configuration

  app.config.file({ file: path.join(__dirname, 'config', 'config.json') });

Add a store to back the configuration for this application. It will automatically load :pwd/config/config.json from disk.

Attach the CLI plugin

  app.use(flatiron.plugins.cli, {
    source: path.join(__dirname, 'lib', 'commands'),
    usage: 'Empty Flatiron Application, please fill out commands'
  });

Attach Command-line functionality to the application. See Configuring your Command-line Application for more detailed documentation.

Start the Application

  app.start();

Initializes the application and then dispatches to the director router using app.argv._. For example when running app config get foo, app.argv._ will be ['app', 'config', 'get', 'foo'].

Configuring your Command-line Application

There is a simple convention for passing options to both flatiron.plugins.cli and flatiron.plugins.http: the settings for the named property added to the application has the same name in the options passed to .use(). For example:

//
// Configuring app.argv
//

app.use(flatiron.plugins.cli, {
  //
  // Settings used by `require('optimist').options()
  //
  argv: {
    login: {
      alias: 'l',
      description: 'Login to use',
      string: true
    }
  }
});

//
// `app.argv.l` is now available
//
console.log(app.argv.l);

There are some important settings you need to be aware of:

• source {string}: File or directory on disk to lazy load commands from. In the case of a directory, the names of the files map to the first parameter in app.argv._.
• usage {string|Array}: Help text to display when using app help
• argv {Object}: Settings for CLI options parsing.
• prompt {Object}: Settings for Flatiron prompt module.
• version: {boolean}: Automatically sets up response for app --version and app -v
• async: {boolean}: If true configures the CLI router attached to the application to be asynchronous. See the director documentation for more information.

Composable Command-line applications

Flatiron enables developing adaptable applications through a lightweight dependency injection system called broadway. Writing broadway plugins is easy and can be used to easily compose applications out of reusable components. This was the motivation behind a new CLI-plugin for Flatiron released today: flatiron-cli-config. This plugin extends the application by adding handlers for app config *. It also adds help for all of these commands available through app help config *:

• app config list: Lists all key-value pairs known to the application.
• app config get <key>: Displays the value for to the user.
• app config set <key> <value>: Sets the to the specified value.
• app config delete <key>: Clears the value for the specified .

Using this plugin is simple:

  app.use(require('flatiron-cli-config'));

That's it. Your application is now fully setup to use edit configuration files from the command-line.

Flatiron Command-line applications in the wild

Since we released Flatiron there have been a number of interesting applications released and refactored:

• issues: Github Issues from the CLI
• box: Powerful key -> value storage for the CLI.
• todo: Todos in the CLI like what.
• npmcount: Silly program that counts number of npm packages from one or more users.
• jitsu: Flawless command line deployment of your Node.js apps to the cloud.

Well that about wraps it up. Can't wait to see what crazy things I'll be able to do from the CLI soon!

The main goal of SPDY is reducing the latency of web pages by multiplexing multiple http streams into one TCP connection and compressing http headers.

So instead of creating a TCP connection for each http request (like styles, scripts, images, or even ajax), SPDY-enabled browser will hold only one connection with the server.

source: http://www.igvita.com/2011/10/20/faster-web-vs-tcp-slow-start

Your server can tell the browser that it supports SPDY in two ways: setting the ‘Alternate-Protocol’ header, or by using a TLS handshake extension NPN (Next Protocol Negotiation). In the second case the server will send the supported protocols (i.e., SPDY/2 HTTP/1.1 HTTP/1.0) to browser before any data transfer will be performed, and the browser will choose the protocol that is preferable for the client.

You can learn more about SPDY by examing the protocol specification: http://www.chromium.org/spdy/spdy-protocol/spdy-protocol-draft2

Node.js and SPDY

The growing interest in both the SPDY protocol and Node.js created the need for a stable Node.js SPDY module when the first version was released last year. There were a number of issues with this first implementation: it was buggy, it did not support the default node.js builds, and it was incompatible with popular frameworks (only partial express support, and no socket.io support at all). Despite all of these issues it still attracted considerable some attention from the Node.js community.

Before continuing development on node-spdy it was clear that node.js core needed to be improved: the tls module was lacking NPN support, and there was no zlib functionality. While adding NPN callbacks to node was quite simple, I found that almost every OS has old SSL libraries without support for it. That’s why latest node version bundles openssl in it’s deps/ folder. Thanks to Isaac Z. Schlueter, zlib module was added in 0.6.0 version of node.js, and required only a little patching: it was missing stream dictionary support.

With all that working out of the box (only in 0.7.0-pre version of node now) I decided to start developing a completely new stable node-spdy version. You can take a look at what I’ve done so far at https://github.com/indutny/node-spdy . Note that to test it you’ll need to build v0.7.0-pre of node.js (currently at github’s master branch):

  git clone git://github.com/joyent/node.git
  cd node
  [sudo] ./configure
  [sudo] make
  [sudo] make install

I have one running SPDY server running at https://spdy-twitlog.indutny.com/. It’s demonstrating socket.io + express support of node-spdy module.

Features & Roadmap

Implemented:

• Node.js compatible interface (same as for require(‘http’).Server )
• Opening/closing streams
• Fallback to https
• Basic flow management (no WINDOW management so far, that means .pause() and .resume() will just buffer data internally instead of sending WINDOW_UPDATE to client)
• Protocol errors handling
• Stream scheduler (for priority management)
• Push streams

Roadmap:

• WINDOW_UPDATE and INITIAL_WINDOW_SIZE
• Fixing bugs

Blacksmith is a static site generator. It uses weld, jsdom and marked to turn json, markdown and 100% genuine HTML and CSS, on the filesystem, into awesome sites like the one you're looking at right now. Moreover, it includes a command line interface for quickly building, editing and testing blacksmith-built websites. Here's a short list of features:

• Easy to use
• Can be hosted anywhere, since it generates static HTML/CSS
• Write and Edit articles on the file system using github-flavored markdown
• Based on JSDOM and Weld
• Easily create custom themes using plain HTML and CSS ( no micro-templating ! )
• Default theme is a port of scribbish by Jeffrey Allan Hardy
• Ships with a robust node.js static server suitable for production

Blacksmith may remind you a bit of jekyll and octopress, and in fact it is influenced by these projects. However, whereas jekyll is "blog-aware" and uses liquid for templating, blacksmith uses the DOM for its templating needs and can easily generate not just blogs, but documentation sites and more. Plus, you don't need ruby or rake since it's all written in node.

Why Static?

We knew from the start that we wanted our blog platform to generate a static site. Why? Static sites are fast, foolproof, and get the job done. Why dynamically generate a page each time if you only have to do it once?

This might seem counterintuitive, considering all of the real-time, client-side code in Nodejitsu. However, while websockets and hash-based routing fit really well for many problems, others, like our blog, simply don't need it.

Why Weld?

We also knew from the start that we wanted to avoid microtemplates. Hij1nx puts it pretty well, but to reiterate: Templates are 100% HTML and css, and that is awesome.

Given this question, you may ask "why not plates?" Truth be told, we began this project before plates was ready for prime time. There is, however, a good reason not to switch: Having access to a full dom allows for doing such things as using jquery server-side, for the utmost in flexibility.

Try it out!

Just visit blacksmith's github repository, install it and have fun!

In software application development we usually store application state in volatile and non-volatile random access memory. We also use network devices to communicate state between our application and a central provider or server. We store state in things like databases, the file-system and cloud hosting providers. This approach is recommended for most applications, but when we start to think about the implications of building large, distributed, and decentralized applications, we begin to rethink about how these applications are going to store and maintain state.

In some way or another, almost anything can store state. Any device that can be written to, then read from later can be used to store state.

Think of it like this:

There are hundreds of thousands of endpoints on the web right now that you can post some sort of data to, and read it back later. Why not make a database out of that?

#

Basic strategies in building a decentralized and anonymous database

1. No central servers. A central server is a central point of failure.
2. No central authorities. A central authority over control of the database indicates another ( and much worse ) central point of failure.
3. Ability to encrypt and obscure data.
4. Ability to run in any combination of local / cloud / hosted. The database should be able to run locally, on hosted servers, and in the cloud.
5. Ability to easily share and replicate data. The data should be able to easily transfer and replicate between multiple systems and users.

The goals of building a decentralized and anonymous database are fairly straight forward. The data needs to have no central point of failure ( technological or political ). The data needs to be able to be secured. The data needs to be accessible and sharable.

Introducing hnet

hnet is a decentralized and anonymous database built in node.js. It works by spreading small amounts of information across several nodes using a variety of non-traditional storage engines.

http://github.com/hookio/hnet/

What is an hnet node?

An hnet node is anything that we can store a small amount of data which we can eventually read back later. The hnet client communicates with many hnet nodes in order to establish a dataset.

What is an hnet storage engine?

A storage engine can be considering a wrapper for any service on the web that can store data, and read it back later. Storage engines are small and pluggable, so it should be very easy to author new engines. Some examples of engines are: gist, pastebin, imgur, google groups, reddit, twitter, irc, etc... The only real requirement for a storage engine is that it can store state, the rest is up to your imagination.

Why hnet?

hnet was originally built as an easy way to provide distributed and semi-anonymous tables of server ips and ports for hook.io hooks. hnet helps provide hook.io a way of auto-discovery other hooks over a Wide Area Network. Note: hook.io already supports auto-discovery over the Local Area Network via mdns.

How does hnet work?

Many dumb hnet nodes, one smart client.

In order to query hnet, first we connect to a few "top-level" nodes

If you are not maintaining your own hnet, the hnet client will fall-back to a few semi-moderated top level nodes. If you don't feel comfortable having someone else moderate your data, it's trivial to setup your own piece of hnet.

You will notice that hnet uses Iriscouch for many of it's top-level nodes and that most nodes eventually link back to http://hnet.iriscouch.com/public. Iriscouch provides free hosted CouchDB as a service. It allows you to signup for a free CouchDB instance instantly and with minimal registration. IrisCouch was chosen for both it's high quality of service, and the fact the lead developer of the service Jason Smith is an active open-source and open-data advocate.

Note: Our node.js deployment tool jitsu also ships with a nifty command jitsu databases create couch, which will instantly give you a CouchDB instance through IrisCouch.

The hnet protocol is JSON

the hnet protocol supports arbitrary JSON data, and optional JSON-RPC commands

Here is an example of what a JSON fragment returned from an hnet node might look like:

example JSON returned from hnet node:

[
  { "foo": "bar", "tar": "val" },
  { "foo": "boo", "something": ["a","b","c"] },
  { "foo": "bar", "tar": "val" },
]

optional JSON-RPC commands can be embedded

hnet optionally parses these JSON-RPC commands.

[
  { "foo": "boo", "something": ["a","b","c"] },
  { "method": "link", params: [ 
      { "type": "couch", "uri": "http://hnet.iriscouch.com/public/0" }
    ] 
  },
  { "foo": "bar", "tar": "val" }
]

If we look at the array item from the previous JSON fragment, we can see:

{ "method": "link", params: [ { "type": "couch", "uri": "http://hnet.iriscouch.com/public/0"} ] },

The link method is particularly important.

It indicates that we should lazily link this document from a remote dataset into the current position in the item array.

This allows us to create large datasets from many small hnet nodes.

hnet client receives data from many nodes

After hnet is able to query a few top-level nodes, it begins to crawl several other linked hnet nodes and merges the data locally.

Saving data to hnet

hnet could be considered an append-only database. hnet is designed to create new nodes instead of attempting to edit existing nodes. Every time a new hnet node is created, it links back to at least two existing nodes.

Since every new node created links back to at least two other known hnet nodes, all newly created hnet nodes will eventually link back to the first nodes we started querying.

Establishing a TTL ( Time To Live ) for circular link resolution

By default, the hnet client will not attempt to resolve circular JSON-RPC links. You can enable circular linking by specifying the ttl parameter in the hnet constructor.

Ex:

var Hnet = require('../lib/hnet').Hnet;

var hnet = new Hnet({
  ttl: 5000
});

hnet.load();

The ttl parameter indicates that the hnet client will resolve any circular links it encounters, after a delay of 5,000 milliseconds. If a ttl is specified, the hnet client will continue to crawl hnet forever, or until the circular link becomes broken due to specific hnet nodes becoming unavailable.

A quick note on Steganography and Cryptography

hnet exposes simple Hnet.get and Hnet.save methods. It's fairly trivial to perform whichever cryptography you want on the data before you save it, and when you retrieve it. hnet also provides an optional interface for specifying un-encrypted metadata with encrypted data. This approach allows you to use hnet with most existing cryptography standards.

Steganography is the art and science of writing hidden messages in such a way that only the sender and intended recipient are aware of the existence of the message. It can be considered a form of security through obscurity. Many hnet storage engines use some form of steganography. The advantage of using steganography in conjunctions with cryptography, is that messages do not attract attention to themselves. Cryptography can protect the contents of a message, but steganography can obscure the fact that a message even exists.

At Nodejitsu, our node.js hosting platform deals with 1000s of live servers spanned across multiple data-centers. As scale increases, minute statistical probabilities become very real problems. The network is unreliable, disks become unwritable, streams break, unexpected input is unexpected, and entire data-centers can go down.

Think of it like this:

If you have 1,000 servers each individually rated at 99.9% uptime, on average, one of those machines is always failing.

#

So we research...

Fault tolerant system
Fallacies of Distributed Computing
Properties of distributed systems
High availability Percentage calculation

and we research more...

Shared nothing architecture
Byzantine fault tolerance
Paxos

Eventually, we hit the The Halting Problem:

Given a description of a computer program, decide whether the program finishes running or continues to run forever. In 1936, Alan Turing proved that a general algorithm to solve the halting problem for all possible program-input pairs cannot exist.

to summarize all of these links in the context of this blog post...

It's mathematically impossible your distributed application is not going to fuck up. Deal with it.

Basic strategies in building fault tolerant applications in node

1. The operating system's RAM, processes, and file descriptors are relatively cheap. Use them as needed.
2. Use JSON to pass data between nodes. ( see: nssocket or dnode ).
3. Embrace crash-only design. Nodes should be able to disconnect, restart and reconnect as fast as possible with minimal impact.
4. Never assume any one node process will never go down. Assume that any process can go down for any reason at any time.

The last point is particularly important. If your application is unable to recover from a "central" node going down, it should not be considered fault tolerant.

hook.io

hook.io is a distributed input/output framework for node.js. It allows you to seamlessly link together several processes and start sending messages between them. hook.io also provides a rich network of stand-alone hook libraries for adding additional i/o sources. In a sense, hook.io can be described as a next generation enterprise service bus.

• Email List: http://groups.google.com/group/hookio
• Video Lessons: http://youtube.com/maraksquires ( mirror )
• Wiki Pages https://github.com/hookio/hook.io/wiki/_pages
• hook.io for dummies
• Distribute Node.js Apps with hook.io:
• #nodejitsu on irc.freenode.net

Basic Paxos

Paxos is a family of protocols for solving consensus in a network of unreliable processors. Consensus is the process of agreeing on one result among a group of participants. This problem becomes difficult when the participants or their communication medium may experience failures.

As of v0.8.6, hook.io now has the ability to do a very basic consensus among all hooks to ensure recovery if any hooks die.

In previous versions of hook.io, if the "master" or "central" server hook went down, all other hooks would stop communicating.

Now, hook.io is able able to recover from the "central" server hook going down.

Previously, when the hub went missing, none of the spokes knew where to connect anymore.

Building fault-tolerant node.js application with hook.io

hook.io can be used programatically or as stand-alone binary. For simplicity, we will assume in this example that you are using hookio as a binary application, and with all it's default settings.

For more examples of using hook.io programtically check out the examples or test folders.

To install hook.io with npm

 npm install hook.io -g

Now, we are going to start up three instances of hookio in our terminal.

Terminal 1

 hookio

Terminal 2

 hookio

Terminal 3

 hookio

Using hook.io's default auto-discovery functionality the first terminal will take the role of server, and the second and third terminals will connect as clients to the first terminal.

Now, kill the first hook terminal with CTRL-C. The second hook will take over the role of the server. The third hook will now know to connect to the second hook instead of the first.

It might not seem very impressive, but it demonstrates a basic example of how to build fault-tolerant multi-process distributed applications in node.js

Applications built with hook.io now have no central point of failure.

But wait, there's more!!!

mdns

Multicast DNS (mdns) is a way of using DNS programming interfaces, packet formats and operating semantics on a small network where no DNS server is running. The mDNS protocol is used by Apple's Bonjour and Linux Avahi service discovery systems. mdns is an easy way to help networked devices find each other without any prior configuration.

As of v0.8.6, hook.io now has alpha support for zero configuration networking via mdns. This means that if you hookio -m on two machines on the same Local Area Network...they will find each other and instantly start communicating without any configuration!

Computer A

 hookio -m

Computer B

 hookio -m

Now these two computers ( connected over a LAN, with no central DNS server ) will automatically discovery each other and begin to transmit messages.

Think of the possibilities!

Conclusion

With the new mdns and paxos features, hook.io is getting better everyday. The current goal is to stabilize this new functionality in the 0.8.x branch across all platforms.

The next steps will be p2p hook trackers and hosted hook services...

It helps you manage your node.js modules and dependencies.

Previously, we created a npm cheat sheet describing various NPM commands.

Now, here is a package.json cheat sheet for exploring various properties of npm's package.json format.

Enjoy the package.json cheatsheet!

These are Travis CI build status images. We started testing our code at Travis CI and we want to tell the whole world about it!

What is Travis CI?

Travis CI is a continuous integration system. Long story short: continuous integrations systems run your tests. How is it different from running them yourself?

• clean environment - your local development environment can be pretty weird and you know it! People who use your code don't necessarily have the same settings as you have
• easy creation of test environments
• it actually runs the tests - continuous integration system won't say 'this change is so small that it can't mess anything up'
• build logs are publicly visible - transparency is important, especially when it's about something other people rely on
• it's nearly realtime - we often push code and almost instantly get notified about test results

tl;dr:

Why is it important for a CI to be realtime?

People make mistakes. Developers forget to npm update before npm test, keep their temporatory files and so on. You can probably write down a long list of silly mistakes you sometimes make. This can lead to pushing failing code.

Of course, Marak doesn't do that. At least nobody can confirm it.

Few days ago, thanks to Travis, we were able to find a regression in union. It turned out to be a dependency failure, which we were able to fix in approximately 5 minutes.

So, we delivered a product in 5 minutes. Sounds like a heaven for project managers.

OK, so how does this "realtime" thing look like? Many of us spend our time on #nodejitsu IRC channel at Freenode. Thus, a natural thing was to enable IRC notifications for builds.

I can definitely agree with Marak here. Point of this notifications is not only to tell you that you failed. We want people to know that their changes were good!

Versions!

We care about innovating. We care about our code being compatible with newer versions of software (node, npm and so on), but we have to care about people using older versions.

How can Travis help? Travis lets you to set up multiple test environments which target different runtime version. It's really easy to run your tests in node v0.6 and node v0.4 at the same time with just 3 lines in your .travis.yml file (detailed instructions for node.js are available here).

language: node_js
node_js:
  - 0.4
  - 0.6

Build you get looks like that.

Travis is easy!

Travis is a hosted service - you don't need a sysadmin to setup a Jenkins server for you. All you need is .travis.yml file and enabled GitHub post-receive hook (which Travis can do for you!).

Besides Node.js, travis-ci.org offers first class support for several other technologies:

• Clojure
• Erlang
• PHP
• Ruby

with more languages coming in the future. It only takes a couple of minutes to add your project to travis-ci.org, give it a try and if you find it useful, consider supporting the project.

Summary

We're having a great time with Travis. It helped us find a few regressions, bugs and compatibility issues. If you're starting or if you have an open source project, you should add it to Travis. Show people that you care about your test results!

It helps you manage your node.js modules and dependencies.

Here is a quick cheatsheet of all my favorite npm commands:

• Installing npm
• Update npm
• Search for npm packages
• View details of a npm package
• Installing a npm package locally
• Installing a npm package into an application
• Understanding Global versus Local installs in npm
• Global Package Installation
• Uninstalling a package locally
• Uninstalling a package globally
• Installing a specific version of a package
• Cloning a module from Github
• Linking any npm package locally
• Linking local npm packages to multiple applications
• Unlinking a npm package from an application
• Unlinking a npm package from your system
• Create a new npm package
• Creating a new user account on npm
• Publishing a npm package
• Unpublishing a npm package
• Managing owners of packages

Installing npm

 curl http://npmjs.org/install.sh | sh

Update npm

There are several ways you can update npm. I prefer either:

 curl http://npmjs.org/install.sh | sh

      or

 npm install npm -g

Search for npm packages

 npm search hook.io

Protip: Try using http://search.npmjs.org

Double Protip: Know the name of the package you want before you start

View details of a npm package

 npm view hook.io

Installing a npm package locally

For the purpose of this demo, we will use http-server.

http-server is a package we've written which provides an easy to use wrapper around node's core http.Server class. This module makes for a good example, since it's API provides both a CLI binary and a requirable node.js module.

 npm install http-server

This performs a local install of http-server in our current working directory

You may also notice a new node_modules/ folder. You can ignore this for now.

Installing a npm package into an application

  mkdir mynewapp/
  cd mynewapp
  npm install http-server
  touch test.js

 node test.js

 mkdir anotherapp/
 cd anotherapp/
 touch test.js

 var HTTPServer = require('http-server');

 node test.js

node.js:134
        throw e; // process.nextTick error, or 'error' event on first tick
        ^
Error: Cannot find module 'http-server'
    at Function._resolveFilename (module.js:326:11)
    at Function._load (module.js:271:25)
    at require (module.js:355:19)
    at Object.<anonymous> (/Users/maraksquires/dev/nodeapps/anotherapp/test.js:1:80)
    at Module._compile (module.js:411:26)
    at Object..js (module.js:417:10)
    at Module.load (module.js:343:31)
    at Function._load (module.js:302:12)
    at Array.<anonymous> (module.js:430:10)
    at EventEmitter._tickCallback (node.js:126:26)

 cd anotherapp/
 npm install http-server

 npm install http-server -g

 npm install http-server -g

 http-server

cd mynewapp/
npm uninstall http-server

npm uninstall http-server -g

cd mynewapp/
npm install http-server@0.3.0

git clone git://github.com/nodeapps/http-server.git
cd http-server/
npm link

cd http-server/
npm link

mkdir newapp/
cd newapp/
npm link http-server

cd newapp/
npm unlink http-server

cd http-server/
npm unlink

mkdir mypackage/
cd mypackage/
npm init

npm adduser

cd mypackage/
npm publish

npm unpublish http-server

npm owner add marak http-server
npm owner rm marak http-server
npm owner ls http-server

So you're writing a module. Cool! Are you ready to document it? I'm here to help.

I'm no expert on readmes. I have, however, read a lot of readmes (both good and bad), written my fair share of readmes and documentation, and---most importantly---have formed strong opinions on what I should see in a readme.

This post focuses on the "first pass" for your readme. These PROTIPS are applicable, however, to readmes of all ages.

1. Write an Example, Right Now

The very first thing that should be written in a readme is a short example code snippet. This snippet should show what your library does, and as little else as possible. In the ideal case, people should be able to infer from your example how your library works.

For example, here's a stubbed-out readme for a small utility by SubStack:

mkdirp

Like mkdir -p, but in node.js!

Example

pow.js

var mkdirp = require('mkdirp');

mkdirp('/tmp/foo/bar/baz', 0755, function (err) {
    if (err) console.error(err)
    else console.log('pow!')
});

Output pow!

And now /tmp/foo/bar/baz exists, huzzah!

This probably isn't the greatest readme of all time. However, the example is a perfect case study. In four lines, a programmer more or less fluent in javascript can tell that mkdirp:

• Is a function.
• Takes 3 arguments: A file path, some octal permissions and an errorback.
• Probably makes directories.

Add in the mention of "like mkdir -p" and, without any other documentation, we can tell how it works.

Of course, mkdir -p is small enough that you can actually cover 100% of the API in one example. Not all projects are so lucky, but they can still strive to get a common use case across.

Here's another good example of an example, from node-tap. The example in this code saves what would otherwise be a terrible readme, to the point where node-tap has a solid contingent of loyal users despite its readme.

Why?

You want to get people to use your project, right? Supposing this is your goal, you have to understand that your readme is your library's best chance of "selling" your library, and you don't have that much time.

When a developer starts looking at libraries that might solve his or her problem, they're thinking two things:

1. Does this project solve my problem?
2. If so, how?

Chances are they're looking at your project's competition as well, and they're not very patient. You need to get the gist of this information across early enough that your audience pauses to read the rest of your readme.

A good example will tell a developer right away if the module does what they need, and any example will give the developer a taste of your API.

You also get the added benefit of having to write a simple case of your API, giving you the opportunity to "sanity check" your API. Does anything look bulky in your example? Could the API be made more obvious? Now you know.

2. Describe the install procedure.

A project that I think does a really good job of this is hook.io:

Getting Start / Demo

 npm install hook.io-helloworld -g

Now run:

 hookio-helloworld

Spawn up as many as you want. The first one becomes a server, the rest will become clients. Each helloworld hook emits a hello on an interval. Now watch the i/o party go!

Two. Lines. You now know how to get up and going with hook.io!

Node.js projects like hook.io are lucky in this regard, as npm makes most installations for node projects a one-liner. This is ideal, as ease of installation is another selling point for your project.

If you're not lucky enough to have an easy installation procedure, then this section becomes even more important. In the case of hook.io, I may have been able to put 2 and 2 together and went looking for it on npm. For a more complicated yet poorly-documented install procedure, developers will screw it up, at which point they will either complain to you or give up and move on.

Why?

Assuming your example made a sale, the second question a developer is going to ask is, "How do I install this thing?" Make it easy for them.

3. Stub out the API docs.

Now that you have a basic example and installation instructions, you can move on to documenting the basic API.

When I write API docs (and this is admittedly a weak point for me), I like to start with obvious entry points (your module only has one obvious entry point, right?) and work from there.

For example, here's a snippet from some API docs I wrote for union:

union.createServer(options)

The options object is required. Options include:

options.before

options.before is an array of middlewares, which are used to route and serve incoming requests. For instance, in the example, favicon is a middleware which handles requests for /favicon.ico.

Union's request handling is connect-compatible, meaning that all existing connect middlewares should work out-of-the-box with union.

In addition, the response object passed to middlewares listens for a "next" event, which is equivalent to calling next(). Flatiron middlewares are written in this manner, meaning they are not reverse-compatible with connect.

options.after

options.after is an array of stream filters, which are applied after the request handlers in options.before. Stream filters inherit from union.ResponseStream, which implements the Node.js core streams api with a bunch of other goodies.

The advantage to streaming middlewares is that they do not require buffering the entire stream in order to execute their function.

options.limit (optional)

This argument is passed to internal instantiations of union.BufferedStream.

In union, the obvious entry point is union.createServer. In this example, I tried to explain what you do with it, and what all the pieces mean.

Why?

Hopefully the idea of documenting your code isn't completely foreign. That said, you may be tempted to think that an example is "enough". I would suggest that this is true only in very rare cases. Even mkdirp could use a short paragraph explaining the arguments and behavior of the module. Inferrence is great, but shouldn't be completely relied upon. Be straightforward.

4. Tests

Your module may or may not have tests. If it does have tests (and tests are a wonderful idea!), you should describe how to run them. For example, here's a snippet that's very familiar to us at Nodejitsu:

Run Tests

Tests are written in vows and give complete coverage of all APIs and storage engines.

  $ npm test

This is pretty much ideal.

Why?

Like installation instructions, testing directions should be as straightforward as possible. If you're using node and npm, npm test is great because it distills testing into a one-liner.

(Choosing the right testing framework is another discussion entirely. I personally like node-tap).

5. Licensing and Contributors

Finally, tag on your license and contributors.

The content of this isn't actually too important in the context of a readme. For example, I usually just write the following:

License:

MIT/X11.

Why?

It's a good idea to tell other people how they can/may work on your code of course, but it's a secondary consideration. First, your commit logs will show who worked on the project, and arguably way better than your readme can. Second, the license is only important for people that want to hack on your code, and even then most people are willing to accept "It's MIT" when it comes to sending a pull request.

Licensing may become more important in the future, but in the short term you can afford to wait and scale it later.

6. But I had other stuff to say!

Write a blog post about it.

No, seriously.

Why?

A readme isn't always the best place to talk about your project. A readme's scope should really be pretty limited to:

• What is it?
• How do I use it?

Anything else---like, "Why I Wrote This Module" or "Addressing Criticisms Of My Module"---don't really fit. But, blog posts are a great way of getting this information out.

A good example of how a blog post can enhance a readme can be found on Nodejitsu's blog:

Scaling Isomorphic Javascript Code

In this blog post, Charlie Robbins discusses the motivations behind writing Flatiron, Nodejitsu's collective idea of what a web framework should look like.

For those interested in design patterns, it's a very good read. In this post, Charlie distills the meaning of MVC, how different existing frameworks implement it, and why (in his opinion) existing tools aren't enough. If that lights your fire, then I highly recommend this article.

That said, for the user that just wants to write a webapp and doesn't really care about which kind of MVC-ish paradigm they're using, such explanations in a readme will only get in the way.

TL;DR

If nothing else, write an example.

Successful open source libraries and initiatives require the same kind of attention to branding that many think is reserved for profit focused products. In other words, Open Source Doesn't Just Market Itself. Lets explore how discoverability, time to start (i.e. how long it takes to use what you've built), community, and some good old fashioned hustling can help build Open Source brands.

Overview

tl;dr? This is the place for you:

• "Don't Make Me Think": Understand your users' motivations.
• Divorce your Ego: Understand your motivations.
• Build a Community: Setting and managing user expectations.
• Open Source at Nodejitsu: A quick overview of Nodejitsu's Open Source projects and initiatives.
• Conclusion: Wrap it up.

"Don't Make Me Think"

A clear concise explanation of your Open Source offering is the best way to get users interested in what you're developing. Zach Holman from Github put it eloquently in his article Open Source Doesn't Just Market Itsef:

My favorite marketing of all time is documentation. We all loathe sleazy marketing, and by definition I like to think documentation can't be sleazy because it solves a real purpose: teaching everyone about the project.

Take a step back and think about discoverability and time to start from a user's point of view: they somehow stumbled upon your project. They probably don't know you or how awesome you are. They are looking for a solution to a problem that your project may or may not solve. The quicker you help them make this decision the better.

Documentation comes in all flavors, each of which serves a different purpose. Every type of documentation increases the momentum and reduces the perceived risk of your project so the more the better. Just like a startup, building momentum is a key metric to success: "Build enough momentum and they will come."

• README: This is Open Source MVP. Without a README (preferrably Markdown on Github), at least half the users that discover your project will simply move on. If you have a lot of Open Source projects be consistent. Use an outline; we do at Nodejitsu.
• Examples: If your library has several use cases, then it is absolutely critical to include example code and briefly discuss it in your README.
• Project Website: Even better. A custom project website allows you to create a true brand for your project; a logo, a catchy tag-line, whatever. Be sure to link back to Github or elsewhere.
• Blog Posts and Tutorials: This is the long tail of Open Source branding. Each blog post or descriptive tutorial you write will gather more momentum and lower the perceived risk of your Open Source offering.

Beyond documentation remember that this is Open Source: people will read your code. If you haven't finished part of your API or there is work left to be done be clear about it and expect feedback. Or, if you've decided on a particular way to write your code (such as promises or a particular coding style), realize many users will be immediately dismissive. This is particularly important in Javascript:

If your code uses a compiled fibers library, or something that transforms the code, then it's much harder for me to figure out what the heck is going on. The introspection is gone.

Divorce Your Ego

To me, the emotional investment in creating Open Source is best summed up by Isaac Schlueter:

It's like having children. Seems like a lot of pointless busy-work, and then you do it, probably without meaning or wanting to, and then suddenly fall so in love, you can't understand why everyone doesn't go as nuts over this little food-to-poop converter as you do. If you're not careful, you'll end up writing 3 or 4 more.

Divorcing yourself from your ego and these emotions is hard. You have probably put in countless hours of your own time just because you thought it was cool. But it is crucial to Open Source success: your users probably don't want your life story about why you created your library, or how it is going to solve a problem they don't have.

They also don't want to get an attitude when they open up an issue or reach out to you over Twitter, Email, or your Mailing list. A user may not completely understand the problem you're solving or is just plain misusing your API. An open-minded and understanding attitude is your greatest ally. This doesn't mean you have to divorce yourself from your opinions, just stay objective and never make it personal.

Build a Community

Beyond having a simple and concise explanation of what your Open Source offering is it is important to create a place where questions and FAQs can be easily found and responded to. Think of this as "beyond documentation."

When working with these community building resources it is crucial to realize your first interaction with a user may be your last. And, if you provide a great experience for your Open Source users they will become your community's greatest asset: responding to Github issues, helping new users, writing Open Source which expands on your own, authoring a critical mass of blog posts, and more. At Nodejitsu most of our new hires come out of this group of users who respond to common GitHub issues and mailing list threads.

So bury your ego and be as friendly as you can. That's not to say you have to do your users' work for them; just set and manage user expectations. We try to respond to every issue within 24 hours; ideally within 5 minutes.

• GitHub issues: You probably already use GitHub issues for your project. Are you checking them every day? Do you have a FAQ tag for commonly opened issues?
• Mailing list: Usually a Google group. This is a great place for feature discussions and commonly asked questions. Just look at the nodejs, expressjs, or hookio mailing lists.
• Twitter: Like it or not, Twitter is a very common place for developers to ask questions. Be prepared for some extra noise in your @ mentions. If you prefer not to use Twitter, be understanding and explain that to users. Also: retweets are one of your best marketing sources.
• Email: Personally I prefer not to answer questions about Open Source projects over email; I ask that a Github issue be opened. If you're feeling generous, however, it is a great way to build a rapport with one user.

When working with these community resources there are some basic expectations which my co-founder, Marak Squires, outlined in his JSConf.eu talk: How to be an Open Source Gangster. The tagline may sound silly, but as the Chief Evangelist at Nodejitsu Marak speaks from experience:

• Be available: If it takes a user a week to get a response to a pull-request, issue, or email, they probably aren't even a user anymore.
• Persistence, Tenacity, Follow-through: Stay focused. Every day is a new challenge. Like most developers Open Source is probably your side-project. Be explicit with yourself about how much time you are going to commit. If you go over that limit then find a way to share the workload with a contributor. Just giving up will undo any hard work you've put in.
• Ignoring Distractions: Don't get distracted by things that don't contribute to your core Open Source offering. "Let the things that don't matter truly slide."
• Recruitment: Realize that you are not a one man army and engage your contributors vigorously. In the long run, they are the greatest asset to your community.

Open Source at Nodejitsu

Last week at Nodejitsu we announced Flatiron. In addition to the libraries that we announced we took this opportunity to reorganize our personal Github repositories into organizations. We realized that as our Open Source offering grew organizing repositories across multiple Github organizations was the simplest path to high discoverability. This decision is not for everyone, and probably not the correct decision for most individual developers. Just like starting a company, these qualities help your Github organization tell a cohesive story. A good Open Source Github organization does a few things:

• Cohesive motivation: What problem is your Github organization solving? You don't get a README on your organization page so this should be inferred from the repositories or the homepage URL.
• Critical mass of repositories: Enough repositories to convince a passive user that the organization is legitimate. This can be extremely difficult if your company is new to Open Source; you may be better off contributing to an existing organization. Releasing several projects at once is a good way to avoid having less than a critical mass.
• Critical mass of members: Enough users to convince a passive user that the organization is legitimate. Githits.me is a great way to determine this; it ranks organizations by the sum of the Github followers of all users. This is not a perfect metric by any means, but it can help you determine this.

At Nodejitsu we have decided to delineate our core Open Source offering into several organizations each of which we feel tells a different story:

• Nodejitsu: Core Open Source libraries from our public Platform-as-a-Service, Nodejitsu.com including jitsu, haibu, and forever.
• Flatiron: Collection of decoupled Open Source libraries for building tools and web applications in Node.js.
• Hook.io: A full featured I/O framework for Node.js.
• Node Apps: A collection of sample applications designed to expedite time-to-start with Node.js on Nodejitsu.com.
• Node Training: Our training materials (when they're finished) along with tutorial-style projects which we use in our training sessions.

Moving and Renaming Personal Repositories

You've probably heard this saying about Computer Science before:

There are only two hard problems in Computer Science: cache invalidation and naming things. -- Phil Karlton

As the quantity of your Open Source libraries increases and they potentially move around to different organizations the first name you picked may no longer be appropriate. Such is the case for the routing component of Flatiron, which today we are renaming from sugarskull to director. We felt that the name director was more in-line with the Flatiron brand.

If you do decide to move personal Open Source repositories to an organization Github will not perform an 301 redirects so the best process is below. It is not ideal by any means, but it is the best Github can do today. If you are concerned about SEO think about this this way: you shouldn't be concerned with the 100 followers you have today; be concerned with the 1,000 followers you want in a year.

1. Move the repository to your organization from your personal Github
2. Immediately fork the moved repository back to your personal Github and add a redirection notice.

Conclusion

Hopefully not all of this is new to you; most of it is a distillation of the common wisdom that most developers follow today. A lot of this can be thought of as Open Source in the age of Github.

Prior to GitHub, word about open source projects travelled primarily through word-of-mouth and messenger bicyclists. The largest open source project had three users and two developers. I think it involved a lot of twine and glitter.

• Make your users first time as easy as possible: Minimize perceived risk to maximize momentum.
• Be an Open Source Gangster: Every day I'm Open Sourcin'.
• Engage your community and recruit actively: Github, Twitter, Mailing list(s); whatever works.
• Create a Github organization for large projects: Remember to tell a cohesive story.

This is the motivation behind flatiron. At it's core flatiron is two things:

An initiative to build a collection of decoupled tools with the same standard of quality and performance that you would expect from anything built by Nodejitsu.

A full-stack web application development framework which packages these tools together to make isomorphic and stream-based application development easier.

This article will introduce you to the components that make-up flatiron as well as how the core flatiron project.

A Sum of its Parts

At Nodejitsu, we have and continue to believe in a simple mantra: build the best tools and the best systems will follow. Tools and their distribution among members of an organization evolve organically overtime. To put it another way: if you use Node.js you are probably already using a component of flatiron. We have simply moved these tools to the Flatiron Github Organization

• api-easy: Fluent (i.e. chainable) syntax for generating vows tests against RESTful APIs.
• complete: Custom command line tab completion for nodejs CLI apps.
• nconf: Hierarchical node.js configuration with files, environment variables, command-line arguments, and atomic object merging.
• neuron: The simplest possible event driven job manager, FIFO queue, and 'task based cache' in node.js.
• node-prompt: A beautiful command-line prompt for node.js.
• sugarskull: A depth-first search based routing library for the browser and node.js.
• winston: A multi-transport async logging library for Node.js.

In addition to migrating existing libraries we have been working on a number of new libraries to round out these tools.

• broadway: Lightweight feature reflection and application composition for node.js and the browser.
• flatiron: An unobtrusive full-stack web framework for node.js and the browser.
• plates: Unobtrusive, thin and fast anti-templating
• revalidator: A cross-browser / node.js validator used by resourceful and flatiron.
• resourceful: A resource-oriented ODM with a high-level API for routing and validation.
• union: A hybrid buffered / streaming middleware kernel backwards compatible with connect.
• utile: A drop-in replacement for util with some additional advantageous functions.

A lot of effort has been made to ensure that each component stands on it's own and that each Github repository clearly explains the purpose, use case, and stability of the library. There are a few particularly interesting highlights among these libraries that we'll explore below.

A quick tour of Flatiron

Application composition

Thanks to Javascript you never have to worry about types again (mostly), but that doesn't mean that we can't learn something from design patterns more prevalent is statically typed languages. Namely: inversion of control and dependency injection. Don't be scared! Promise you won't have to register anything, ever.

Broadway exposes a simple "plugin" API which allows the application developer to extend the top-level application object easily. This model translates well between the browser and the server because developers can write different plugins to suite the needs of the different environments. Broadway includes several plugins by default which can be overridden if you don't agree with the particular module or library choice we've made:

Figure 1: app.use() in Broadway

Doesn't remind you of dependency injection? It shouldn't! Because of the loosely typed nature of Javascript it isn't necessary to bring along all of the heavyweight type registration APIs that are so common in most dependency injection frameworks. Instead, plugins must be responsible and properly implement three methods

• .attach(options): This synchronous method is called immediately on app.use(somePlugin).
• .init(done): This asynchronous method is called once when the application initializes and before it starts.
• .detach(): This synchronous method is called on app.unuse(somePlugin).

Figure 2: Plugin Lifecycle in Broadway

All of these functions are evaluated in the scope of the app (i.e. this inside of a plugin is the app itself). Lets take a look at a quick sample from broadway itself:

exports.attach = function (options) {
  //
  // Here setting `this.config` will expose the `.config` 
  // property on the application itself
  //
  options = options  || {};
  this.config = new nconf.Provider(options);
};

exports.init = function (done) {
  //
  // Later on we can initialize the configuration 
  // we attached earlier.
  //
  this.config.load(function (err) {
    return err ? done(err) : done();
  });
};

Ok. So what? Application extensibility allows us to reuse more components across different kinds of applications while writing less code. In almost every express application, you will see a server.js file which is essentially wiring up the same libraries to the same methods. If you write a couple of small applications it might not seem like a problem, but when building a product like Nodejitsu it simply isn't DRY enough.

Streaming or not streaming? No problem!

One of the current problems in node.js core is that there is no good way to buffer a stream when not closing the underlying file descriptor. Don't worry though; the geniuses behind node.js are on it and already have a solution on the drawing board. In the meantime though, something needs to fill the gap and that is union and sugarskull.

The problem is that 90% of the time you actually want to buffer a small amount of request data (< 10k) into memory then parse it and attach it to the http.ServerRequest object. Think: a POST with some JSON attached to it. Occasionally though, you may want to take advantage of a stream. Think: a large file upload to Amazon S3. In this case, buffering is cancer and .pipe() is king.

The solution is two-fold:

• Pipe the concrete http.ServerRequest to a BufferedStream implementation by default.
• Attach metadata to routes indicating if the route requires a stream or not. If a stream is not required, attempt to parse the entire body before continuing to the Router.

First, just a little about me... in July I joined Nodejitsu as COO. What's that mean? For the purposes of this discussion, not a whole lot. What's more important is that I'm an application developer and I'm new to Node. In this post, I'll discuss a lot of the things that attracted me to Node.

What is Node?

Node (aka Node.js, NodeJS) is Javascript. It's the same thing that's been running in your browser for years. Now it's better, faster, and stronger. Oh, and now it's everywhere. Node is what allows you to bring Javascript, a language traditionally run in the browser, to the server. And why does Node matter? I'm glad you asked that.

Javascript isn't going anywhere

There's no getting around having Javascript in your app. Javascript is THE language for making dynamic web interfaces bar none. If you're thinking, "Hey, but isn't that JQuery?" You're right. But JQuery is Javascript. Promise. And if you're thinking, "Come on, but what about Flash?" Well... what about it?

Javascript is going to be in your application somewhere. Node allows you to standardize on that from the top to the bottom. Some applications have Javascript on the front-end on top of a Python web application that speaks to APIs served by something more performant like Scala or Erlang. That's a lot of different competencies to hire for and to maintain and when something breaks, not everyone is going to be conversant with everything. Deferring specialization is a huge benefit of Node.

Wow, is it fast!

Node is an order of magnitude faster than Ruby and Python. This translates roughly to say that it's very easily the most responsive under load of any of the most popular high level languages (Javascript, Ruby, Python). Node and Javascript benefit from all the work that Google, Apple and Microsoft have poured into making their browsers more performant as well as more efficient to run on less resources (such as a netbook or iPad). Lucky us.

Realtime

Node uses an asynchronous event-driven model. This means that it can make more efficient use of resources to do more things at one time and is an excellent fit for the architectures of many Internet applications. This model also makes these applications easier, and more intuitive for engineers to create.

Facilitation of engineer AND code versatility

Javascript everywhere means that while there are still going to be developers that are more familiar with particular front or back end APIs or environments, it's a lot easier for the team to pitch in where help is needed. In a similar fashion, Javascript everywhere facilitates the reuse of code all through your stack which is a good thing as it cuts down on redundancy and helps maintainability of your application.

Yeah, we think it's pretty cool too

This is of course, just a taste. Here are some other excellent resources to get you started.

• Convincing the Boss
• Understanding Node.js
• "Data-Intensive Real-Time" Applications

Also, check us out on the Twitter (@Nodejitsu), read some of the other articles on our blog, or maybe attend some of our future training sessions. We also write a tremendous amount of open-source code that you can utilize in your applications.

This approach has been acceptable for other languages such as Java, Ruby, and Python it is simply not good enough for Node.js for one reason: Javascript is now an isomorphic language. By isomorphic we mean that any given line of code (with notable exceptions) can execute both on the client and the server. On the surface this seemingly innocuous property creates a number of challenges that are not solved by current MVC-based patterns. This article will explore some of these existing patterns, how both their implementation and concerns vary across languages and environments, and how they are not good enough for a truly isomorphic Javascript codebase. In conclusion, we will explore a new pattern: Resource-View-Presenter.

At a glance

Design patterns are the bread and butter of application development. They encapsulate and outline the concerns of the application and the environment of the in which it exists. Between the browser and the server these concerns can vary widely:

• Is the view ephemeral (e.g. on the server) or long-lived (e.g. in the browser)?
• Is the view reusable across different use-cases or scenarios?
• Should the view be annotated with application-specific tags or markup?
• Where should the bulk of the business logic reside? (in the model? in the controller?)
• How is the application state persisted or accessed?

Lets explore some of the existing patterns and how they answer these questions:

• Model-View-Controller
• Model2
• Model-View Presenter and Model-View-ViewModel
• Modern Javascript Implementations
• Real-time Implications
• tl;dr? Introducing Resource-View-Presenter
• Conclusion

MVC

Figure 1: Model-View-Controller

The traditional Model-View-Controller pattern assumes a long-lived view and a swappable Controller. For example, in a given view you may have different Controller logic depending on who is logged in. At a high-level no decision is made about how the view is rendered (i.e. what templating engine is used).

Given the assumption of a long-lived view and that user interaction is by definition coming into the view, traditional MVC is a very useful pattern for front-end development. As we will explore later, a slightly modified version of this pattern is in fact what Backbone.js uses.

Model2

Figure 2: Model2 Model-View-Controller

Don't be scared if you've never heard of Model2 before; it is a design pattern that dates back to 1999 when it was coined in an article by Govind Seshadri: Understanding JavaServer Pages Model 2 architecture. It can be argued that Model2 does not necessitate a fully-realized MVC pattern, but most modern implementations (such as Ruby on Rails) do formalize it in that way.

Figure 3: Rails Model2 Model-View-Controller

The common wisdom is that in Model2-like frameworks such as Ruby on Rails one should have "fat models and thin controllers". This is not the case for every application, but in practice it is generally what this author has seen. In the traditional MVC Controllers tend to be heavier (i.e. more business logic) due to their need to listen and react to input from the view so this decision seems to add up.

Given the stateless nature of HTTP the Model2 View is truly ephemeral: no state maintained in the view itself between requests. In most server-side frameworks any application state is stored via Session Cookies. This makes the decision of a one-way communication between the Controller and View very logical, but also unsuitable for any front-end development.

MVP and MVVM

Both the Model-View-Presenter and Model-View-ViewModel patterns are similar to the traditional MVC pattern with a few key differences:

• View does not have a direct reference to the Model
• Presenter (or ViewModel) has a reference to the view and updates it based on changes to the model

The MVP pattern is discussed at length by Martin Fowler (here, and here) and is generally discussed in the context of two implementations:

• Passive View: Design the view to be as naive as possible with the all but the absolute necessary presentation and business logic contained in the Presenter.
• Supervising Controller: Used in systems in which the declarative view can be expanded to encompass simple logic. In this incantation the Presenter should only take over when the declarative logic in the view cannot meet the system requirements.

Figure 4: Model-View-Presenter

Figure 5: Model-View-ViewModel

MVP and MVVM are almost indistinguishable with one key exception: MVVM assumes that changes in the ViewModel will be reflected in the view by a robust data-binding engine. Niraj Bhatt outlined the difference eloquently in his MVC vs. MVP vs. MVVM article: "For instance if View had a property IsChecked and Presenter was setting it in classic MVP, in MVVM ViewModel will have that IsChecked Property which View will sync up with."

The positive end of the MVP and MVVM is that the Presenter (or ViewModel) is easier to unit-test because the state of the View is by definition contained in methods invoked by (MVP) or properties set on (MVVM) by the Presenter or ViewModel respectively.

For front-end development either of these patterns are perfectly acceptable choices. A routing layer can pass control to the appropriate Presenter (or ViewModel) which in-turn can update and respond to a persistent view in the browser. With some massaging either could also be re-worked for use on the server for one reason: there is no connection between the model and the view. This allows for an ephemeral view which is rendered by a given Presenter (or ViewModel). As we will see later on, this modified pattern can be truly isomorphic.

Modern Javascript Implementations

The design patterns presented above have many modern implementations today:

• Backbone.js
• Batman.js
• Angularjs
• JavascriptMVC
• Sammy

These frameworks are generally used for creating Single-Page Applications (although more traditional AJAX is also possible). User Interaction within a single-page application comes in two distinct flavors:

• OnHashChanged or pushState events: Occur when the URL in the browser changes. For example: navigating to http://myapp.local/#/some-page
• DOM events: Occur when a user makes a specific interaction with the current DOM. For example: clicking on an anchor tag.

Lets consider some of the patterns and architectures used. If you're interested in further reading check out Peter Michaux's article on MVC Architecture for Javascript Applications.

Backbone

Backbone.js is one of the most popular client-side development frameworks available today. At it's core it is an implementation of the traditional Model-View-Controller pattern. When we examine it in more detail, however, we can see that there are some deviations from the traditional MVC pattern explored earlier.

Figure 6: Backbone Model-View-Controller

In the diagram above we have separated the control-flow represented by OnHashChanged and DOM events to illustrate the separate entry points offered by Backbone. By illustrating this nuance it is clear that Backbone has one important difference from traditional MVC: the View manipulates the Model. When we examine the Backbone Todo Sample, we can see this is clearly the accepted common wisdom:

window.AppView = Backbone.View.extend({
  // ....

  //
  // When a TodoView is instantiated it is passed an
  // instance of the Todo model.
  //
  addOne: function(todo) {
    var view = new TodoView({model: todo});
    this.$("#todo-list").append(view.render().el);
  }
  
  // ....
});

window.TodoView = Backbone.View.extend({
  // ....
  
  //
  // Here the TodoView updates the state of the Todo model.
  // This breaks from the traditional MVC in which the view
  // only listens for changes on the Model.
  //
  toggleDone: function() {
    this.model.toggle();
  }
  
  // ....
});

The decision to break from the traditional MVC pattern gives large Backbone applications all similar feel: thin controllers and models combined with heavy views. These business logic-heavy views are essentially Presenters when looked at objectively. In a large Backbone codebase you should expect a large number of views composited together via a DOM framework like jQuery or zepto.

There is nothing wrong with breaking from the traditional MVC pattern; in the context of front-end development having a reference to the Model from the View removes a lot of bookkeeping logic from the application. It does, however, make it a pattern that is not isomorphic.

Batman

Batman.js is a new Javascript framework that was discussed at JSConf 2011. Although the entities within Batman are Model, View, and Controller. The presence of a strong data binding engine and pure HTML views suggests that it is actually an implementation of the Model-View-ViewModel.

Figure 7: Batman Model-View-ViewModel

Having not worked with Batman much it is difficult to say with confidence what a large Batman codebase looks like. That said: the emphasis on the binding engine and thin views suggest that business logic will end up spread between the Controllers and Models within a given application.

As with Backbone, Batman breaks from the traditional Model-View-ViewModel pattern in that the Model communicates directly with the View and the ViewModel (i.e. Controller) does not manipulate the View directly. In addition, because of the references between the Model and the View it is not easily reusable as a server-side pattern. It is, however, more easily massaged into a server-side pattern if a Composite or Adapter pattern is implemented in the Model layer to render a static view and respond to real-time requests.

Real-time implications

Among all of the developer buzz one topic continues to be at the forefront: realtime web applications. Nodejitsu is an official sponsor of the upcoming Keeping it Realtime conference in Portland; check it out! So how do some of the patterns we've examined stack up for realtime support (such as WebSockets)?

• Model-View-Controller (Yes): Model(s) can listen for real-time events and update the view appropriately.
• Model2 (No): The Ephemeral View concept is baked into the Model2 pattern. That is: Controller(s) do not listen for events from the Model.
• Model-View-Presenter (Yes): Model(s) can listen for real-time events, which will propagate to the Presenter(s) who will update the view appropriately.
• Model-View-ViewModel (Yes): Model(s) can listen for real-time events, which will propagate to the ViewModel(s) who will update the view appropriately.

The acceptability of MVC, MVP, and MVVM patterns make Backbone.js or Batman.js workable realtime frameworks for front-end developments. On the server-side this fact no longer holds true because as we have shown: traditional MVC, MVP, and MVVM patterns will not work for static views due to the tight coupling between the View and the Model.

Enter the Resource-View-Presenter pattern.

Introducing Resource-View-Presenter

As we have shown: the MVC, MVP, and MVVM patterns will not work on both the client and the server. At the crux of Resource-View-Presenter is the realization that no pattern will work flawlessly both on the client and the server without some modification. As was suggested when examining the MVP and MVVM patterns, it is possible to make these patterns truly isomorphic due to their decoupled Model and View layers.

The main decisions which Resource-View-Presenter are:

• Decouple the Model and View
• Recognize and plan for the differences between the Client and Server.
• Expect a thin View, and a heavy Presenter and Resource.
• Prefer business logic in the Resource over the Presenter.
• Allow for both ephemeral (i.e. static server-side views) and persistent Views (i.e. the DOM).
• Prefer a Presenter over a ViewModel to preserve the purity of the Markup (i.e. HTML).
• Assume a persistent Presenter and Model.

Although these decision may seem as though they are random, each has a specific purpose:

• By decoupling the Model and the View we allow for both ephemeral and persistent Views
• A thin View is consistent with modern, logic-less templating engines such as weld and mustache.
• Having a Presenter instead of a ViewModel is consistent with designer-friendly templating such as weld.
• Assuming a persistent Presenter and Model both on the Client and Server enables encapsulation of real-time functionality in Presenters on both sides.

Lets dig deeper. On the Client-side Resource-View-Presenter resembles a traditional MVP pattern. The choice to rename the Model to Resource is heavily influenced by our assumption to prefer business logic in the Resource over the Presenter. This makes the Resource in RVP more similar to the heavy Models found in Model2, not traditional MVP the nomenclature denotes that. When implementing RVP the litmus test for what logic belongs in the Presenter is two fold: any Presentation-specific logic which is too heavy for a "thin" View or any business logic which utilizes global application state.

As with Backbone and Batman, Client-side RVP implementations should support both OnHashChange / pushState routing and DOM events.

Figure 8: Client-side Resource-View-Presenter

On the Server-side Resource-View-Presenter is almost identical with one notable exception: the View is ephemeral and does not pass calls or have a reference to the Presenter. In fact, when using RVP for JSON-based web-services the view is practically non-existent; it is just a call to JSON.stringify().

Figure 9: Server-side Resource-View-Presenter

At first glance, Server-side RVP may seem similar to Model2 MVC, but the persistent-nature of both the Presenter and Model to support real-time events makes this similarity is only skin deep. RVP does this by listening for events and changes on the model which may be backed by a real-time data source such as Redis PubSub or CouchDB changes.

The real-time support is of particular note because it enables the application developer to focus on the business logic, not the underlying network transport. This may seem trivial, but examine the pattern (or lack there-of) offered by Express and Socket.io (the most popular node.js Framework and Realtime IO libraries respectively) and it becomes clear that it is not.

Figure 10: Express and Socket.io

This is not meant to be an attack on Express or Socket.io; they both are explicit in what they offer and they both do it exceptionally well. A higher-level design pattern is simply not a concern.

Conclusion

Writing large applications is hard. Encapsulating and reusing components of those applications across the Client and the Server is even harder. After considering this analysis it is hopefully clear how utilizing the Resource-View-Presenter pattern within your own applications will help to make this easier.

Don't have the time to implement it yourself? No problem! This article is the result of a lot of careful thought and hard work born from the massive open-source undertaking Nodejitsu has been apart of from the beginning. Our philosophy remains the same: build the best tools and the best systems will follow.

And follow they will. If you haven't guessed already, we have an epic open-source release coming up that will implement the Resource-View-Presenter pattern. So if you have any feedback, let us know!. And stay close, I will be announcing the release at SWDC 2011 in Stockholm, Sweden on November 9th..

A few weeks ago we released http://docs.nodejitsu.com, just in time for Node Knockout! docs.nodejitsu.com is a collection of community written tutorials and how-to articles for node.js. There are over fifty articles so far, from the basics all the way up to more difficult topics. We like to think of it as our unofficial, non-competing entry into NKO. We also, of course, hoped that the actual competitors would find it useful.

Like our software, all of the articles are hosted on github. Contributors can fork the project on github, edit the files in markdown (or add new ones), and submit a pull request. We are actively working on adding more features to the site, such as a tighter integration to http://search.npmjs.org

Aside from Nodejitsu's West Coast Team, we've already had a number of other contributors including: Marco Rogers., Dean Landolt, Adam Malcontenti-Wilson, and Pavan Sunkara. Thank you all! We still need more articles, so if you are interested in becoming a community contributor (and possibly getting compensated for your work), please contact us at docs@nodejitsu.com.

We hope that the Node.js community finds http://docs.nodejitsu.com a handy resource and, through community contributions, it becomes the de facto source for practical Node.js documentation.

The problem here is that the extremely popular middleware style of web application development is often incompatible. If you use a body decoder middleware, for example, every request will need to be buffered in full before the body can be properly decoded. This increases memory usage and reduces performance, compared to simply relaying un-altered requests and responses to other destinations - it all has to be waited for, read into memory, possibly altered, then sent again, adding to both the time and resource cost of each request.

More importantly, though, if any subsequent middleware in your middleware chain tries to listen for 'data' or 'end' events, or otherwise treat the current HTTP connection as a stream, it just won't work, as a buffered request is no longer a stream - it is now just a data object.

Despite these concerns, 'middlewares' are an easy concept to grasp, easy to use, and are likely to remain popular. There are some gains to be had, as well - middlewares are a simple way to add new features to your application stack. If you find yourself wishing for a certain feature, the ability to use or write a middleware to add that feature can save everyone a lot of time, compared to the amount of time a pull request can take to be reviewed and merged. It also allows for conflicting features to exist more easily - finding out that two middlewares won't work well together is a much smaller problem than dealing with successive breaking API changes in a library.

Due to these facts, and the number of requests we've received from users who want to proxy to that style of application, we have added middleware support to Node HTTP Proxy.

Here's a basic example of how to use a middleware - specifically, the connect-gzip middleware. This would gzip-compress every file you serve. It can be a useful trick, at times, when applied to specific file types - replace the /.*/ with /png/, for example.

var http = require('http'),
    httpProxy = require('http-proxy'),
    gzip = require('connect-gzip');

httpProxy.createServer(
  gzip.gzip({ matchType: /.*/ }),
  9000, 'localhost'
).listen(8000);

http.createServer(function (req, res) {
  res.writeHead(200, { 'Content-Type': 'text/plain' });
  res.write('request successfully proxied to: ' + req.url + '\n' + JSON.stringify(req.headers, true, 2));
  res.end();
}).listen(9000);

The basic idea is that when you call httpProxy.createServer, you can now pass in any middlewares as arguments, in the order in which you'd like them to be used. If one of your middlewares doesn't set your proxy destination for you, pass in the port and host you want to proxy to at the end of the list, as shown above.

Here is another simple middleware usage example, this one using an example middleware by Nodejitsu's Mad-Scientist-in-Residence, Dominic Tarr, called proxy-by-url. Available on npm, this middleware provides a sugar syntax for defining both forward and reverse proxy destinations in the same block, and routing to them based on URL matching.

httpProxy.createServer(
  require('proxy-by-url')({
  '/github': { port: 80, host: 'github.com' },
  '/nodejitsu': { port: 80, host: 'nodejitsu.com' },
  '/localstuff': { port: 8080, host: 'localhost' } 
  })
).listen(8000);

Now let's take a look at how to make a middleware - even if you don't plan to make one of your own, the structure is very straightforward, and understanding it will help you better understand how middlewares work. This middleware is for very simple logging - it will write the headers of each request to a logfile.

var fs = require('fs'),
    http = require('http'),
    httpProxy = require('../lib/node-http-proxy');


module.exports = function (logging) {
  // Code here is run when the middleware is initially loaded.
  var logFile = fs.createWriteStream('./requests.log');

  return function (request, response, next) {
    // Code here is run on each request.
    if (logging) {
      logFile.write(JSON.stringify(request.headers, true, 2));
    }
    next();
  }
}

This middleware can now be used like the others:

httpProxy.createServer(
  require('./example-middleware')(true),
  8000, 'localhost'
).listen(9000)

In other Node HTTP Proxy news, we at Nodejitsu have also been fortunate enough, in the last several weeks, to receive a number of high-quality patches and bug fixes from isaacs. As a result, the resilience and stability of Node HTTP Proxy have been significantly enhanced.

So the next time you run into isaacs, say thanks! (With beer.)

You can also find a mirror of these videos @ https://github.com/hookio/tutorials

Enjoy!

Exploring the Core API

1. #12 A quick lesson in fault tolerance Part 1
2. #13 A quick lesson in fault tolerance Part 2
3. #14 Spawning child hooks Part 1
4. #15 Spawning child hooks Part 2
5. #16 Full-featured child hook spawning with Haibu and Haibu-Carapace
6. #17 Using Hook.io to Install NPM Packages Anywhere

Note: I apologize for small text size, I will try to improve this on the next batch of screencasts

hook.io is a radical departure from how web applications are currently being built. hook.io itself is very easy to use, but the concepts behind it can be hard to understand from only reading the documentation and unit tests.

For this reason, I have created eleven short video tutorials for explaining the basic usage and functionality of hook.io

You can also find a mirror of these videos @ https://github.com/hookio/tutorials

UPDATE: 6 more videos on Spawning hook.io Hooks

DOUBLE UPDATE: Due to popular demand, I have converted and uploaded all videos to youtube. You can subscribe here: http://www.youtube.com/maraksquires

Enjoy!

1. The Basics Part 1
2. The Basics Part 2
3. The Basics Part 3
4. IRC Hook Configuration Part 1
5. IRC Hook Configuration Part 2
6. 3 Minute IRC Bot
7. Hook.io Libraries
8. HTTP Webhook Server
9. Socket.io Bridge to Browser
10. Emit events from CouchDB _changes feed
11. Extending an existing application

Note: I apologize for the low quality audio and strange video overlays. I've fixed this problem in the newer batches of videos

First, install our CLI tool from NPM

npm install jitsu -g

Now lets fire up our Node Knockout wizard!

cd /path/to/mynewapp
jitsu knockout

That's it!

From here you will be prompted for your Knockout Secret Code, and then jitsu will generate and deploy a simple helloworld application for you that will register with Node Knockout's website.

Additional Deploys

Need to update your app? Add new package.json dependencies? Websockets? Custom C++ addons? No problem!

For new deploys just type:

jitsu deploy

It's super simple!

Questions? Problems?

We've got the full Nodejitsu team on duty ready and waiting to help you. The best place to get support is in #nodejitsu on irc.freenode.net. You can also email support@nodejitsu.com

Warning: These numbers should not be used for anything meaningful, and are possibly wrong. The data is pulled from http://search.npmjs.org and uses SubStack's nifty tool: npmtop

Hourly tallies can be found here: http://npmtop.nodejitsu.com

  Generated: Fri Aug 19 2011 08:41:16 GMT+0000 (UTC) 

  rank   percent   packages   author
  ----   -------   --------   ------
     1    1.85 %     62.00    James Halliday
     1    1.85 %     62.00    TJ Holowaychuk
     3    1.59 %     53.00    AJ ONeal
     4    1.05 %     35.00    Dominic Tarr
     5    0.90 %     30.00    Isaac Z. Schlueter
     6    0.87 %     29.00    Marak Squires
     6    0.87 %     29.00    Nickolay Platonov
     8    0.72 %     24.00    Charlie Robbins
     8    0.72 %     24.00    Felix Geisendorfer
    10    0.66 %     22.00    George Stagas

Tied for 1st - James Halliday and TJ Holowaychuk

James Halliday, aka SubStack, author of libraries such as: dnode, browserify, burrito, etc. Co-founder of Browserling. All around Robot.

TJ Holowaychuk, aka visionmedia, author of Express, Connect, Expresso, Jade, etc. Engineer at Learnboost, an all-around nice guy, and he has a badass ferret named Tobi.

I'll keep the profiles short for these two, they are open-source champions.

3rd - AJ Oneal

AJ is a Software Consultant living in Provo Utah. He's done a lot of work re-packaging and re-bundling client-side modules ( such as jQuery ) to work server-side and to work with the Ender.js framework. He also has some pretty neat looking libraries like Futures. When I asked him for a bios he told me he was single, looking for ladies, and his favorite animal is a Trogdor. I think AJ might be my new internet best friend.

4th - Dominic Tarr

Resident Mad Scientist at Nodejitsu. Dominic enjoys writing test frameworks for testing test frameworks, this requires a lot of support libraries. There is only one project of his that needs to be pointed out, and it's testbed. In usual fashion, there is no documentation and the site is currently offline. I think I might have to schedule an open-source intervention for Dominic to help get his project back online. Please send words of encouragement and pull requests to madscience@nodejitsu.com

5th - Isaac Z. Schlueter

Friend, author of NPM, works for Joyent, node.js core contributor, dislikes semi-colons. Including several support libraries for NPM, Isaac is also the author of libraries such as: node-supervisor, nave, and several TAP ( Test Anything Protocol ) related libraries. Isaac's projects will help you improve your node.js development workflow. If you are intending to write large amounts of node code, you should read through his project list, it will make you a better programmer.

Tied for 6th - Marak Squires and Nickolay Platonov

I am Marak Squires, huzaah. I'm a co-founder of Nodejitsu and the creator of the hook.io framework. I have a fairly eclectic collection of projects including: say.js, Faker.js, JSONLoops, colors.js, webservice.js, translate.js, and node-raps. I also assist in maintaining several Nodejitsu projects. The majority of my new projects are hook.io hooks for helping create a Standard Hook Library.

Nickolay Platonov, aka SamuraiJack, is a JavaScript expert from Russia and the author of the Joose framework. Many of his projects are related to supporting Joose and ExtJS to build full-stack javascript solutions with rich user interfaces. If you are interested in building enterprise class applications using Joose and ExtJS, I don't think you will find someone with more expertise than Nickolay.

Tied for 8th - Charlie Robbins and Felix Geisendorfer

Charlie Robbins ( aka indexzero ) is a co-founder at Nodejitsu, maker of hot sauce, and author of libraries such as: Forever, Winston, Nconf. He also leads development for Nodejitsu on such projects as: node-http-proxy, haibu, and jitsu. Charlie's projects are among the highest quality projects available, and I do not say this lightly. I would recommend not only incorporating Charlie's work into your production stack, but actually diving into his projects to better understand best practices for Git workflow, Github project management, and node.js software development.

Felix is a very tall and friendly man from Germany. He is the co-founder of debuggable, the first node.js consulting company. Felix has made numerous contributions to node's core and is best known for his libraries such as: node-formidable and node-mysql . Felix also has a lot of smaller libraries for performing very specific and useful things. You should take your time to scan through Felix's projects, there might be a gem in there that will save you several hours of time.

Felix, like Charlie, tends to only release high-quality projects. It's a safe bet that a project from either of these two will be useful, and work as intended.

10th - George Stagas

George is a developer from Greece and an early Nodejitsu beta-tester. In addition to writing a lot of utility libraries, George also has some neat projects like Maga, a lightweight framework for developing multiplayer physics-based games, and Maptail, a visual client for mapping logs of ip address to a web map based on geographic location in real-time.

Getting Daily Tallies

Since people do enjoy tracking meaningless numbers, I've setup a quick application on Nodejitsu which wraps Substacks npmtop and outputs the results every hour. It's all open-source, check it out!

http://npmtop.nodejitsu.com

Remembering what's important

These numbers are meaningless, it's the data and people behind the numbers that really matter. Through exploring this data we can get a better understanding of how developers contribute to node.js. Most developers don't have the time to maintain a lot of projects, but they do create and maintain projects that are of very high quality and actively maintained. Remember not to assume more === better.

At Nodejitsu we believe in sharing knowledge. We believe that open source software fosters innovation, relationships and offers in return something better than the sum of its parts; a continuous growth cycle. If we tell you our good ideas, you'll make them better. Now go back two steps and repeat this process.

Passionate open source projects, we understand that they should be both engineered and crafted. A patient attention to detail and refinement in our products and platform remains our methodology. These characteristics: open, crafted, and connected have evolved into the current incantation of the Nodejitsu brand.

Build a product that works.

We have a platform that has become mature and ready for public consumption. One which is accessible to everyone, yet able to exceed the requirements of the enterprise. A platform tailored to the concerns of the developer. One with a pricing model that reflects usage rather than tenancy.

Design simple.

Simple designs are highly palatable. Less is more. Signal to noise ratios are critical. Carrying over these ideas from software engineering we reached out to Tyrus Smalley, friend, industry eminent, and creative force. Going through the motions of developing a visual language was a positive experience for a lot of reasons. The most important reason being that three technical co-founders coding 20 hours a day don't spend a lot of time thinking about presentation.

Design relevant.

Our new design has lots of wide open space. Space for you to think about the content. We wanted to describe a self-healing, distributed scaling intelligence in visual terms. Draw relevant diagrams that actually help illustrate these concepts. Answer questions such as What is infrastructure opacity?, What is distributed scaling? Some of these matters culminated in this animated diagram.

Make a foundation.

At first we started using a bonsai to depict an organic form that could be shaped and crafted. We retained the concept, but simplified the form. The simplification of the branches took shape as a cloud. And the rest of the tree was depicted as a node growing outward into the cloud with more nodes.

The word mark has mature modern forms with playful ligatures that keep things from getting too serious. We added a primary color (the blue color you can see it here on this blog or on the home page). We let the dark grey take the place of the secondary color, this keeps things looking subtle.

Painting a moving picture

These forms offer us a visual foundation to build on, and we'll continue to capitalize on them. But this is a fast moving technology. Node.js is in development, our product continues to evolve. So as focused as we are on the technology side, we wont forget to keep throwing paint on it and make sure it looks as good as it works.

It was only after I had begun to heavily use Github and Node.js that I realized these mystical beasts were actually real people, who work very hard on their projects, and generally strive for quality feedback so they can improve on their work.

Unfortunately, as I now help maintain and contribute to over 100 open-source projects, I've seen a lot of inappropriate behavior from developers trying to seek support. I think much of this behavior is unintentional. There are not a lot of guidelines for developers to follow who are new to open-source. In this article, I will dive into many of the common issues open-source project maintainers encounter and I will help developers better understand how they can file for a refund on the open-source project they are using.

There are no refunds in open-source

Most people understand this, sadly, some do not. If you are going to take an aggressive or angry approach when filing a request support, you are better off finding a new project. You need to understand that the person you are reaching out to has probably spent 100s of hours working on this project, for free. They do not owe you anything. The maintainers are extending a courtesy by giving away their work for free and then making themselves available to support it.

The point is, you should try and be nice when filing for support. The maintainer of the project has literally no obligation to help you.

Wrong place, wrong time

Want to really annoy an open-source maintainer? Then ignore any communication channels they have setup for support. If they setup a support wiki, send them a personal email. If they setup a Github Issues tracker, email the wrong mailing list about your question. If they setup an IRC room for support, send them direct messages over Twitter. If they setup a mailing list, post your issue to an unrelated forum.

The point is, take the time to actually research if the maintainer has setup a specific channel for support. If they have, use this channel. If you are uncertain, just reach out to the maintainer and ask where the best place to post support issues are.

Maintainers are not psychics

Here is a list of things I do not know: Who you are, what project you are asking about, what software you are using, what operating system you are on, what versions of the software you are using, what your stack traces look like, what you had for breakfast, and what the code looks like that caused your issue.

If you open up a support issue without providing this information, don't be surprised when the only response you get is asking for more information. Also, I probably don't care what you had for breakfast.

Read the f*cking documentation

If the author has taken the time to write documentation, you should read it. I don't write documentation for myself, I write it for other people. If the information you need isn't covered in the documentation, let the author know. They probably will want to add this information so other people don't ask the same question.

If there is no documentation to read, you should ask the maintainer why. If you want to help contribute to the documentation, you might find the maintainer of the project wants to be your new best-friend. If the maintainer is unwilling to help document the project, or assist you in documenting it, you should probably find a new project to use.

Run the f*cking tests

If the author has taken the time to write unit tests for the project, you REALLY need to confirm the unit tests work before trying to get support. Unit tests help provide a common way to test and isolate issues in a project. If you are having issues and one of your tests is also failing, it will be much easier to diagnosis the problem.

If you file a support issue without confirming you ran the tests, don't be surprised when the response you get is asking you to confirm the tests are working.

Read the f*cking code

Okay, this is a bit harsh. I don't expect everyone to read every single line of the code for a project they are trying to use, that isn't very reasonable. What I do see though, is that a lot developers have a mental barrier to actually opening up the source code for the project they are trying to use. They will read the documentation, run the tests, use the example code, but when they are faced with a problem that could be solved through a one or two line change in the source code, they shut down completely.

The point is that you shouldn't be afraid to jump into the source code. Even if you don't fully understand the source code, in many cases you should be able to isolate your issue to a specific block. If you can reference this block ( or line numbers ) when opening up your support request, it will help the author better understand your problem.

If you don't like it, fix it

Does something about a project really bother you? Does the author not care as much about it as you do? Do you find yourself angrily yelling at the nice people at the bus station due to your hatred of that inefficient algorithm you found? Here is a wonderful idea: Fix it.

Developers want to improve their project. If you find an issue, bring it up. If it's a valid concern, the author will probably want to have it fixed. In many cases, the author will consider it a valid issue, but simply not have the personal time or need to address it immediately. This is where open-source is great. Just fork the project and fix it your damn self.

If you fix something, let the author know

Did you actually fix a known issue? Let the author know about it. Not everyone has time to adhere to the specific coding styles for a project, so if you can't do a full blown pull-request, there is NOTHING wrong with opening a pull-request that only has the intention of showing the author how you solved the problem.

On many occasions, I've opened up requests for support in the form of a Github pull request. This way, I am telling the author: I have found a potential problem with your library, here is how I fixed it for my circumstance, here is the code I used for reference. You get extra internet points if you open the pull request with: "I don't expect this pull request to get merged, but I wanted to you show you what I did".

Don't get upset, rejection is normal

This is a follow up from the last section. Don't assume that because you opened up a pull request, that the author will accept it. There are many reasons that a maintainer might choose to not merge in your specific patch, many of which have nothing to do with you. If your patch isn't accepted, try to assume it's for a valid technical reason and not because the author hates you.

If you feel there has been an oversight, it's okay to not give up. As long as you are being logical and open to other people's views, you will find that you might learn something new, or even teach something to the maintainer.

If you don't like a project, don't use it

You would think this is obvious, but it's not. If you don't like a project and have no inclination to fix it, just don't use it. If you feel you are being "forced" to use a project, perhaps you should re-evaluate the life decisions which brought you to this point.

Conclusion

There are no refunds for open-source. If you go in with the attitude that you are owed something, you will be thoroughly disappointed. On behalf of all open-source developers and project maintainers, I ask you try and be polite the next time you ask for support. Try to remember that there is a real human being on the other side of the screen, and they actually want to help you.

Need to get up to speed on Node? No problem. Free Training
In the run up to Knockout, we are also going to be teaching two free training sessions on Saturday, Aug 20th and Sunday, Aug 21st. Our friends at Pivotal Labs NYC have been kind enough to provide space for these events.

Not on a Node Knockout team? Also no problem.
Even if you don't have a team to compete with, sign up and show up. There will certainly be adhoc teams developing on the scene. Bottom line, you don't have to bring a party to enjoy the party.

Thanks to all of those that made this possible!
We would like to thank all of those who are supporting our efforts in New York including 10gen, Amazon, Done, Foursquare, Mashery, Rackspace and others.

Schedule
Friday, Aug 26th
6pm-7pm: Nodejitsu / General Assembly Happy Hour 7pm-8pm: Welcome Presentations
8pm-1am: Game on! Let the coding begin. Saturday, Aug 27th 9am-1am: Doors at General Assembly are open!
Sunday, Aug 28th
7pm-9pm: Optional demos, first come, first serve 9am-8pm: Last sprint for finishing up projects.

The distillation of a philosophy to coding in a particular language (or environment) takes time, trial and error, and above all a group of extremely talented and dedicated professionals. Most of the ideas central to this philosophy are not new to the practice of Software Engineering or the study of Computer Science, but merely the selection of choices that appear often in the community with motivation from choices made in node.js core. With best efforts, sources and references will be supplied to support the arguments made here.

You first may ask "has there been enough time?" Node.js is still very much in its infancy as put so eloquently by Isaac Schlueter (author of npm) earlier this year at NodeConf. That is, Node.js has barely been around for two years. It may be therefore premature to write this, but there has the immense growth seen in both node.js core and in the ecosystem surrounding it. Nodejitsu has been intimately involved in this growth and thus, in the opinion of the author, enough time has passed for a few granules of truth to trickle through.

There surely has been a large amount of trial and error: as of writing this article there are over different 3200 modules on npm (the node package manager) many of which are failed attempts at one problem or another. And last but not least there is of course an immensely talented group of professionals working both on node.js code itself and on important modules which grow the node.js ecosystem.

Looking back over the evolution of the past two years, there is truly one central tenet to the development of node.js core set out by Ryan Dahl (the creator of node.js) that has defined the node.js philosophy this article is attempting to capture. That is, node.js core should be kept as small as possible. This is very clear on both the node.js and node.js development mailing lists where frequently feature additions are not included because they simply "do not belong in core."

The focus on a very small, performant, cross-platform compatible core was the impetus for the vast number of modules available through npm today. This is because many commonly requested features:

• "Framework": templating, routing, cookies, etc.
• Asynchronous flow control
• Pluggable Middlewares

were not available in node.js core. The need for these features from developers eager to use node.js allowed talented and prolific module authors to emerge and fill the void. This continually expanding and changing ecosystem of modules is in many ways the defining factor around the node.js philosophy: experimentation with small kernels of functionality which rely on loosely coupled components.

Experimentation

We are no strangers to mad science at Nodejitsu, but at it's core why is experimentation so core to the node.js community? With no large existing codebase to solve common web application development problems combined with Javascripts minimalistic language design many modules emerged to solve the same problem(s). Revisiting established problems caused many developers to ask "are the accepted approaches to solving the problem at hand necessarily the best?" This focus on experimentation has shown clear benefits: socket.io, dnode, the recent redesign of npm are some clear examples. I think it was put best by Mikeal Rogers (organizer of NodeConf): "if you think node.js needs less modules you're $%#@!^$ crazy."

Small kernels of Functionality

One of the benefits of having a well established package manager like npm (especially now that it is not possible to have version conflicts in node.js modules), is that a module developer can really focus on developing the best tool for an individual task. In other words, why package two modules together if you can simply break them apart into two kernels of functionality which are codependent? Exactly. There is no reason to do this unless your source code distribution (e.g. package manager) is deficient in someway.

Another way to think about this is to consider a phrase that I use often to describe node.js core: "close to the metal." For example, there are not a lot of abstractions or conventions between the developer and the raw HTTP stream. Each step along the way is a small kernel of functionality (the net module, the http module, etc.). This low-level functionality clearly has impacted the approach of the community at large.

Loosely coupled components

Coupling is poison to large codebases. It allows developers to hide improperly thought out abstractions and bad APIs by strictly defining and limiting the consumer to a fixed surface area. The idea of "loosely coupling" your components is that you can easily switch one out for another. This is something that was borrowed by Rails from Merb when they merged into Rails 3.0 and something that I am very glad is engrained into the approach taken by node.js developers. Early experiments on tight coupling such as Connect have not been repeated in other approaches and (in the case of Connect) considered "harmful" by their original authors; in this case Tim Caswell.

The argument in favor of loosely coupled components becoming the de-facto standard in node.js is (again) precipitated by npm. Without a strong, easy-to-use package manager with well defined semantics, module authors would be incentivized against decoupling components and releasing them separately.

The Road Ahead

Node.js is still not yet 1.0, yet it would be surprising for the core tenets of the philosophy to change. It is a philosophy that has led to important early-stage design choices within node.js, npm, and high-profile modules that continue to make node.js the technology to watch. It is this philosophy that results in the vibrant ecosystem, continued flow of interesting projects and modules, and intelligent debate around core functionality

However, if you talk to Node.JS users you may hear something along the lines of, "Javascript is actually pretty nice for writing command line apps." As it turns out, node.js is a great platform for writing command line applications, and with npm they are as easy to install---nay, easier---than similar programs written in ruby or python and installed with rubygems or pip. In addition, node.js has all sorts of helpful libraries for writing command line apps, such as prompt, optimist, and cliff, that make writing command line interfaces easy!

Here, I hope to showcase a few of the command line apps written in node.js. Many of them are still written with web development in mind. They are all written in javascript, and they are all used from the command line.

forever

Forever, written by Nodejitsu's Charlie Robbins, is a process supervisor written in node.js. More important, however, is that forever is robust and simple to use.

There are many process supervision frameworks out there. The problem with a lot of them is that they're often written with system init scenarios in mind and have their own DSLs. Other times, they're not written in a particularly user-friendly manner, or they aren't very robust. Sometimes, all of these things are true. Because of this (and other problems), myself and others before me have written their own ad-hoc process supervisors, using some unholy combination of nohup, screen and/or bash.

Forever is a much better solution. In fact, forever might be the best solution. It's certainly one of the best I've seen.

For an example, suppose you're a grad student (as I was in a past life) and you have a simulation you need to run that, due to buggy software, might as well look like this:

#!/usr/bin/env node

console.log("=================Super Number Cruncher 5000=================");
console.log("--Running important simulations for your thesis since now!--");

var i = 0;
setInterval(function() {
    if (Math.random() < 0.05) {
        throw Error('FLAGRANT ERROR');
    } else {
        //Generate some output data
        console.log((i++)+','+(273.15+60*Math.random()));
    }
}, 500);

The output of this program looks something like this:

josh@pidgey:~/broken_science_simulation$ ./simulation 
==================Super Number Cruncher 5000==================
---Running important simulations for your thesis since now!---
0,274.32085393052546
1,273.85998188573865
2,332.435783926025
3,312.9851095430553
4,323.98653633948413
5,298.41442882139233
6,314.28931980449704
7,291.39460802134124
8,297.300358219631
9,309.1636345820501
10,281.5037131268531
11,292.7776571927592
12,332.8452686598524
13,275.4756304487586

/home/josh/broken_science_simulation/simulation:9
        throw Error('FLAGRANT ERROR');
              ^
Error: FLAGRANT ERROR
    at Error (unknown source)
    at Timer.callback (/home/josh/broken_science_simulation/simulation:9:15)

What? Flagrant error?

(Of course, given that it's academia, it's probably written in java and MATLAB instead, but you get the idea. Also, ignore the implications of buggy science software. It's better if you don't think about it too much.)

Now, supposing that we can't track down this error because it's buried inside a proprietary software suite, we can just use forever to make it restart every time it crashes:

josh@pidgey:~/broken_science_simulation$ forever -o output.log -e err.log simulation
warn: Forever detected script exited with code: 1
warn: Forever restarting script for 1 time
warn: Forever detected script exited with code: 1
warn: Forever restarting script for 2 time
warn: Forever detected script exited with code: 1

This command puts your simulation's output into output.log, and the errors into err.log. As you can see, Forever restarts your process like a boss. Here's some example text from output.log:

51,275.67675554249433
52,323.125968520157
53,299.9657142587006
54,285.97948576379565
==================Super Number Cruncher 5000==================
---Running important simulations for your thesis since now!---
0,317.5622826308012
1,311.84217803701756
2,330.1117456337437
3,331.6873355587944
==================Super Number Cruncher 5000==================
---Running important simulations for your thesis since now!---
0,298.9083861099556
1,288.8923557434231
2,300.0416149344295
3,312.97375301420686

and here's some example text from err.log:

Error: FLAGRANT ERROR
    at Error (unknown source)
    at Timer.callback (/home/josh/broken_science_simulation/simulation:9:15)

/home/josh/broken_science_simulation/simulation:9
        throw Error('FLAGRANT ERROR');
              ^
Error: FLAGRANT ERROR
    at Error (unknown source)
    at Timer.callback (/home/josh/broken_science_simulation/simulation:9:15)

/home/josh/broken_science_simulation/simulation:9
        throw Error('FLAGRANT ERROR');
              ^

Clearly, errors are still a problem. However, with forever you can keep the bad news from your advisor. Just restart the process, and rewrite it to serialize output and start where it left off!

• Author: Charlie Robbins
• Web site: https://github.com/indexzero/forever
• Install: [sudo] npm install -g forever

http-server

http-server, written by Nodejitsu's Marak Squires, is a zero-configuration http-server. This means that all you have to do to serve the files in a folder is to run http-server from the command line, in that folder, without having to set any options. It's as easy as file serving gets!

For example, here's the content of a pure client-side web page that I wrote, and normally serve using github pages:

josh@pidgey:~/virtual-window$ ls
favicon.ico  frame.png  index.html  jquery.js  README.md  wall.jpg  window.css

Of course, the smart thing to do is to test the project locally before pushing changes to github. Plus, github pages is mildly annoying to work with anyway, right? With http-server, testing it from my computer is trivial:

josh@pidgey:~/virtual-window/$ http-server
Starting up http-server, serving . on port: 8080
http-server successfully started: localhost:8080
Hit CTRL-C to stop the server

Now all I have to do is go to localhost:8080, and there it is! Just like it is on github pages, without having to do any configuration at all.

http-server is also nice for quickly sharing material from my shell at a moment's notice, and for (legally, mind you) sharing games at LAN parties (for those of us that have time for such things).

• Author: Marak Squires
• Web site: https://github.com/nodejitsu/http-server/
• Install: [sudo] npm install -g http-server

browserify

You may have heard of browserify already. Browserify began life as a node.js module for making client-side javascript more awesome by:

• Giving you a node.js-compatible, commonjs-style module system for your client-side code, helping to keep everything clean and organized.
• Allowing you to grab node.js packages from npm and use them client-side, not only enabling simple code reuse, but also allowing one to run code intended for the server in the browser in a "it's so crazy it just might work" fashion. For example, the author of browserify has used it to run pure javascript stack traces in the browser.

However, not all has been peachy in the world of browserify. Many people complained about having to run this "heavyweight" solution, requiring a node server to be running in order to serve these bundles instead of, say, bundling the javascript in a "build" step and serving the results with Apache. Other solutions along a similar vein, such as ender.js, allow you to do this, so why not Browserify?

Well, the author listened, and browserify recently gained a command line interface.

Here's a quick example: Suppose I would like to bundle the hashish library for manipulating some objects on the client side, and that I would also like to include some es5-shim action with the shimify browserify middleware, just in case Mom is insisting on using IE7. Here it is:

josh@pidgey:~$ browserify --require hashish --plugin shimify -o bundle.js

Like that, assuming traverse and shimify are both installed, browserify bundles traverse with es5-shim into a file called bundle.js. Awesome!

• Author: James Halliday
• Web site: http://github.com/substack/node-browserify/
• Install: [sudo] npm install -g browserify

uglifyjs

Suppose you built the bundle from the browserify example, and you checked its size:

josh@pidgey:~$ wc -c bundle.js
59390 bundle.js

60 kilobytes?!

Okay, that's not too bad, but certainly we can do better, right? After all, the source from that output isn't minified or anything. For example, here's a short snippet from bundle.js:

// shallow copy
Hash.copy = function (ref) {
    var hash = { __proto__ : ref.__proto__ };
    Object.keys(ref).forEach(function (key) {
        hash[key] = ref[key];
    });
    return hash;
};

Goodness, that is way too readable!

Programmatically, browserify supports many more features than it does at the command line, including using minifiers such as uglifyjs as source filters. Uglifyjs is a high-quality, speedy javascript minifier that's a favorite amongst node.js developers.

Lucky for us, uglifyjs comes with a command line application! Check it out:

josh@pidgey:~$ browserify --require hashish --plugin shimify | uglifyjs | wc -c
21243

That's much better.

• Author: Mihai Bazon
• Web site: https://github.com/mishoo/UglifyJS
• Install: [sudo] npm install -g uglify-js

ngist

Ngist is a command line github gist tool, similar to Chris Wanstrath's gist from ruby. Its use is pretty similar.

Here's a really simple example of its use, after the initial git configuration step:

josh@pidgey:~$ npm search " cli " > /tmp/cli.npm.txt
npm info it worked if it ends with ok
npm info using npm@1.0.18
npm info using node@v0.4.9
npm info ok
josh@pidgey:~$ ngist /tmp/cli.npm.txt
Gist: https://gist.github.com/1115238
josh@pidgey:~$ 

Here, I searched npm for all modules with the word "cli" in their descriptions, then used ngist to gist the output. It worked great, as you can see for yourself.

Ngist can also use the contents of the clipboard, post multiple files at once (or a combination of files and clipboard contents), allow you to set the gist description, and even use someone else's github username and token.

Finally, ngist isn't only available as a command line app; you can also use it programmatically! This seems to be a common pattern in node.js cli apps, and it's one I really like.

• Author: Jacob Chapel
• Web site: https://github.com/chapel/ngist
• Install: [sudo] npm install -g ngist

jshint

jshint is a linting tool for javascript. While a fork of the more well-known jslint, its stated goals are to be more flexible and less opinionated than Doug Crockford's tool. The project, in addition to the web-based linter, also has a handy-dandy node-based cli tool for linting your projects!

For example, I decided to run it against a project of mine to see what would happen:

josh@pidgey:~$ jshint bf.js
bf.js/bf.js: line 16, col 22, Bad line breaking before ','.
bf.js/bf.js: line 21, col 21, Bad line breaking before ','.
bf.js/bf.js: line 24, col 43, Bad line breaking before '?'.
bf.js/bf.js: line 28, col 21, Bad line breaking before ','.
bf.js/bf.js: line 31, col 43, Bad line breaking before '?'.
bf.js/bf.js: line 35, col 21, Bad line breaking before ','.
bf.js/bf.js: line 40, col 21, Bad line breaking before ','.
bf.js/bf.js: line 47, col 21, Bad line breaking before ','.
bf.js/bf.js: line 57, col 21, Bad line breaking before ','.
bf.js/bf.js: line 82, col 22, Unnecessary semicolon.
bf.js/bf.js: line 91, col 21, Bad line breaking before ','.
bf.js/bf.js: line 92, col 21, Bad line breaking before ','.
bf.js/bf.js: line 93, col 19, Bad line breaking before ','.
bf.js/bf.js: line 96, col 2, Missing semicolon.
bf.js/test/tests.js: line 8, col 2, Missing semicolon.

15 errors
josh@pidgey:~$ 

Oh my! It seems that jshint does not like how I break my lines or place my semicolons! Because, whatever, I do what I want, I'll just dump the following into ~/.jshintrc so I can continue to write code that makes people want to stab their eyes out:

{
  "asi": true,
  "laxbreak": true
}

This tells jshint to allow automatic semicolon insertion and "lax" linebreaks. Now, when I run jshint:

josh@pidgey:~$ jshint bf.js
bf.js/bf.js: line 82, col 22, Unnecessary semicolon.

1 error

There, that's not so bad! The offending code on line 82 is for loop shenanigans I picked up from doing too much code golf, and I should probably rewrite it. Thanks, jshint!

• Author: Brent Lintner (node package)
• Web site: http://github.com/jshint/node-jshint/
• Install: [sudo] npm install -g jshint

Wrapping up

Forever, http-server, browserify, uglify, ngist and jshint are just a few of the command line applications written in node.js and available on npm! They are easy to install, and are crazy useful. In another blog post, I will discuss how to build your own command line applications with node.js, using options parsers like optimist, command line formatting tools like colors and cliff, and prompt, an easy to use module for prompting the command line for input. Stay tuned!

Nodejitsu is very happy to announce the latest stable version of hook.io. This release marks a combined effort from the Nodejitsu team working across six time-zones and four countries. But what does this mean for the application developer and what is hook.io anyway? Hook.io is a full-featured I/O framework for node.js that enables a simple way to distribute your application across multiple node.js processes using the new EventEmitter2 API and leveraging the power of Crash-only software.

In a nutshell, hook.io allows you to split your node.js program into multiple worker processes, that will be automatically restarted upon any user-generated faults, allowing you to increase performance and fault-tolerance. Interested in learning more? Read on!

Wait a second, EventEmitter2?

No, you didn't read it wrong; Nodejitsu has been working on EventEmitter2 for the last couple of months and there has been positive feedback about bringing it into node.js core. EventEmitter2 is a node.js event emitter implementation with namespaces, wildcards, TTL and browser support. Even if it is not brought into node.js core, it will continued to have the full support and backing of Nodejitsu; now and forever.

Lets take a look at how EventEmitter2 works:

Now that we've had the time to build up our infrastructure, we need you to come in and start applying the UNIX philosophy to our systems. Your role will be to "own" our servers and work with the entire team to make sure our services always stay up. We are currently expanding our platform to become a utility provider agnostic system, and we'll need your expert systems programming knowledge to help facilitate integrating with cloud computing providers ( such as: Rackspace, AWS, Joyent, etc...) and setting up redundant dedicated environments across the globe.

Requirements:

• Expert Level Knowledge of Linux Server Administration in High Availability Production Systems
• Expert Knowledge in Shell Scripting, C, Assembly, OSI Model, Systems / Networking Security
• High aptitude for self-management
• Availability to report on-site in New York City or San Francisco

Preferable:

• Dislikes JavaScript
• Prefers talking to our 1000s of servers instead of people
• Has read every single BOFH story

Compensation

• 100-120k per year depending on experience
• Equity in Nodejitsu ( vested over time )
• Health Benefits

Interested candidates can apply by emailing hiring@nodejitsu.com

We do Mad Science.

First there is the Science part, a Mad Science project has to be a bold idea about how some big problem might be solved. Then there is the Mad part. This is the way that once you've thought of your mad science idea, you can't forget it. It niggles away at you, and whenever you encounter your problem again you remember it, thinking, this problem could be solved with some Mad Science.

The only option you have left is to try and build it.

One mad science project associated with nodejitsu is Marak's Hook.io. Hook.io is a generic implementation of "web events". Events are happening everywhere on the internet, all the time. But how do you listen to them? And what amazing applications could you build if you could easily connect all these different events together in an organized way?

Marak did a talk on hook.io at JsConf 2010. since then hook.io has been on the back burner as nodejitsu has been ramping up, but it has recently reemerged with a new approach, based on a marriage between dnode (substack's RPC protocol), socket.io (cross browser websocket support) and EventEmitter2 (another nodejitsu Mad Science project which is a reworking of node's native EventEmitter, but with proper namespaced events)

I also have a mad science project, Meta-Modular. Meta-Modular is based on the idea that version numbers are wrong, and are not the way your dependencies should be specified. Instead, you should declare your dependencies by specifying what tests a dependency should pass. Then, in theory, any implementation of a dependency that passes the test should work in that slot. However, implementing a system like this needs not just a way to dynamically inject dependencies into a test run, but also to analyze large amounts of multi-dimentional data, to see what combinations of dependencies work with what tests.

Imagine the time that could be saved if there was a service that could tell you what forks passed a particular test. And imagine what could then be done with that time, and that knowledge!

As you explore a mad science idea you inevitably encounter difficulties. Success usually feels like it is a month away, and that the new approach it the one, but you never know.

To illustrate this problem I like to use the example of the three parallel rivers in Yunnan province in China. In South West China the Yangtze, Mekong, and Salween rivers, three of the largest rivers in Asia, emerge from the Tibetan Plateau and flow parallel through three gorges between four mountain ranges. Suppose you are the plucky adventurer at one side of this, and you want to know whats on the other side...surely it is a lush fertile plain. So you climb the first mountain range, and what do you find? A gorge, and another mountain range. Surely the plain is on the other side of that, so you cross the river and the next mountain range. By now many members of your party have died of exposure or drowned in dangerous river crossings. You get to the top of this range...only to see another gorge, and another mountain range. This is when you start to ask yourself, just how many gorges are there?

Should you give up? Or push on into the unknown?

For the truly heroic adventurer, this is not even a question.

So far, Marak has attacked the evented web problem three times in node.js, with two other attempts using Python's Twisted and even Coldfusion. I've tried meta-modular in Java, Ruby, and now Node.js. Currently it manifests itself as testbedjs.org which is an open-source community based continuous integration system.

There are many Mad Scientists working in Node.js. Infact, Node is a mad science project. Ryan has been working on the idea of a tinyclouds framework for years. Ultimately, the idea is many lightweight asynchronous evented servers, which communicate by message passing. Before Node got popular, Ry tried out many solutions using Ruby, C, Haskell, Lua, Nginx, whatever he thought would be the best way to solve his problem. Check out some of ry's older work at http://tinyclouds.org/libebb/ and https://github.com/ry/ebb/.

It seems Ry struck gold when he switched to v8 and JavaScript, and suddenly many other mad scientists saw the potential and started working with Node.js. We see people like Isaacs, Tim Caswell, Guillermo Rauch, Matt Ranney, Paul Querna, James Halliday, and Jeremy Ashkenas all gravitating towards Node.

Ryan's hard work and Mad Science has enabled other Mad Scientists to expand on their own visions. Just imagine what the next generation of Mad Science will yield. I predict that hook.io will make Ruby on Rails look like PHP and the concepts behind Meta-Modular will make Github style code hosting look like SourceForge.

The only question now is, what is YOUR mad science project?

Right off the bat on the very first day, I got assigned to modifying the blog code. If you go to the blog homepage and look at the very awesome RSS feed button on the right, that was my very first thing I did for nodejitsu. It took a few hours to do the very simple two line change, but in the process I learned Wheat, Haml and more about Github and I got the change pushed to production the very same day!

Later, I was set on bug fixing duty for haibu. Haibu can be described as the "node.js application server" at nodejitsu. Haibu's job is to spawn up the applications, make sure that they stay up, and clean up after the applications terminate. We run thousands of Haibu servers and communicate with them through simple RESTful requests. There was a bug related to an unclean exit where Haibu would get killed, but the children would never get cleaned up. Charlie gave me a few pointers on the relevant files and the general structure of the code and then left the rest up to me. This is where I learned about better structuring node.js programs. I came from a fairly dominate background of php as well as being a young self-taught programmer, so I was used to a very adhoc organization technique, so aside from fixing an important production issue, I also got to see and work on properly structured code.

At nodejitsu, through the database api, users are able to provision various no-sql databases from places like RedisToGo, CouchIris, and MongoHQ. Originally there were three code bases to provision a database from each of these services. However, my task was to integrate these three code bases together into one library that abstracted away the different libraries. After the refactor, I implemented the library in the core stack, so it could be accessible through the api interface. This allowed me to allow jitsu to interface with these providers to easily provision and deprovision databases. In case you have not heard of it, Jitsu is the command line tool to manage your apps on the nodejitsu cloud. It also has well written code that uses several tricks ( and CLI libraries like: prompt, cliff, nconf, colors, optimist ) to make writing new commands super easy. These changes are currently in the pipeline, so sometime soon, you will see the database service exposed to you and hopefully it will be fully functional!

Another large task I've been working on is integrating the MailChimp api into the entire nodejitsu stack and API. During my first week, I got assigned to working with the MailChimp API to provide methods for users to do basic subscribing and unsubscribing of emails. Later, I got assigned to integrating the api I had just created into develop.nodejitsu.com. This was quite a bit of work which involved managing the MailChimp keys, learning the client-side code which used Porter and SugarSkull, and getting all the errors to propagate correctly to the user. This project will have a few more tricks up its sleeve, but already, you can see the fully functional implementation at develop.nodejitsu.com under the addons tabs.

More recently, I got placed in charge of managing the open-source project cliff. The primary enhancement I made was to the documentation. I gave many examples of how to use each of the functions and showed you the output in full color detail. I also changed how the length of a string was being calculated, so now you can use the console escape codes to color your output and the padding will be correctly calculated. Go ahead and check it out: cliff

Aside from all these projects, I've also been working with a few people at nodejitsu on a new community geared project which has not yet been released. There has been quite a bit of effort put into it and hopefully this new project will be really helpful to the node community!

Installation:

Installation is via npm. If you don't have npm yet:

 curl http://npmjs.org/install.sh | sh

Once you have npm:

 npm install http-server -g

This will install http-server globally so that it may be run from the command line.

Usage:

 http-server [path] [options]

The entire /mypath tree will now be available at http://localhost:8080/.

Available Options:

-p Port to listen for connections on (defaults to 8080)

-a Address to bind to (defaults to 'localhost')

-i Display AutoIndex (defaults to 'True')

-s or --silent In silent mode, log messages aren't logged to the console.

-h or --help Displays a list of commands and exits.

Now let's get hacking!

This guide is geared toward beginners and people who are unfamiliar with reverse HTTP proxying, websocket proxing, load balancing, virtual host configuration, request forwarding, and other web proxying concepts - those who already know what they're doing and just want to see the syntax should skip down to the sample code.

When should I use http-proxy?

Let's go over a very basic use case: you have one server, but you have more than one application hosted, each of which needs to handle requests for a specific domain.

The solution is to set up a proxy that listens for connections on one public port (i.e. 80), and then proxies the requests it receives to the proper application (each on its own non-public port) based on the hostname in each request.

But that sounds complicated!

Not for you it isn't! The http-proxy API is extremely simple, and you'll be proxying requests in no time at all. First, though, you'll need to install http-proxy. The recommended installation method is via npm. If you don't have npm yet:

 curl http://npmjs.org/install.sh | sh

Once npm is installed (or if you have it already) the simplest installation is as follows:

 cd myapp
 npm install http-proxy

This will install http-proxy to myapp/node_modules/http-proxy - once that's done, it's time to write some code! Let's look at the simplest kind of proxy first:

var http = require('http'),
    httpProxy = require('http-proxy');
//
// Create a basic proxy server in one line of code...
//
// This listens on port 8000 for incoming HTTP requests 
// and proxies them to port 9000
httpProxy.createServer(9000, 'localhost').listen(8000);

//
// ...and a simple http server to show us our request back.
//
http.createServer(function (req, res) {
  res.writeHead(200, { 'Content-Type': 'text/plain' });
  res.write('request successfully proxied!' + '\n' + JSON.stringify(req.headers, true, 2));
  res.end();
}).listen(9000);

Wait... really? Just one line of code?

Yes, really: one line of code to send all the requests that hit localhost:9000 to the waiting http server at localhost:8000, without requiring you to expose localhost:8000 directly.

This doesn't solve our problem, though: this code will proxy all the requests to the same place, rather than routing them by hostname. Don't worry: adding the extra functionality we need will be easy.

var http = require('http'),
    httpProxy = require('http-proxy');

//
// Just set up your options...
//
var options = {
  hostnameOnly: true,
  router: {
    'domainone.com': '127.0.0.1:9000',
    'domaintwo.net': '127.0.0.1:9001',
    'domainthree.org': '127.0.0.1:9002'
  }
}

//
// ...and then pass them in when you create your proxy.
//
var proxyServer = httpProxy.createServer(options).listen(80);

That's all the code you need to route to different ports/apps based on domain. It's also possible to leave out the hostnameOnly option and route based on path as well:

var http = require('http'),
    httpProxy = require('http-proxy');

//
//Leave out the hostnameOnly field this time, or set it to false...
//
var options = {
  router: {
    'domainone.com/appone': '127.0.0.1:9000',
    'domainone.com/apptwo': '127.0.0.1:9001',
    'domaintwo.net/differentapp': '127.0.0.1:9002'
  }
}


//
//...and then pass in your options like last time.
//
var proxyServer = httpProxy.createServer(options).listen(80);

That's it! It's really that easy. Set up your options, and you're only one line of code away from a battle-hardened, production-proven reverse HTTP proxy.

But what if I want to set up custom proxying logic?

No problem - http-proxy still has your back.

var http = require('http'),
    httpProxy = require('http-proxy');

//
// Create a proxy server with custom application logic
//
httpProxy.createServer(function (req, res, proxy) {
  //
  // Put your custom server logic here
  //
  
  if (req.url.match(/apples/)) {
    req.url = req.url.replace(/apples/, 'pears');
  }

  proxy.proxyRequest(req, res, {
    host: 'localhost',
    port: 9000
  });
}).listen(8000);

http.createServer(function (req, res) {
  res.writeHead(200, { 'Content-Type': 'text/plain' });
  res.write('request successfully proxied: ' + req.url +'\n' + JSON.stringify(req.headers, true, 2));
  res.end();
}).listen(9000);

That example was a little contrived, wasn't it?

Fine, smarty-pants - how's a round-robin load-balancing strategy for you?

var httpProxy = require('http-proxy');
//
// A simple round-robin load balancing strategy.
// 
// First, list the servers you want to use in your rotation.
//
var addresses = [
  {
    host: 'ws1.0.0.0',
    port: 80
  },
  {
    host: 'ws2.0.0.0',
    port: 80
  }
];

httpProxy.createServer(function (req, res, proxy) {
  //
  // On each request, get the first location from the list...
  //
  var target = addresses.shift();

  //
  // ...then proxy to the server whose 'turn' it is...
  //
  proxy.proxyRequest(req, res, target);

  //
  // ...and then the server you just used becomes the last item in the list.
  //
  addresses.push(target);
});

// Rinse; repeat; enjoy.

That was cool, but I'm still not impressed.

Well how about proxying socket.io websockets? Something like this can help socket.io to scale properly:

var sys = require('sys'),
    http = require('http'),
    colors = require('colors'),
    websocket = require('./../vendor/websocket'),
    httpProxy = require('node-http-proxy');

var utils = require('socket.io/lib/socket.io/utils'),
    io = require('socket.io');

//
// Create the target HTTP server...
//
var server = http.createServer(function (req, res) {
  res.writeHead(200);
  res.end();
});
server.listen(8080);

//
// ...now setup socket.io on the target HTTP server to handle websocket requests...
//
var socket = io.listen(server);
socket.on('connection', function (client) {
  sys.debug('Got websocket connection');

  client.on('message', function (msg) {
    sys.debug('Got message from client: ' + msg);
  });

  socket.broadcast('from server');
});

//
// ...next, create a proxy server with node-http-proxy...
//
var proxy = httpProxy.createServer(8080, 'localhost');
proxy.listen(8081);

//
// ...and setup the web socket to wait for connections from our proxy.
//
var ws = new websocket.WebSocket('ws://localhost:8081/socket.io/websocket/', 'borf');

ws.on('open', function () {
  ws.send(utils.encode('from client'));
});

ws.on('message', function (msg) {
  sys.debug('Got message: ' + utils.decode(msg));
});
 

For further information on http-proxy, please visit the node-http-proxy github page.

In this article I will discuss our approach for fixing the node_mailer library and how we actually helped fix issues in another mailer project along the way.

Libraries

First we evaluated many of the modules at hand that are commonly used for mailing (funny enough, there are several modules matching /node[-_]?mailer/i). There were various issues with each project and none of the other projects supported the full feature set we wanted for sending emails. We had to make a hard choice: Rewrite our entire node_mailer project ( which would have taken a lot of development hours ) or abandon it ( which would leave 100s of developers without a working library ). After further reasearch we realized there was a third option: Apply patches to another mailer project, then use this as a dependency for our mailer. We could get the functionality we wanted with minimal work and help improve someone else's project. We considered this approach the best, both for ourselves and for the community as a whole.

NodeMailer

We ended up choosing to use NodeMailer as a dependency because it was feature rich, mature, extensible, and easy to use. It was also recommended in this poignant node_mailer issue. NodeMailer is inspired by Google's Mail Python API library and has very nice documentation. Also, NodeMailer uses a pipelined state callback letting us easily hook both before and after events while extending it. For those of you out there who use unicode in your account info, it supported UTF-8! Amazingly, the turnaround time on tickets that were reproducible bugs was around a day. This library was looking very good.

But, there were some downsides:

• NodeMailer did not support StartTLS.
• We couldn't queue up multiple messages.

So, it was decided to kick some bug ass and get those 2 issues resolved.

Thanks to TooTallNate's example floating around we were able to get StartTLS going in short order.

For message queueing we had to dig a bit deeper. The SMTP logic was in both the SMTPClient and Message interfaces. If we wanted robust queueing we would need to put it all in one place unless we wanted to spawn up entirely new clients as a queue (yikes!). So, that bug too was defeated handily. For now though, that lives alone in node_mailer. We will get that pushed out to NodeMailer once we get some design stuff ironed out.

Along the way we had some minor fixes for some SMTP reply codes, a parser bug that would only occur in very edgy situations, and a whole lotta kicking ass.

The new fine tuned node_mailer on Github is working better than ever and feels good to use. We were able to fix the problems we had with our node_mailer library and help patch up NodeMailer. Both libraries are better now and it's a win for the node community.

Future

There are still a few things to do to make this module something we can call the definitive answer to Node's emailing needs. Hopefully, the community can work with us to get a couple more features bore into Nodemailer to make it better than ever! If you want to get started head on over to the github issues for Nodemailer, and the node_mailer you know and love will get them downstream.

Dependency management. Need I say more? Entire developer ecosystems live and die by how dependency management works. Ask a Ruby developer what they would do without RubyGems, or ask (most) Java developers how to get things done without Maven. Most importantly: ask a node.js developer how to get things done without npm.

Npm recently received a pretty serious makeover in Version 1.0. If you haven't read up on what changed between 0.3.18 and 1.0, you should read Isaac Schlueter's (or simply @isaacs), the author of npm, blog post now.

Npm installs modules and applications based on an interpretation of the package.json specification. The most important part of this is how it determines the which version(s) of your dependencies to download. In this article I will examine:

1. How npm works with dependencies in a package.json manifest.
2. Best practices for specifying modules and node.js dependencies.

tldr;? Jump down to Best Practices

Semantic versioning in npm

There are two separate (but equally important) entities that must have specified versions in a package.json manifest: modules dependencies and node version. All of these version numbers are interpreted using a slightly modified semantic version (or semver) specification.

What is slightly modified about it? It accepts the '-' character as a valid character in build strings. So 0.2.0-1 is valid in npm semantic versioning, but not in the official semver spec. Personally, I like this flavor of semver and hope that it will get pulled in up-stream.

Ok, so npm uses semver, awesome. Now what? Lets take a look at a naive example:

{
  "name": "naive",
  "description": "A package using naive versioning",
  "author": "A confused individual <iam@confused.com>",
  "dependencies": {
    "express": ">= 1.2.0",
    "optimist": ">= 0.1.0"
  },
  "engine": "node 0.4.1"
}

The versions for both the dependencies and node properties in the example above are interpreted using node-semver, a library created and maintained by isaacs for working with semantic version strings.

Now suppose that we install the above example using npm:

1. The node version installed on the current system is compared against the version specified in the package.json. If the node version does not satisfy the target version, the installation fails.
2. For each dependency specified in the dependencies property, the highest version which satisfies the specified semantic version string is downloaded and placed into the node_modules folder in the target directory. If there is no version available in the npm registry which satisfies the target version, the installation fails.

Dependencies: Best Practices

"Best practices" is a term that is often misused and thus can be ambiguous in this context. For such reasons, it is important to make clear the goals behind this particular versioning scheme:

1. Strive to be forwards compatible with node.js (within reason)
2. Keep the dependencies of multiple installations of the same app cohesive
3. Reduce the surface area of potential bugs due to breaking changes

Here is a sample package.json that uses the best practices we will discuss:

{
  "name": "best-practices",
  "description": "A package using versioning best-practices",
  "author": "Charlie Robbins <charlie@nodejitsu.com>",
  "dependencies": {
    "colors": "0.x.x",
    "express": "2.3.x",
    "optimist": "0.2.x"
  },
  "devDependencies": {
    "vows": "0.5.x"
  },
  "engine": "node >= 0.4.1"
}

When specifying modules dependencies: use 1.0.x syntax

Until recently, I was guilty of not following this guideline: I continued to use the >= 0.2.0 syntax illustrated above in the naive.package.json example. At first glance there doesn't seem to be anything wrong with that style. You're saying "If there are changes in the future I want them."

The problem arises because authors are conveying meaning in their versions. Not every version will be completely backwards compatible with the particular version you were using when you wrote your application. This is conveyed in the version string:

e.g.: 0.3.18

• Major Version (0)
• Minor Version (3)
• Patch Version (18)

Changes to the major and minor parts of the version mean that changes have happened, although there is no convention to convey they are breaking. Changes to patch versions are used to express that a fix has been made and it is (usually) safe to upgrade.

Conversely, when using the 0.2.x syntax you're saying: "If there are patch changes in the future I want them, but no minor or major versions." Given the description of the meaning conveyed by each of the version components above this means you won't be tearing your hair out over breaking changes in a module you depend on.

When specifying node versions in modules: use >= 0.4.0 or 0.4.x | 0.5.x syntax

Don't write modules and publish them on npm? Skip this and jump down to specifying node versions in apps.

As a module author it is your responsibility to consciously follow anything you take as a dependency and ensure that you update your module(s) accordingly. Beyond that, you must also realize that (if you're lucky) your module will be used on the bleeding edge. As such, you must accommodate these valid scenarios. At the end of the day it is the choice of the user to decide on their target node version and you should not limit their choices unless you are explicitly aware of a problem in future versions of node.js core.

When specifying node versions in apps: use 0.4.x or 0.4.0 syntax

As with any sufficiently important problem, there are nuances to dependency management you should be aware of. One such nuance is the difference between specifying dependencies in an application vs. modules.

Module authors encounter a much wider surface area of use cases, and as such should be more liberal in the versions of node.js they accept. On the other hand, application developers typically have no upstream dependencies and should be more explicit about what version of node.js they are expecting on the target system.

For example, at Nodejitsu we will use the highest version of node.js which satisfies the node specified version. So if you use 0.4.x, but are really expecting 0.4.5, you should indicate this in your package.json or else you may encounter unexpected behavior.

Take advantage of devDependencies

The package.json specification used by npm has an additional property for separately listing dependencies which are only required for development purposes. Why separate you might ask? Besides not having to download these dependencies from the registry when running in production, npm it is required if you wish to take advantage of a couple of neat npm features such as test:

  $ pwd
  /Users/Charlie/GitHub/nconf

  $ npm test
  npm info it worked if it ends with ok
  npm info using npm@1.0.6
  npm info using node@v0.4.8
  npm info pretest nconf@0.1.9
  npm info test nconf@0.1.9

  > nconf@0.1.9 test /Users/Charlie/GitHub/nconf
  > vows test/*-test.js --spec

  (...)

If I had not specified my test library, vows, in the devDependencies section of the package.json for nconf I would have received:

  $ npm test
  npm info it worked if it ends with ok
  npm info using npm@1.0.6
  npm info using node@v0.4.8
  npm info pretest nconf@0.1.9
  npm info test nconf@0.1.9

  > nconf@0.1.9 test /Users/Charlie/GitHub/nconf
  > vows test/-test.js --spec

  sh: vows: command not found
  npm info nconf@0.1.9 Failed to exec test script
  npm ERR! nconf@0.1.9 test: `vows test/-test.js --spec
  npm ERR! sh "-c" "vows test/*-test.js --spec"` failed with 127
  npm ERR! 
  npm ERR! Failed at the nconf@0.1.9 test script.

  (...)

Conclusion

Node.js is evolving rapidly: it has been less than a month since npm 1.0 was released which dramatically changed the way in which dependencies are handled. More importantly: the modules on-top of node.js critical to building scalable web applications are evolving even faster, making the best practices outlined here that much more important if you want to cope.

As with any complex ecosystem, this is a process and these practices may change over time as the goals outlined change. We'll keep you posted at Nodejitsu.

The Single Page App is not hype, it's an option.

First off, we need to clarify some nomenclature. Two characteristics clearly differentiate Web Sites from Web Apps. A Web Site is primarily informational, it has limited interactive behavior per page. A good example of a Web Site is eBay. Conversely, a Web App is far more interactive. It attempts to emulate or capitalize on familiar behaviors found in the applications on your desktop. An example of this might be the New York Times Web Reader App

A Single Page App is just a Web App, but what's unique is the anatomy. It's all in the name. Essentially, all of the markup is served up on the first request. Subsequent requests are then made to a REST API which provides JSON. Content transformations may also occur by means of a Web Socket or a remote procedure call.

The Advantages.

Aside from the obvious gains in user experience such as not having to reload the entire page for content changes, there's a long term Return of Investment to this approach. Since your data layer is decoupled from your UI, you can easily create a native app for a mobile device and leverage your existing APIs without the sky falling on you. Black boxes are cool. It's a lot simpler to write a next-generation app against an API of data than an router full of templates.

The bottom line is that single page apps are efficient. They have a small footprint on the server, consume less bandwidth and are more conducive to a service oriented architecture.

The Disadvantages.

What about SEO and the hash-bang shit storm I've been hearing about? This is FUD. There are certainly some radical perspectives on this subject, and usually extremists are narrow sighted. If you rely on SEO and you need your app to load without Javascript, you might not want to build a Web App. You should be building a Web Site. Advertising and promotion shouldn't drive the usability of an application.

Here's some art to try to explain the anatomy of a Single Page App.

Single Page Apps are simple. First, serve what's needed of the html. Second, speak with APIs using Javascript. Keep in mind there may be slightly more html than usual, so it should be very well organized and commented.

The Tools

SugarSkull

Single Page App is a misleading inscription. We actually have a single body of markup and we want to present the user with many views. Enter SugarSkull. SugarSkull is a client side URL router. It's the smallest amount of glue needed for building dynamic single page applications. Not a jQuery plugin: no dependencies.

// define a set of artificial urls
var router = Router({

  '/dog': { // a regular expression that represents a part of a route
    '/angry': { // another potential route part
      on: [growl, freakout] // a list of functions to execute when that route is requested
    }
    on: bark // a single function to execute whent the route is requested
  },

  '/cat': {
    '/squish': {
      once: freakout // a function that will only get executed once after the route is requested
    }
    on: meow
  }

});

You might also find that backbone.js is quite complimentary to SugarSkull with its Views and Models. Long lived data on the client means less round trips to the server, and that's one of the main goals.

Local Storage

Persistent local storage reduces round trips. The state of the application can be complex, user preferences and other unshared data can be decentralized. There are some good hacks for dealing with shitty browsers, Google Gears for example. But since you are creating a Web App you should be mostly concerned with modern browsers. You can find out more about it here as you dive into HTML5

Socket.io

"Socket.io aims to make realtime apps possible in every browser and mobile device, blurring the differences between the different transport mechanisms." What else can be said? This is exactly what we want for single page apps, responsive network interactions. Let's take a closer look.

On the client side

// Let's create a new socket and assign it to a local variable.
var socket = new io.Socket();

// Establish a connection back to the server from the browser.
socket.connect();

// Attach a handler to the window load event.
window.onload = function() {
  
  // Attach an event to the click event of document's body.
  document.body.onclick = function () {
    
    // Come up with some kind of stupid message.
    var message = 'The time is now ' + new Date();
    console.log('Sending the message "' + message + '"');
    
    // Send the message to the server.
    socket.send(message);
  };
};

// Create a handler for when a message arrives from the server.
socket.on('message', function(message) { 
  // When a message arrives, replace the body of the document with the message.
  document.body.innerHTML = message;
});

On the server side

// lets include some modules
var http = require('http'), 
    io = require('socket.io'),
    fs = require('fs');

// for the sake of this example, lets have some html to serve up.
var html = fs.readFileSync('socketio.html').toString();

// business as usual, create an http server.
var server = http.createServer(function (request, response) {
  // serve up the client page.
  response.writeHead(200, {'Content-Type': 'text/html'});
  response.end(html);
});

// listen on port
server.listen(80);

// attach socket.io to the server
var socket = io.listen(server);

// set up an event that handles connections that get made to the server.
// the callback for this event will supply the socket as a parameter.
socket.on('connection', function(client) {
  // on the socket we can attach events, lets respond to the client when
  // we recieve a message.
  client.on('message', function(message) {
    // we can log the message on the server side.
    console.log(message); 
    // then send it back to the client.
    client.send('Thanks for telling me "' + message + '"');
  });
});

Our communication vector is short, concise and that means snappy user interfaces. Socket.io is awesome. So is dnode. But Web sockets aren't available everywhere. Much of the web has started to uses REST. Porter is a nice little way to organize your REST communications.

Porter

Porter is a lightweight, resourced oriented, abstraction layer for JSON-REST. It will generate methods needed to access resources based on a JSON configuration. It will balance your code's signal to noise ratio by simplifying the communication interfaces. Applications with a lot of ajax calls can get ugly.

var porter = Porter({

  // define a resource group
  users: {
    // define a resource, a verb and a uri with some tokens replace
    search: ['get', '/api/users/:partialname'],
    update: ['post', '/api/apps/:username']
  },

  apps: {
    list: ['get', '/api/apps/:username'],
    create: ['post', '/api/apps/:username/:appname']
  }

});

// porter will transformt the above defintion into usable methods...
porter.users.search(

  { partialname: 'bill' }, // supply the value of the token to replace in the url

  function(error, response) {
    // do something...
  }

);

Conclusion

The idea is simple. Serve up a base presentation document. Manipulate it with Javascript. Get more data from the server and repeat previous step. From a performance perspective, Single Page Apps are nearly on parity with native apps (in the context of a modern browser). Nearly every operating system supports a modern browser. And with HTML, CSS and Javascript as the SDK way more people can join the party.

For all of these reasons I list above, I become considerably disappointed when I see an article like this appear on IBM Developer Networks. I can understand that node.js is still a relatively new technology ( it's not even v1.0.0 as per SemVer yet ), but that is no excuse for an industry leader like IBM to allow such a blatantly misleading article to appear on their site. It's not my intention to single anyone out, but it's apparent that the author of the article has a limited understanding of node.js, has performed a trivial amount of research, and most importantly has no understanding of how open-source projects work in the age of Github.

Normally if I encounted such an unresearched and misleading article on the internet I'd just ignore it. Unfortunately, I can't ignore this. IBM is a serious name that people trust. To have an article on IBM.com spreading misinformation about node.js is wrong. IBM needs to update or remove this article. In this blog post I will distill and address the main points which are "wrong" in IBM's article

Not understanding open-source in the age of Github

A major recurring theme I noticed in the article was a complete lack of mentioning any node.js module authors, any github repos, or anything about npm and the npm module repository. node.js was built on Github. It is one of the first major open-source projects to begin it's lifecycle on Github. Anyone who has committed to node.js or created a node.js module uses Github. There are numerous listings and rankings of node.js module authors and node.js Github members. There is a vibrant IRC room ( #node.js on irc.freenode.net ) with 600+ active members. There is a mailing list with 100s of messages per day. I follow 500+ node.js developers on my Github activity stream. There is an incredible amount of project activity based around Github.

Incorrect / Confusing Terminology

Early in the article I found this line explaining how node.js works internally:

"...node solves the problem by changing how a connection is made to the server. Instead of spawning a new OS thread for each connection (and allocating the accompanying memory with it), each connection creates a process, which doesn't require the memory block to go with it."

The ambiguity and glossing over of the word "process" is wrong. This is bad technical writing.

I asked Isaacs ( author of npm ) if he could clearly explain why this is wrong and his response was:

[17:21] <isaacs> Marak: i don't even know where to start

Moving on...

"Yes, node is a server program. However, it is definitely not like Apache or Tomcat. Those servers are stand-alone server products, ready to install and deploy applications instantly. You could be up and running with a server in a minute with these products. node is definitely not this. Apache can add a PHP module to allow developers to create dynamic web pages, and programmers using Tomcat can deploy JSPs to create dynamic web pages. node is definitely not this."

Ahh! This is where I start to lose it. The author doesn't understand that node is much more then just a web-server. The http server in node is a part of ONE module out of twenty-five core modules and currently 2000+ npm modules. The analytics on npm are a bit terrifying. We are seeing ten new packages released per day. Why did you not mention npm at all?

At the risk of having a stroke, I continue reading...

"At this early point in node's life (currently on version 0.4.6), it is not a ready-to-run server program, where you can expect to install it, drop your files into it, and have a fully functioning web server. It still requires a non-trivial amount of work to get even the basic functionality of a web server up and running after installation is complete."

You can't just drop in pages and starting build node.js web apps in seconds? I disagree with this assertion. I'll explain towards the end of my post, until then let's move on to the next section in the IBM article titled "How node works", first paragraph:

"Server-side JavaScript is a relatively new concept, one which was written about a couple of years ago here on developerWorks when discussing the Aptana Jaxer product (see Resources)."

Umm? Server-side javascript has been around since 1996. WTF does Jaxer have to do with how node.js works?

At this point I start to fear I might have an aneurysm.

Asserting invalid facts as truth

Enough of the nitpicking, let's jump down to the author's conclusion about what node.js is good for and bad for.

What does the author think node is good for?

He asserts node is good for A RESTful API and a Twitter queue. These sound pretty reasonable to me. No objections here.

Then he goes on to state:

"A company that has a large distributed website (think Facebook or Flickr), could decide to devote entire boxes to simply serving up images. node would be a good solution for this problem because the company can use it to code an easy file retriever and then handle tens of thousands of connections. node would look for the image file, return it or a 404 error, and do nothing else. This setup would allow these types of distributed websites to reduce the number of server boxes they need to serve static files such as images, .js files, and .css files."

Hrmmm? Last time I checked binary file support in JavaScript was not that great. I've also heard Ry talk about node's performance in regards to serving static files compared to other well-developed tools as as nginx. The conclusion is you shouldn't use node if all you need is a static file server.

What does the author think node is bad for?

"Dynamically created pages"

"Currently, node doesn't provide a default way to create dynamic pages. For example, when using JavaServer Pages (JSP) technology you can create an index.jsp page that contains loops in JSP snippets like <% for (int i=0; i

Do you have any idea how node works? There are literally 20x different node.js templating engines that support pretty much every single style of markup you'd ever want. I understand you really like Apache and Tomcat, but your understanding of node is just WRONG.

"...Therefore, if you wanted to provide a server-side solution for this in node, you'd have to code the entire solution yourself."

Express? Connect? Geddy? webservice.js? I guess these projects haven't been being worked on for over a year. Someone should definitely step up and build a node.js MVC framework, I think there might only be 30x of them available now.

"Relational database heavy applications"

"node is designed to be fast, asynchronous, and non-blocking. Databases don't necessarily share these goals. They are synchronous and blocking, as calls to the database on writes and reads block until a result is generated. So, a web application that requires a lot of database calls, a lot of reads, and a lot of writes with every request would be a bad match for node, since the relational database itself would be negating many of the strengths of node. (The new NoSQL databases are a better match for node, but that's another topic entirely.)"

Don't make broad assertions about node on topics such as "relational database heavy applications". This is highly ambiguous and leads me to believe the author doesn't understand the finer points of application architecture and design. There are several non-blocking implementations of most SQL and NOSQL databases in node.

Why would I ever design an application in node.js that performed "lots" of database I/O on every single request? It sounds like an uninformed opinion based on experiences in a narrow field of software development.

Lack of Research

There is no excuse for this. If you are going to put an article up on IBM.com, you should spend more than an afternoon researching it. I didn't start blogging about node.js until I had already built a project and published it to npm. I didn't dare start blogging about the node.js ecosystem or usage until I had been with the project for almost a year.

It is irresponsible to spread incorrect information like this on such a well respected channel ( IBM developer networks ).

Unfair comparisons to legacy technologies

I feel I've touched on this a few times already, but it should be apparent that the author has a particular bias for his favorite technologies. He compares node.js unfairly to technology which he obviously has a vested stake in. If you want to promote your own technologies, that is fine by me. Just don't spread misinformation about node.js in an effort to make your technology look better.

Conclusion

IBM should pull the article or ask the author to rewrite it. The article is providing a dis-service to any new developers who might stumble along it as their first introduction to node.js.

Artwork courtesey of http://memegenerator.net/

Making sure your package.json has all the right ingredients is important! Nodejitsu is happy to announce the release of a simple and elegant tool for determining and analyzing your dependencies, aptly named: require-analyzer. This post will walk you through how to use the library as well as outline our jitsu integration and upcoming npm integration.

Using require-analyzer

There are two distinct ways to use the require-analyzer library: from the command line or through code. The command line tool is designed to work with package.json files; make sure that you have created one for your project beforehand.

Lets take a look at how I used require-analyzer to analyze its own dependencies:

  $ require-analyzer
  info:  require-analyzer starting in /path/to/require-analyzer
  warn:  No dependencies found
  info:  Analyzing dependencies...
  info:  Done analyzing raw dependencies
  info:  Retrieved packages from npm
  info:  Additional dependencies found
  data:  {
  data:    async: '>= 0.1.8',
  data:    findit: '>= 0.0.3',
  data:    npm: '>= 0.3.18'
  data:  }
  info:  Updating /path/to/require-analyzer/package.json
  info:  require-analyzer updated package.json dependencies

If you're a power user (you know who you are) don't fret: the require-analyzer API is fully available through pure node.js code. The easiest way to use require-analyzer programmatically is through the .analyze() method. This method will use fs.stat() on the path supplied and attempt one of three options:

1. If it is a directory that has a package.json, analyze require statements from package.main
2. If it is a directory with no package.json analyze every .js or .coffee file in the directory tree
3. If it is a file, then analyze require statements from that individual file.

Lets dive into a quick sample usage:

  info:   Analyzing your application dependencies in bin/server
  info:   Found new dependencies
  data:   {
  data:       colors: '>= 0.5.0',
  data:       http-proxy: '>= 0.5.0',
  data:       loggly: '>= 0.3.2',
  data:       node-static: '>= 0.5.3',
  data:       optimist: '>= 0.1.9',
  data:       request: '>= 1.9.5',
  data:       winston: '>= 0.2.7'
  data:   }
  warn:   New dependencies will be added for you automatically.

This article marks the release of node-http-proxy v0.5.0 which has some breaking changes to make the API more extensible for the next WebSocket and HTTPS support. We'll be reviewing changes since v0.3.0, which is still in use in many environments. tl;dr? Checkout the Changelog for the project.

Who is affected?

From the beginning, node-http-proxy has been designed with an extremely flexible API that allows you to use it as a stand-alone proxy server or in conjunction with another HTTP server (for example, a connect server). If you're using the stand-alone proxy you have nothing to worry about, that API hasn't changed at all:

  require('http-proxy').createServer(port, host, options).listen(8000);

APIs for other more-custom tailored use-cases have changed, so read on below. All of these code samples are also available as examples on GitHub.

Use http.Agent in node.js 0.3.6+

We took advantage of the new http.Agent APIs introduced in node.js v0.3.6+. This allowed us to remove the "pool" module dependency because instances of http.Agent now perform socket pooling.

No more eager request buffering

It is no longer necessary to create an instance of HttpProxy for each incoming connection. The motivation for that design choice was that we could eagerly buffer the incoming request in-case the user decides to wait before proxying (i.e. latent proxying). This turns out to be less than ideal for memory performance because not every scenario will require request buffering. The onus is therefore now on the user using the new .buffer() method:

Old (v0.3.0)

var http = require('http'),
    httpProxy = require('http-proxy');
    
//
// Create a proxy server with custom application logic
//
httpProxy.createServer(function (req, res, proxy) {
  //
  // Wait for two seconds then respond: this simulates
  // performing async actions before proxying a request
  //
  setTimeout(function () {
    proxy.proxyRequest(9000, 'localhost');      
  }, 2000);
}).listen(8000);

New (v0.5.0)

var http = require('http'),
    httpProxy = require('http-proxy');
    
//
// Create a proxy server with custom application logic
//
httpProxy.createServer(function (req, res, proxy) {
  //
  // Buffer the request first
  //
  var buffer = proxy.buffer(req);
  
  //
  // Wait for two seconds then respond: this simulates
  // performing async actions before proxying a request
  //
  // Note: Now we have to pass both `req` and `res` to 
  // `.proxyRequest()`.
  //
  setTimeout(function () {
    proxy.proxyRequest(req, res, {
      host: 'localhost',
      port: 9000,
      buffer: buffer
    });      
  }, 2000);
}).listen(8000);

It's also important to note in the above v0.5.0 code sample how the arguments passed to .proxyRequest() have changed from a list of arguments to the following signature: proxy.proxyRequest(req, res, options) where options must include the host and port properties.

Using WebSocket Support

Websockets are handled automatically when using the httpProxy.createServer(), but if you want to use it in conjunction with a stand-alone HTTP + WebSocket (such as socket.io) server here's how:

var http = require('http'),
    httpProxy = require('http-proxy');
    
//
// Create an instance of node-http-proxy
//
var proxy = new httpProxy.HttpProxy();

var server = http.createServer(function (req, res) {
  //
  // Proxy normal HTTP requests
  //
  proxy.proxyRequest(req, res, {
    host: 'localhost',
    port: 8000
  })
});

server.on('upgrade', function(req, socket, head) {
  //
  // Proxy websocket requests too
  //
  proxy.proxyWebSocketRequest(req, socket, head, {
    host: 'localhost',
    port: 8000
  });
});

Proxying over HTTPS

Proxying over HTTPS is a common scenarios for reverse proxies. You want your proxy server running on HTTPS while each of your target servers is running HTTP. Both HTTPS -> HTTPS and HTTPS -> HTTP proxying is now possible in node-http-proxy v0.5.0.

var fs = require('fs'),
    https = require('https'),
    httpProxy = require('http-proxy');
    
var options = {
  https: {
    key: fs.readFileSync('path/to/your/key.pem', 'utf8'),
    cert: fs.readFileSync('path/to/your/cert.pem', 'utf8')
  }
};

//
// Create a standalone HTTPS proxy server
//
httpProxy.createServer(8000, 'localhost', options).listen(8001);

//
// Create an instance of HttpProxy to use with another HTTPS server
//
var proxy = new httpProxy.HttpProxy({ https: true });
https.createServer(options.https, function (req, res) {
  proxy.proxyRequest(req, res, {
    host: 'localhost', 
    port: 8000
  })
}).listen(8002);

//
// Create the target HTTPS server for both cases
//
https.createServer(options.https, function (req, res) {
  res.writeHead(200, { 'Content-Type': 'text/plain' });
  res.write('hello https\n');
    res.end();
}).listen(8000);

Roadmap

Clearly node-http-proxy has come a long way since v0.1.0. But where do we go from here?

1. Add Whitelist / Blacklist Support
2. Support Gzip
3. Proxy Authentication
4. Add something yourself!

It's a JavaScript Trampoline Party at the House of Air!

We'll be giving out 25 free tickets!

There is also Trampoline Dodgeball!

4/28/2011 between 7pm-9pm

Nodejitsu is sponsoring a small kick off party in San Fransisco the week before JSConf and NodeConf. Our team will be at both conferences and we'll sponsoring / speaking at NodeConf. We are doing training sessions before and between JsConf and NodeConf, so please be sure to sign up if you haven't' already.

So back to the Trampoline Javascript Party....where is it going to be? What is the House of Air? Who will the lucky 25 be that get to go...

The House of Air is a trampoline park located at Building 926 926 "Old" Mason Street West Crissy Field The Presidio of San Francisco, CA 94129. It has multiple trampoline areas, including a free-jump colosseum and an arena for non-stop five minute rounds of trampoline dodgeball. It's an amazingly fun place to go to and I've never had a bad time going there. I'd advise against eating any bacon cheeseburgers before attending the park...but perhaps I am getting a bit ahead of myself.

We've got 25 tickets to give away and we'll be giving them on a first come first serve basis on the following ranked items:

1. If you are a Node.js Ninja
2. If you are a JavaScript Ninja
3. If you are have a Github account
4. If you are following Nodejitsu team members on Github
5. If you have a twitter and are following @Nodejitsu or our team members

You'll be allowed to bring guests...so let us know how many guests you intend to bring. Bonus points if you are planning to bring your development team as your guests.

We've only got a limited amount of spots, so we won't be able to accommodate everyone... If you still want to go and don't know if you'll make the list, you can always buy a ticket directly on the House Of Air website on the the date and time of the event here. I can't guarantee any spots will be available outside of our block, but there should be at least a few.

If you'd like to apply please submit your github handle and desired guest count to:

jumparound@nodejitsu.com

After the event, we'll be moving the party to an undisclosed location nearby where we can sip on tasty drinks and eat delicious food.

I look forward to seeing some JavaScript Ninjas in action

Here's a good video of the free-jump area:

And here is a video of some Ninja Trampoline Dodgeball Skills

• "I would kill myself without BDD"
• ZOMG! How could you write software without TDD!?!
• "Testing? Meh. I try to have unit tests..."

I think more developers than not fall into the mindset of the last quote; we want to test our software but we don't have the time, or resources to make 100% test coverage a reality. The underlying fact behind this dirty secret is that testing can be quite difficult and requires the utmost dedication to flesh out completely.

This excess of verbosity and difficulty was the motivation for writing APIeasy: a simple tool for writing BDD tests for any REST API. This article will explore a sample API written in node.js and how it can be tested with APIeasy.

Update: So the Internet and I were talking and it turns out there is already a library called RESTeasy. But I mean, who still uses Java anyway? In response to a request from the Internet at-large (i.e. the authors of the existing RESTeasy library) I have changed the name of this library to APIeasy.

Getting started

APIeasy will let you test any API, not just APIs written in node.js. So if you're writing in PHP, Ruby, or Python and looking for a good way to get started in node, here's your answer. For the sake of this example, however, we will be writing our API in node.

The following code sample creates a simple JSON-based web service using journey with a couple of simple routes:

• GET /ping: Responds with { pong: true }
• POST /ping: Responds with the POST data
• GET /login: Responds with { token: /\d+/ }
• GET /restricted: Responds with { authorized: true } if the correct token is sent
  
  

var http = require('http'),
    journey = require('journey');

//
// Create a journey router. Not familiar with journey? Checkout:
// http://github.com/cloudhead/journey
//
var token, router = new journey.Router({ 
  strict: false,
  strictUrls: false,
  api: 'basic'
});

//
// Create a simple (and not production ready) function
// for authorization incoming requests.
//
function isAuthorized (req, body, next) {
  return parseInt(req.headers['x-test-authorized'], 10) !== token 
    ? next(new journey.NotAuthorized())
    : next();
}

//
// GET /ping: 
//   * Responds with 200
//   * Responds with `{ pong: true }`
//
router.get('/ping').bind(function (res) {
  res.send(200, {}, { pong: true });
});

//
// POST /ping
//   * Responds with 200
//   * Responds with the data posted
//
router.post('/ping').bind(function (res, data) {
  res.send(200, {}, data);
});

//
// GET /login
//   * Responds with 200
//   * Responds with { token: /\d+/ }
//
router.get('/login').bind(function (res) {
  if (!token) {
    token = Math.floor(Math.random() * 100);
  }
  
  res.send(200, {}, { token: token });
});

//
// Filter requests to /restricted using isAuthorized.
//
router.filter(isAuthorized, function () {
  //
  // GET /restricted
  //   * Responds with 200
  //   * Responds with { authorized: true }
  //
  this.get('/restricted').bind(function (res) {
    res.send(200, {}, { authorized: true });
  });
});

//
// Create a simple HTTP server to 
// consume our router.
//
http.createServer(function (request, response) {
  var body = "";

  request.addListener('data', function (chunk) { body += chunk });
  request.addListener('end', function () {
    //
    // Dispatch the request to the router
    //
    router.handle(request, body, function (result) {
      response.writeHead(result.status, result.headers);
      response.end(result.body);
    });
  });
}).listen(8000);

Creating an APIeasy suite

So what are the concerns that one might have when testing such an API taking into consideration the benefits of descriptive tests (read: BDD)?

1. Ensure the correct HTTP response code is returned
2. Ensure the correct response body is returned
3. Ensure that a custom assertion on the HTTP response is satisfied
4. Organize tests so that each test has a meaningful description

APIeasy addresses all of these concerns in a simple, fluent (i.e. chainable) API. The core of this API are helper methods for making common types of HTTP request. Each of these methods will perform an HTTP request against the current path for the suite configured by calls to .path() or .unpath(). The optional parameters configure the subpath to the resource, URI paramaters, and optional request body.

• .get(/* [uri, params] */):
• .post(/* [uri, data, params] */)
• .put(/* [uri, data, params] */)
• .del(/* [uri, data, params] */)
• .head(/* [uri, params] */).

After a call to one of these methods has been made the user can add tests for this particular request by using .expect() which has a couple of options:

• .expect(200): Ensures that a 200 response code is returned
• .expect(200, { some: 'body' }): Ensures that a 200 and that the response body is { some: 'body' }
• .expect(function (error, response, body) { ... }): Ensures that the custom assertion on the error, response and body is met.
  
  

var APIeasy = require('api-easy');

//
// Create a APIeasy test suite for our API
//
var suite = APIeasy.describe('api');

//
// Add some discussion around the vowsjs tests.
// Not familiar with vows? Checkout:
// http://vowsjs.org 
//
suite.discuss('When using the API')
     .discuss('the Ping resource');

In the above code sample we create a APIeasy suite, and then add some discussion around the tests we are about to create. Each method on the APIeasy suite returns the suite itself so we can continue to add additional tests and test context to the suite.

Adding tests with APIeasy

APIeasy allows you to add both tests and set default options for each outgoing HTTP request against your API. Lets add some tests for the /ping route in our sample API:

//
// Here we will configure our tests to use 
// http://localhost:8080 as the remote address
// and to always send 'Content-Type': 'application/json'
//
suite.use('localhost', 8000)
     .setHeader('Content-Type', 'application/json');
     //
     // A GET Request to /ping
     //   should respond with 200
     //   should respond with { pong: true }
     //
     .get('/ping')
       .expect(200, { pong: true })
      //
      // A POST Request to /ping
      //   should respond with 200
      //   should respond with { dynamic_data: true }
      //
     .post('/tests', { dynamic_data: true })
       .expect(200, { dynamic_data: true })

Adding sequential tests

As you build more complex test suites, you will want to add and remove discussion text using the .discuss() and .undiscuss() methods. This design is similar to the native Javascript Array .shift() and .unshift() except that the operations are switched: .discuss() will append the string to the set of discussion used in the current suite and .undiscuss() will remove the latest discussion string.

Testing APIs also frequently requires that certain tests be run in sequence (i.e. one test must be run after another). APIeasy makes this possible through the .next() method. Tests after an invocation of .next() will be added to a separate vows batch ensuring that they run after the current tests.

//
// Add more tests
//
suite.undiscuss()
     .discuss('when authenticating')
     .get('/login')
     .expect(200)
     .expect('should respond with the authorize token', function (err, res, body) {
       var result = JSON.parse(body);
       assert.isNotNull(result.token);

       suite.before('setAuth', function (outgoing) {
         outgoing.headers['x-test-authorized'] = result.token;
         return outgoing;
       });
     })
     //
     // Before we can test our request to /restricted 
     // the request to /login must respond. To ensure this
     // we make a call to .next() before our next call to .get()
     //
     .next()
     .get('/restricted')
       .expect(200, { authorized: true })

As you can see in the above code sample, .next() is extremely important because we can only make a successful request to /restricted after we have gotten an authentication token from our request to /login.

Running your test suite

APIeasy is built on top of vows, a test library for node.js. Running your APIeasy tests is simple: just use the vows test runner:

  vows your-test-suite.js

More documentation

There is more information available on APIeasy at the GitHub project page. Most of the documentation is written using the literate programming tool, docco. Read the source code here: http://indexzero.github.com/api-easy. APIeasy is a work in-progress, so submit your issues and feedback. Your feedback is valuable.

These workshops will be in: New York, Boston, and Portland ( before NodeConf ) on the following dates:

• April 23 : New York, New York
• April 30 : Boston, Massachusetts
• May 4th : Portland, Oregon

The training workshops will feature members of the Nodejitsu development team walking through a full day program that will take you from the basics of async programming in JavaScript all the way to running an entire Node.js stack in production.

Morning

• From the Browser to the Server
• Characteristics of Async Programming
• Buffers, Streams, and EventEmitters
• Speaking HTTP like a Native
• File System
• Assertions
• Async Iteration

Afternoon

• Getting started with npm
• Lets build a Web Service
• Routing with Journey
• Relax! with Cradle and Resourcer
• Serve Static Assets with node-static
• Handle session and cookies
• Logging with Winston
• Get Real-(time) with Socket.io
• Testing with Vows
• Go Prod with Forever and Cluster

We look forward to helping get as many developers production ready with Node.js as we can! We'll be answering any Node questions you can think up and we'll be sticking around after each workshop to hack and possibly get some tasty drinks!

For those of you who are planning to be in Portland for NodeConf or JSConf this year, I highly recommend you try to make it to our Oregon session. We'll have the entire team there, some special guests, and I'll even be giving a presentation on the basics of building and publishing open-source node modules.

If you can't make it to any of these three node.js training sessions, don't fret! There will be more in the future! If you'd like Nodejitsu to come to your event or company we'd be glad to start a dialog! Feel free to reach out to us at info@nodejitsu.com anytime!

Until the next batch of amazing news....Ninja Out!

• On-site @ Nodejitsu Corporate offices in Union Square, Manhattan ( at General Assembly )
• On-site @ Nodejitsu's West Coast Hack Haus in Lower Haight, San Francisco
• Telecommute ( international may apply )

Nodejitsu is a platform as a service and marketplace for Node.js applications. We build a lot of cool and complex software using cutting edge technologies and we maintain a lot of open-source projects. Check out our github page or hop into #nodejs on irc.freenode.net to find out more. Compensation will be a competitive hourly rate based on experience. Full-time and part-time are available.

Responsibilities:

• Become a master of Node.js and JavaScript
• Help maintain and contribute to Nodejitsu's open-source projects
• Build your own open-source projects
• Work directly on Nodejitsu's platform and production stack
• Help promote Node.js through informative blog posts and tutorials

Requirements:

• Familiarity with the basic tenants of software development
• Experience building a full-stack web application ( front-end, http server, back-end, database)
• Intermediate knowledge of JavaScript, DOM, HTML, CSS
• Ability to work at least 20 hours a week

Preferred Experience:

• Experience building Node.js applications or modules
• Active Github user
• Intermediate knowledge of Node.js
• Intermediate knowledge of linux systems administration
• Experience with community building / blogging

If you are interested please send an email to hiring@nodejitsu.com and be sure to use the one of following subjects in your email to let us know which location you are applying for:

Nodejitsu Internship - NYC

Nodejitsu Internship - SF

Nodejitsu Internship - Telecommute

We look forward to hearing from the next batch of potential Nodejiterns!

1. cradle: A high-level, caching, CouchDB library for Node.js
2. findit: Walk a directory tree in node.js
3. node_redis: Redis client for node
4. node-static: RFC2616 compliant HTTP static-file server module, with built-in caching.
5. optimist: Light-weight option parsing for node.js
6. xml2js: Simple XML to JavaScript object converter.

cradle: A high-level, caching, CouchDB library for Node.js

If you're using CouchDB you should be using cradle. Cradle stands above the other CouchDB libraries in the node.js community: it has a robust LRU (least recently used) cache, bulk document processing, and a simple and elegant API:

//
// Create a connection
//
var conn = new(cradle.Connection)('http://living-room.couch', 5984, {
  cache: true,
  raw: false
});

//
// Get a database
//
var database = conn.database('newyorkcity');

//
// Now work with it
//
database.save('flatiron', {
  description: 'The neighborhood surrounding the Flatiron building',
  boundaries: {
    north: '28 Street',
    south: '18 Street',
    east: 'Park Avenue',
    west: '6 Avenue'
  }
}, function (err, res) {
  console.log(res.ok) // True
});

findit: Walk a directory tree in Node.js

A common set of problems that I see on the nodejs mailing list are advanced file system operations: watching all the files in a directory, enumerating an entire directory, etc. Recently, when working on my fork of docco to respect directory structure in the documentation produced I needed such a feature. It was surprisingly easy:

var findit = require('findit');

findit.find('/dir/to/walk', function (file) {
  //
  // This function is called each time a file is enumerated in the dir tree
  //
  console.log(file);
});

node_redis: Redis client for Node.js

There have been a lot of redis clients released for node.js. The question has become: which client is the right one to use? When selecting an answer to this question for any library you want to look for a few things including: the author, the recent activity, and the number of followers on GitHub. In this case the author is Matt Ranney, a member of the node.js core team. The most recent commit was yesterday, and the repository has over 300 followers.

Redis is really fast, and extremely useful for storing volatile information like sessions and cached data. Lets take a look at some sample usage:

var redis = require("redis"),
    client = redis.createClient();

client.on("error", function (err) {
  console.log("Error " + err);
});

client.set("string key", "string val", redis.print);
client.hset("hash key", "hashtest 1", "some value", redis.print);
client.hset(["hash key", "hashtest 2", "some other value"], redis.print);
client.hkeys("hash key", function (err, replies) {
  console.log(replies.length + " replies:");
  replies.forEach(function (reply, i) {
      console.log("    " + i + ": " + reply);
  });
  client.quit();
});

node-static: RFC2616 compliant HTTP static-file server module, with built-in caching

I bet you're wondering "What the $%^@ is RFC2616?" RFC2616 is the standards specification for HTTP 1.1, released in 1999. This spec is responsible for outlining how (among other things) files should be served over HTTP. Thus, when choosing a node.js static file server, its important to understand which libraries are standards compliant and which are not: node-static is. In addition, it has some great built-in caching which will speed up your file serving in highly concurrent scenarios.

Using node-static is easy, lets make a static file server in 7 lines of Javascript:

var static = require('node-static');

//
// Create a node-static server instance to serve the './public' folder
//
var file = new(static.Server)('./public');

require('http').createServer(function (request, response) {
  request.addListener('end', function () {
    //
    // Serve files!
    //
    file.serve(request, response);
  });
}).listen(8080);

optimist: Light-weight option parsing for Node.js

One of the great things about node.js is how easy it is to write (and later publish with npm) simple command-line tools in Javascript. Clearly, when one is writing a command line tool one of the most important things is to have a robust command line options parser. Our library of choice for this at Nodejitsu is optimist by substack.

Lets take a look at a sample CLI script reminiscent of FizzBuzz:

#!/usr/bin/env node
var argv = require('optimist').argv;

if (argv.rif - 5 * argv.xup > 7.138) {
  console.log('Buy more riffiwobbles');
}
else {
  console.log('Sell the xupptumblers');
}

Using this CLI script is easy:

$ ./node-optimist.js --rif=55 --xup=9.52
Buy more riffiwobbles

$ ./node-optimist.js --rif 12 --xup 8.1
Sell the xupptumblers

This library has support for -a style arguments and --argument style arguments. In addition any arguments passed without an option will be available in argv._. For more information on this library check out the repository on GitHub.

xml2js: Simple XML to Javascript object converter

Writing clients in node.js for APIs that expose data through JSON is almost too easy. There is no need for a complex, language-specific JSON parsing library that one might find in languages such as Ruby or Python. Just use the built-in Javascript JSON.parse method on the data returned and voila! you've got native Javascript objects.

But what about APIs that only expose their data through XML? You could use the native libxmljs module from polotek, but the overhead of dealing with individual XML nodes is non-trivial and (in my opinion) can lead to excess complexity. There is another, simpler option: the lesser known xml2js library available on npm and GitHub.

Lets suppose that we had some XML (/me dies a little inside):

<?xml version="1.0" encoding="UTF-8"?>
<root>
  <child foo="bar">
    <grandchild baz="fizbuzz">grandchild content</grandchild>
  </child>
  <sibling>with content!</sibling>
</root>

Parsing this using xml2js is actually surprisingly easy:

var fs = require('fs'),
    eyes = require('eyes'),
    xml2js = require('xml2js');

var parser = new xml2js.Parser();

parser.on('end', function(result) {
  eyes.inspect(result);
});

fs.readFile(__dirname + '/foo.xml', function(err, data) {
  parser.parseString(data);
});

The output we would see is:

{
  child: {
    @: { foo: 'bar' },
    grandchild: {
      #: 'grandchild content',
      @: { baz: 'fizbuzz' }
    }
  },
  sibling: 'with content!'
}

If you haven't already noticed, xml2js transforms arbitrary XML to JSON in the following way:

• All entity tags like <child> become keys in the corresponding JSON.
• Simple tags like <sibling>with content</sibling> become simple key:value pairs (e.g. sibling: 'with content!')
• More complex tags like <child>... and <grandchild>... become complex key:value pairs where the value is an Object literal with two important properties:
  1. @: An Object representing all attributes on the specified tag
  2. #: Any text content for this XML node.

This simple mapping can greatly simplify the XML parsing logic in your node.js application and is worth checking out if you ever have to deal with the three-headed dog we all love to hate.

Just getting started

This is the first in a series of articles where we will outline at a high level the best-of-the-best for modules, libraries and techniques in node.js that you should be aware of. If you're interested in writing your own node.js modules and publishing them to npm, check out isaacs new article: How to Module over at howtonode.org.

( terrible camera phone shot )

The West Coast Hack Haus is located in Lower Haight, San Francisco. It's super spacious and we've got plenty of couches for wayward programmers visiting from out of town. Anytime, day or night, is a good time for a JavaScript party! If anyone wants to come to hang out or work on some code, just shoot me a message over Github or in the node.js IRC room.

If anyone knows any good meetups in the Bay Area, please reach out. I'd love to attend and I'm always ready to give a lightning talk :-)

Let the JavaScript Party Begin!

The problem

Today, most templating requires you to pepper your markup with 'stubs' or 'placeholders'. Like Java JSPs or Django Templates. It may remind you of the 90's...

<div id='<%=futureData%>'>foo</div>

<ul>{%for d.length}<li id='{%d.futureData}'></li>{%end for}</ul>

It's non-standard, and it's a hack. Blending decision making into your presentation layout will lead to maintenance problems as the complexity of your program grows. As well, it will become more challenging for designers to manipulate and understand the new aspects and rules of the front-end they are working with. Why did anyone ever do this? It's just the evolution of web frameworks. In the beginning, web server frameworks treated html files like raw strings because they didn't have a working DOM. Here's an example...

<div id='foo'><%=futureDataGoesHere%></div>

<div id='foo'>some new value</div>

<%=if(userAge=='35'){>
  <div class='<%=whyIsTheCSSborked%>'></div>
<%=}>

The solution

If the web server had a working DOM, we could maintain the purity of the document and the node we want to manipulate would be accessible to our externalized decision-making code. We should have the same abilities on the server as on the client...

Markup...

<div id='myId'></div>

$("#myId").html("blam-o!!");

Why is that good? It's standards compliant. No workarounds like <%=foo%> or {$bla}. It promotes portable code by decoupling presentation from decision making. Can this be really done? Yes!

Node.js has a working DOM, its called JSDOM. This in concert with Weld, our data/DOM mapping tool, is all you need to build markup that is populated with data and ready to serve. Let's take a look at these tools in action. First we'll load jQuery, then get some data. It doesn't matter where the data comes from. For simplicity's sake let's just use an array of objects.

Welding data to the DOM.

<ul class="contacts">
  <li class="contact">
    <span class="name">My Name</span>
    <p class="title">Leet Developer</p>
  </li>
</ul>

<ul class="contacts">
  <li class="contact">
    <span class="name">Paolo</span>
    <p class="title">Code Slayer</p>
  </li>
  <li class="contact">
    <span class="name">Elijah</span>
    <p class="title">Code Pimp</p>
  </li>  
</ul>

<ul class="contacts">
  <li class="contact">
    <span class="name">Hello my name is <span class="firstAndLast">My Name</span></span>
    <p class="title">Leet Developer</p>
  </li>
</ul>

var data = [
  { 
    name: 'Paulo',  
    title: 'code exploder' 
  },
  { 
    name: 'Elijah', 
    title: 'code pimp' 
  }
];

weld('.contact', data, { 
  bind: { 
    'name': '.firstAndLast', 
    'title': '.title' 
  } 
});

<ul class='contacts'>
  <li class='contact'>
    <span class='name'>Hello my name is <span class='firstAndLast'>hij1nx</span></span>  
    <p class='title'>code slayer</p>
  </li>
  <li class='contact'>
    <span class='name'>Hello my name is <span class='firstAndLast'>tmpvar</span></span>  
    <p class='title'>code pimp</p>
  </li>  
</ul>

1. An instant message application that can be included on any webpage with one line of JavaScript
2. A chat room application that can be included on any webpage with one of JavaScript
3. A real-time multiuser sketchpad
4. A file and media drop box
5. A streaming Twitter client
6. A blogging / Content Management System
7. An online project management tool with Vows / Cucumber integration.
8. A social bookmarking application
9. A browser based hackable and family friendly massive multi-player game.
10. A Micro-wiki application with distributed version control

Now before we dive into each proposed application, let's set a few ground rules. Each application needs to be full-featured, open-source, and most importantly, easily configurable and deployable. The goal is for any newbie developer to be able to start using and customizing any of these applications with minimal effort either locally or on a Node.js hosting platform such as Nodejitsu. I also tried to keep the scope of this list limited to applications which could hit version v1.0.0 in a few months with a small team of developers.

The list

1. Instant Messaging Application

Any serious JavaScript developer has probably been asked to build a "facebook chat" in the past few years. There are even some services which provide this as a SaAS. The best current solution I've seen for node.js so far is AjaxIM by Joshua Gross. The initial version relied on PHP, but it appears they are moving away from this dependency. It looks like a great start, but like most node applications it's still under active development.

2. A Chat Application

Doing a Chat application is the hello world of node's http module. There are many many solutions that do this already, but none of them are full-featured or mature. There needs to be a concerted effort to push one of the existing libraries forward, or simply start a new code base with the recent advancements to node's core and libraries like socket.io

3. A browser based real-time multiuser sketchpad

There are a lot of really great things we can do with node.js and HTML5's canvas. Mr. Doob released a multi-user sketchpad which he created over a weekend. I chatted with Mr. Doob about releasing the source, but there are a few security concerns he'd have to address before it could be stable. Real-time sketching collaboration tools have been around for a while, but there has never been an easy to hack open-source one that works using a standard browser. This type of tool could be used for a variety of purposes and industries across almost any device.

4. A file and media drop box

Think what drop.io was, but open-source. You can setup a private file server somewhere, or you pay for hosting on a node.js hosting service like Nodejitsu. There would be drag and drop file transfers and a nice web gui to view / manage files and media assets.

5. A streaming Twitter client

I'm not a huge fan of Twitter, but a lot of other people are. Building a custom Twitter client can be a good starting point for new developers. I'm also told developers get requests to build custom Twitter clients. Streamie by Malte Ubl is a hackable node.js Twitter client which is pretty solid. I'd recommend checking it out.

6. A blogging / CMS engine

Node needs a goto Content Management System / blogging engine. Ruby on Rails has Radiant, PHP has Wordpress, node needs it's own. The Nodejitsu blog is a custom build of Tim Caswell's blogging software, Wheat. I applaud Tim for his great work in building Wheat, it's a great little git based blogging engine. The problem is, it's currently just a blogging engine. Wheat doesn't have any of the features you'd want in a full-featured CMS or Blog. Should these features be added to Wheat? The answer is probably no, but the point is Wheat is a great example of the first steps in building such an application.

7. An online project management tool with Vows / Cucumber integration.

This is more a techie tool, but it's an essential tool for anyone who wants to do serious node.js project development. This is about building a solid suite of tools to be able to quickly develop node.js applications. The Nodejitsu team has started an initiative for this called Prenup ( see related blog post: http://blog.nodejitsu.com/kyuri-and-prenup-our-node-knockout-entries ), but like most node projects, it's still alpha and could use a bit of work. I urge anyone who is interested in agile software development in node to check out Kyuri and Prenup.

8. A social bookmarking site

Think Reddit or Digg. Reddit actually is already open-source, but is written in Python and is not geared towards real-time at all. There is a unique opportunity to rethink these services in the context of real-time applications. In many ways, social bookmarking has replaced traditional forum software creating larger and more agile online communities. An easy to customize real-time social bookmarking application in node will lead to a whole new generation of communities. I haven't seen any good initiatives for this yet, but I have been working on a few prototypes.

9. A browser based and family friendly massive multi-player game

What node could really use is a browser based game that was easy to use, customize, and be G rated enough to used as an educational tool. There is Orona, a tank game by Stephan Kochen, but it's very alpha. Tim Caswell has done some experiments with his Adventure game and world-db, but these are just experiments. Someone could step up and make a two dimensional tile game where you went around and collected items / solved puzzles with other players. This would make a great educational tool as it would be easy for anyone ( such as a teacher ) to setup a private game server ( for their students ), and then begin to dive into customizing the game as a way of teaching programming.

10. A micro-wiki application with distributed version control

Node needs a generic Wiki application that is both small in size and able to maintain versions of documents across multiple instances. This will allow for a decentralized Wikipedia which could also be sharded onto local storage, and to smaller devices like mobile phones. TiddlyWiki is a neat project by Jeremy Ruston which is built on this idea. Some work has been done with node and TiddyWiki, but I couldn't find anything about using node to synchronize multiple TiddlyWiki instances.

Conclusion

It's easy to say that something needs to get done, but without execution an idea is worth little. I can only hope that by listing these applications, someone will be inspired to build something. If they happen to build a useful application in node.js, I know that we at Nodejitsu will be happy to support it on our platform.

ninja out

This article will walk through the code that I presented to build a simple RESTful bookmarking API using node.js and:

• Journey: A liberal JSON-only HTTP request router for node.js
• Cradle: A high-level, caching, CouchDB library for Node.js
• Winston: A multi-transport async logging library for node.js
• Optimist: Light-weight option parsing for node.js

Getting Started

The first step to doing anything with node.js is creating a server to accept incoming HTTP requests. You can also create a TCP server or work with UDP, but I won't be going into that here. So what does such a server look like?

var http = require('http'),
    winston = require('winston');

/**
 * Creates the server for the pinpoint web service
 * @param {int} port: Port for the server to run on
 */
exports.createServer = function (port) {
  var server = http.createServer(function (request, response) {
    var data = '';
    
    winston.info('Incoming Request', { url: request.url });
    
    request.on('data', function (chunk) {
      data += chunk;
    });
    
    response.writeHead(501, { 'Content-Type': 'application/json' });
    response.end(JSON.stringify({ message: 'not implemented' }));
  });
  
  if (port) {
    server.listen(port);
  }
  
  return server;
};

The above example is simple (almost trivial): it exports a server that reads the request body, logs the request using winston, and responds with the appropriate HTTP response code (501 - Not Implemented). Not very exciting but enough to get us started.

Running the development server

The development server lives in bin\server and has been configured so that we can run the development server at the various stages of development outlined in this tutorial. Running it is simple, just remember to pass the -t argument which specifies which directory under /lib to use for the server:

• 00getting-started: The full source code from 'Getting Started'
• 01routing: The full source code from 'Adding some Routes'
• 02couchdb: The full source code from 'Interacting with CouchDB'
• 03authentication: The full source code from 'Adding HTTP Basic Auth'

  $ bin/server -t 00getting-started
  Pinpoint demo server listening for 00getting-started on http://127.0.0.1:8000
  4 Feb 17:59:22 - info: Incoming Request url=/

Adding some Routes

Now that we have a server that tells the world we haven't done anything it's time to think about what our application does:

• List: GET to /bookmarks should respond with a list of bookmarks
• Create: POST to /bookmarks should create a new bookmark
• Show: GET to /bookmarks/:id should respond with a specific bookmark
• Update: PUT to /bookmarks/:id should update a specific bookmark
• Destroy: DELETE to /bookmarks/:id should delete a specific bookmark

Pretty simple right? Absolutely. To accomplish this routing tasks we are going to use Journey. Journey is a great library with a lot of features. 0.3.1 was just released, which we will be taking advantage of in this post.

exports.createRouter = function () {
  return new (journey.Router)(function (map) {
    map.path(/\/bookmarks/, function () {
      //
      // LIST: GET to /bookmarks lists all bookmarks
      //
      this.get().bind(function (res) {
        res.send(501, {}, { action: 'list' });
      });
      
      //
      // SHOW: GET to /bookmarks/:id shows the details of a specific bookmark 
      //
      this.get(/\/([\w|\d|\-|\_]+)/).bind(function (res, id) {
        res.send(501, {}, { action: 'show' });
      });
      
      //
      // CREATE: POST to /bookmarks creates a new bookmark
      //
      this.post().bind(function (res, bookmark) {
        res.send(501, {}, { action: 'create' });
      });
      
      //
      // UPDATE: PUT to /bookmarks updates an existing bookmark
      //
      this.put(/\/([\w|\d|\-|\_]+)/).bind(function (res, bookmark) {
        res.send(501, {}, { action: 'update' });
      });
      
      //
      // DELETE: DELETE to /bookmarks/:id deletes a specific bookmark
      //
      this.del(/\/([\w|\d|\-|\_]+)/).bind(function (res, id) {
        res.send(501, {}, { action: 'delete' });
      });
    });
  }, { strict: false });
};

The above code generates a Journey router that matches the routes we outlined using regular expressions. In each route, we respond with 501 Not Implemented and the corresponding action so that we can be sure we've hit the correct route.

We use the map.path syntax to scope the subsequent routes behind /bookmarks. The changes that need to be made to our development server are minimal. We just need to add code to create our router and later it to route our request with the associated request body within our HTTP server:

request.on('end', function () {
  //
  // Dispatch the request to the router
  //
  router.route(request, body, function (route) {
    response.writeHead(route.status, route.headers);
    response.end(route.body);
  });
});

You can view the entire file on GitHub here.

Testing Routes using http-console

One of the things I did differently in this demo than I do in my own projects is that there are no vowsjs tests. I choose not to include tests in this demo because the additional overhead of understanding how a particular test framework works seemed a little high for the complete beginner.

The alternative was to use http-console: a simple, intuitive HTTP REPL. Getting http-console is easy to install using npm:

  [sudo] npm install http-console

So lets fire-up http-console for an interactive session a couple of our newly minted routes:

  $ http-console http://127.0.0.1:8000
  > http-console 0.5.1
  > Welcome, enter \help if you're lost.
  > Connecting to 127.0.0.1 on port 8000.

  http://127.0.0.1:8000/> GET /bookmarks
  HTTP/1.1 501 Not Implemented
  Date: Sat, 05 Feb 2011 02:57:46 GMT
  Server: journey/0.3.0
  Content-Type: application/json
  Content-Length: 17
  Connection: close

  { action: 'list' }
  http://127.0.0.1:8000/> GET /bookmarks/foobar
  HTTP/1.1 501 Not Implemented
  Date: Sat, 05 Feb 2011 02:57:52 GMT
  Server: journey/0.3.0
  Content-Type: application/json
  Content-Length: 17
  Connection: close

  { action: 'show' }
  http://127.0.0.1:8000/>

We made two request to /bookmarks and /bookmarks/foobar respectively and got back 501 in both cases with valid JSON representing the specified action that is not yet implemented. We also got back the application/json header which was automatically set for us by Journey.

Interacting with CouchDB

Interacting with a persistent data store is a must have for any webservice or web application. At Nodejitsu we use CouchDB and our library of choice is cradle. We will define a Bookmark resource with a couple of methods for performing basic CRUD on our bookmark object. Before we get to that we need to configure our Couch with a Design Document for Bookmark resources. If you want to learn more about Design Doucments see CouchDB: The Definitive Guide.

var cradle = require('cradle');
 
var setup = exports.setup = function (options, callback) {
  // Set connection configuration
  cradle.setup({
    host: options.host || '127.0.0.1',
    port: 5984,
    options: options.options,
  });
  
  // Connect to cradle
  var conn = new (cradle.Connection)({ auth: options.auth }),
      db = conn.database(options.database || 'pinpoint-dev');
      
  if (options.setup) {
    initViews(db, callback);
  }
  else {
    callback(null, db); 
  }
};

var initViews = exports.initViews = function (db, callback) {
  var designs = [
    {
      '_id': '_design/Bookmark',
      views: {
        all: {
          map: function (doc) { if (doc.resource === 'Bookmark') emit(doc._id, doc) }
        },
        byUrl: {
          map: function (doc) { if (doc.resource === 'Bookmark') { emit(doc.url, doc); } }
        },
        byDate: {
          map: function (doc) { if (doc.resource === 'Bookmark') { emit(doc.date, doc); } }
        }
      }
    }
  ];
  
  db.save(designs, function (err) {
    if (err) return callback(err);
    callback(null, db);    
  });
};

The above code requires cradle, configures it with the remote host, port and authentication (if required). If options.setup is set then it asynchronously creates the Design Document in CouchDB and later responds with the database connection db.

Now that we have a connection to CouchDB for our application, we can go ahead and create a Bookmark resource that consumes the connection. Within this resource we want to define functions for each of the CRUD operations we've outlined: create, show, list, update and destroy.

/**
* Constructor function for the Bookmark object..
* @constructor
* @param {connection} database: Connection to CouchDB
*/
var Bookmark = exports.Bookmark = function (database) {
  this.database = database;
};

/**
* Lists all Bookmarks in the database
* @param {function} callback: Callback function
*/
Bookmark.prototype.list = function (callback) {
  this.database.view('Bookmark/all', function (err, result) {
    if (err) {
      return callback(err);
    }
    
    callback(null, result.rows.map(function (row) { return row.value }));
  })
};

/**
* Shows details of a particular bookmark
* @param {string} id: ID of the bookmark
* @param {function} callback: Callback function
*/
Bookmark.prototype.show = function (id, callback) {
  this.database.get(id, function (err, doc) {
    if (err) {
      return callback(err);
    }
    
    callback(null, doc);
  });
};

/**
* Creates a new bookmark with the specified properties
* @param {object} bookmark: Properties to use for the bookmark
* @param {function} callback: Callback function
*/
Bookmark.prototype.create = function (bookmark, callback) {
  bookmark._id = helpers.randomString(32);
  bookmark.resource = "Bookmark";
  
  this.database.save(bookmark._id, bookmark, function (err, res) {
    if (err) {
      return callback(err);
    }
    
    callback(null, bookmark);
  })
};

/**
* Updates a new bookmark with the specified id and properties
* @param {object} bookmark: Properties to update the bookmark with
* @param {function} callback: Callback function
*/
Bookmark.prototype.update = function (id, bookmark, callback) {
  this.database.merge(id, bookmark, function (err, res) {
    if (err) {
      return callback(err);
    }
    
    callback(null, true);
  });
};

/**
* Destroys a bookmark with the specified ID
* @param {string} id: ID of the bookmark to destroy
* @param {function} callback: Callback function
*/
Bookmark.prototype.destroy = function (id, callback) {
  var self = this;
  this.show(id, function (err, doc) {
    if (err) {
      return callback(err);
    }
    
    self.database.remove(id, doc._rev, function (err, res) {
      if (err) {
        return callback(err);
      }

      callback(null, true);
    });
  });
};

I won't go into the details of how each of these methods work, but rest assured that they do. It's all very basic usage for cradle, so if you're interested in the specifics I invite you to read the documentation on the cradle GitHub page.

So now that we've configured CouchDB and we have a Bookmark resource, we need to connect our resource to the Journey router that we defined earlier.

exports.createRouter = function (resource) {
  return new (journey.Router)(function (map) {
    map.path(/\/bookmarks/, function () {
      //
      // LIST: GET to /bookmarks lists all bookmarks
      //
      this.get().bind(function (res) {
        resource.list(function (err, bookmarks) {
          if (err) {
            return res.send(500, {}, { error: err.error });
          }
          
          res.send(200, {}, { bookmarks: bookmarks });
        });
      });
      
      //
      // SHOW: GET to /bookmarks/:id shows the details of a specific bookmark 
      //
      this.get(/\/([\w|\d|\-|\_]+)/).bind(function (res, id) {
        resource.show(id, function (err, bookmark) {
          if (err) {
            return res.send(500, {}, { error: err.error });
          }
          
          res.send(200, {}, { bookmark: bookmark });
        });
      });
      
      //
      // CREATE: POST to /bookmarks creates a new bookmark
      //
      this.post().bind(function (res, bookmark) {
        resource.create(bookmark, function (err, result) {
          if (err) {
            return res.send(500, {}, { error: err.error });
          }
          
          res.send(200, {}, { bookmark: result });
        });
      });
      
      //
      // UPDATE: PUT to /bookmarks updates an existing bookmark
      //
      this.put(/\/([\w|\d|\-|\_]+)/).bind(function (res, id, bookmark) {
        resource.update(id, bookmark, function (err, updated) {
          if (err) {
            return res.send(500, {}, { error: err.error });
          }
          
          res.send(200, {}, { updated: updated });
        });
      });
      
      //
      // DELETE: DELETE to /bookmarks/:id deletes a specific bookmark
      //
      this.del(/\/([\w|\d|\-|\_]+)/).bind(function (res, id) {
        resource.destroy(id, function (err, destroyed) {
          if (err) {
            return res.send(500, {}, { error: err.error });
          }
          
          res.send(200, {}, { destroyed: destroyed });
        });
      });
    });
  }, { strict: false });
};

In each of the new routes, we send the appropriate request data to the Bookmark resource, and asynchronously respond with the appropriate HTTP response code when complete. In the event of an error, we always send 500 Internal Server Error with the error message.

Adding HTTP Basic Auth

The main focus of Journey 0.3.0 was to add a feature where the programmer could specify a filter function that takes the request and body. This filter function will intercept requests before they are passed to any route handler. If the filter function returns a pre-defined Journey error, the router will short-circuit and respond with the status code. We will use this feature to define a filter function that performs HTTP Basic Auth.

var auth = exports.auth = {
  username: 'admin',
  password: 'password',
  basicAuth: function (request, body, callback) {
    var realm = "Authorization Required",
        authorization = request.headers.authorization;
    
    if (!authorization) {
      return callback(new journey.NotAuthorized("Authorization header is required."));
    }

    var parts       = authorization.split(" "),           // Basic salkd787&u34n=
        scheme      = parts[0],                           // Basic
        credentials = base64.decode(parts[1]).split(":"); // admin:password

    if (scheme !== "Basic") {
      return callback(new journey.NotAuthorized("Authorization scheme must be 'Basic'"));
    }
    else if(!credentials[0] && !credentials[1]){
      return callback(new journey.NotAuthorized("Both username and password are required"));
    }
    else if(credentials[0] !== auth.username || credentials[1] !== auth.password) {
      return callback(new journey.NotAuthorized("Invalid username or password"));
    }
    
    // Respond with no error if username and password match
    callback(null);
  }
};

We can set this method on the Journey router by passing it in the options hash. Any routes that we wish to be behind the authentication filter need to be wrapped in a call to map.filter(function () { ... })

exports.createRouter = function (resource) {
  return new (journey.Router)(function (map) {
    //
    // Resource: Bookmarks
    //
    map.path(/\/bookmarks/, function () {
      //
      // Authentication: Add a filter() method to perform HTTP Basic Auth
      //
      map.filter(function () {
        //
        // All of the previous routes we had go in here, but they
        // are now behind HTTP Basic Auth
        //
      });
    });
  }, { 
    strict: false,
    filter: helpers.auth.basicAuth 
  });
};

Wrapping up

Now that we have completed our web service, lets fire up http-console for an interactive session with our Bookmark resource.

  $ http-console http://127.0.0.1:8000
  > http-console 0.5.1
  > Welcome, enter \help if you're lost.
  > Connecting to 127.0.0.1 on port 8000.

  http://127.0.0.1:8000/> Authorization: Basic YWRtaW46cGFzc3dvcmQ
  http://127.0.0.1:8000/> \json
  http://127.0.0.1:8000/> \headers
  Accept: /
  Authorization: Basic YWRtaW46cGFzc3dvcmQ
  Content-Type: application/json
  http://127.0.0.1:8000/> POST /bookmarks
  ... { "url": "http://nodejs.org" }
  HTTP/1.1 200 OK
  Date: Sat, 05 Feb 2011 05:38:47 GMT
  Server: journey/0.3.0
  Content-Type: application/json
  Content-Length: 77
  Connection: close

  {
      bookmark: {
          url: 'http://nodejs.org',
          _id: 'xnIgT8',
          resource: 'Bookmark'
      }
  }
  http://127.0.0.1:8000/> GET /bookmarks/xnIgT8
  HTTP/1.1 200 OK
  Date: Sat, 05 Feb 2011 05:39:01 GMT
  Server: journey/0.3.0
  Content-Type: application/json
  Content-Length: 121
  Connection: close

  {
      bookmark: {
          url: 'http://nodejs.org',
          _id: 'xnIgT8',
          resource: 'Bookmark',
          _rev: '1-cfced13a45a068e95daa04beff562360'
      }
  }
  http://127.0.0.1:8000/> GET /bookmarks
  HTTP/1.1 200 OK
  Date: Sat, 05 Feb 2011 05:39:05 GMT
  Server: journey/0.3.0
  Content-Type: application/json
  Content-Length: 369
  Connection: close

  {
      bookmarks: [
          {
              _id: 'xnIgT8',
              _rev: '1-cfced13a45a068e95daa04beff562360',
              url: 'http://nodejs.org',
              resource: 'Bookmark'
          }
      ]
  }

I hope this has been helpful for those of you looking to get started with node.js. Check out the rest of our blog for more advanced libraries and tutorials. Come back soon for more on the Art of Nodejitsu.