There are a number of reasons we are thrilled to bring IrisCouch into the Nodejitsu team:

1. Importance of Database-as-a-Service: Data is everything today. It drives forward development of both a technology and a business. Tightly integrating your database with your application can mean the difference between a successful or failed product. We have already been working with IrisCouch and our other database partners (like MongoLab) to ensure that the databases they created are co-located with our datacenter targets, and this acquisition will only improve that effort for Redis and Apache CouchDB.

2. Commitment to node.js: Node.js has been in our DNA since Nodejitsu was founded three years ago. Everything we’ve created is built on Node.js, Apache CouchDB, and Redis. One of the best parts about node.js is npm (the node package manager). It makes publishing and installing node modules as fun as it is to write them.

If you’re using npm today, you’re already using IrisCouch. But don’t take our word for it, listen to what Isaac Z. Schlueter, creator of npm and leader of the Node.js project had to say:

"IrisCouch has been the official host of the public npm registry since they were part of CouchOne (now Couchbase). They have provided top-notch service to the Node.js and npm community since day one, even going so far as to make modifications to Apache CouchDB itself for our security needs. I cannot recommend them highly enough."

Through the acquisition of IrisCouch, every node.js developer in the world will be using Nodejitsu every day. Our commitment to node.js and npm remains steadfast, you can count on us to ensure the public npm registry remains open, available, and reliable.

3. The Team: As part of the acquisition, both Jeff Jackson (IrisCouch CEO) and Jason Smith (IrisCouch CTO) will be joining the leadership team at Nodejitsu. Adding this wealth of high-caliber industry expertise to Nodejitsu will ensure that we are well equipped for the road ahead as we begin hyper-growth.

What’s going to change?

As part of this acquisition we will be rolling IrisCouch’s Database-as-a-Service for CouchDB and Redis directly into our public Platform-as-a-Service. This means that our users will be able to deploy their applications and databases from the same set of tools all backed by node.js. If you’re an existing IrisCouch user you will be notified and given ample time to migrate your IrisCouch account into a Nodejitsu account.

In addition to tightly integrating Database-as-a-Service into the Nodejitsu Public Cloud, we will be sunsetting their IrisNpm product over the next few months. If you’re an IrisNpm user today, you will be given ample time to ensure your private npm packages can be transferred.

Sunsetting IrisNpm was a tough decision but it will allow us to focus on offering the best Enterprise npm product to node.js power users. Interested in how a private npm workflow can help your team? Get in touch with our sales team today!

The road ahead

IrisCouch’s collection of technology, services, and talent join our recent announcement of OpsMezzo, fortifying the breadth of our offering to both developers and businesses.

Jason Smith marked this occasion in his own words by saying simply: “this is the first step of a thousand mile journey.”

Apache CouchDB, CouchDB, and the project logo are trademarks of The Apache Software Foundation

Join us on Tuesday 9:00 AM - 10:30 AM PT for Nodestack: State of the Stack, where Charlie and Nuno will be holding it down with Isaac Schleuter and Aaron Heckman:

#

Charlie will be tag teaming with izs to discuss 0.10. Then Aaron from 10gen will be discussing Replica Sets & Text Search in MongoDB 2.4, and finally, Nuno will close out the session with a talk about running Production Node on SmartOS.

There are still a few tickets left. See you there!

Initially we gave two weeks notice, and scheduled price changes for Wednesday May 15th. Many customers have felt this was not enough time. My team and I heard your feedback, and have pushed the changes out one more week. Wednesday May 22nd is now the day that our new prices will go into effect.

We also received questions regarding the ascending price of our plans. For example, some people have wondered why a Micro plan (1 drone) costs $9 whereas a Small plan (3 drones) costs $33 dollars. This is because our Micro plans (our lowest hosting tier), are being slightly subsidized. We want to continue encouraging experimentation on our PaaS, albeit at a higher price point than before.

In the same spirit of experimentation, Nodejitsu is upholding our commitment to offer a free drone, the equivalent of one Micro plan, to host an open source app to any node.js developer. You simply need to apply through opensource.jit.su.

Nodejitsu's success thus far is due entirely to the support of the node.js community. We understand these price changes have upset some of our customers, but these changes allow us to serve our community sustainably and continue contributing open source software to the node ecosystem. These changes also provide our small but talented team with an opportunity to improve our PaaS, dealing more quickly with production incidents and fixing underlying issues. The goal is to provide our customers with a better product. My team and I are excited to move forward towards a strong future for node, and we look forward to doing so with many members of the node community on our platform.

Ken has been putting great amounts of work into pkgcloud, including fixing Rackspace integration and a huge internal refactor which corrects mistakes we made when designing some of the low level APIs. We are extremely thankful for his help and dedication, so we decided to give him push access to our repository.

Here are few words Ken has to say about himself:

I'm the lead Node.js Developer Advocate at Rackspace, primarily working on our node.js SDKs. Immediately before Rackspace, I was a full-stack node.js developer with Clipboard. I also spent almost 9 years at Microsoft, most recently as a Program Manager in Live Labs working on projects like Seadragon, Photosynth, and Pivot. Most of my time outside of work is spent playing with my kids and enjoying the weather whenever it's nice here in Seattle.

He also published few interesting blog posts about changes he wants to introduce to pkgcloud:

• Cleaning up multi-provider compute methods in pkgcloud
• Plans for the (v0.8?) release of pkgcloud

You should follow Ken on Twitter and subscribe to his blog.

We'd also like to thank Rackspace for giving Ken a chance to work on open source as his full time job. It's great to see companies giving back to the community.

Last week we announced OpsMezzo, a complete solution to provisioning, orchestration and configuration management geared towards enterprise customers. That annoucement reflects our commitment to OpenSource and the node.js community. In fact, more than 80% of the nodejitsu stack is now Open Source.

Our prices are the lowest in the Platform-as-a-Service market. Last year when we announced our pricing model we thought we could make this model sustainable, but this was built on several assumptions that turned out to be wrong. With this new model we will be able to keep our Public Cloud sustainable for years to come.

Please note: the changes we are about to discuss apply only to customers using Individual Plans on Nodejitsu’s platform and will not affect Business Plan customers.

First, the two core assumptions that we made at Nodejitsu when we compiled our current pricing:

1. 64MB of RAM: Our original plan was to give drones on Individual Plans 64MB of RAM, but technical limitations combined with the high expectations of our existing Individual users led us to keep 256MB RAM as standard for all drones.
2. The power of Node.js: Node.js is an amazing technology enabling developers to do a lot with minimal resources. When we decided to only give flexible drone sizes and multi-datacenter access to our Business Plan customers, we assumed that this would be incentive enough to attract our customers who need to handle large volumes of traffic. Turns out node.js is even more powerful than we thought; and as of writing this dozens of businesses are running on our Individual Plans. While its exciting that the combination of node and our technology can deliver these results, this is ultimately hurting Nodejitsu.

With all this in mind, we still realize that we made a commitment to our customers and when restructuring our Individual Plans we wanted to ensure that they are still competitive. With the new prices below Nodejitsu is still the most affordable Platform-as-a-Service on the market.

• Micro (1 drone): $9 / month
• Small (3 drones): $33 / month
• Regular (5 drones): $50 / month
• Large (10 drones): $90 / month

In addition to these pricing changes we will be clarifying some additional limitations for applications running on Individual Plans:

• 3 drone limit: If you’re on an Individual Plan, jitsu apps setdrones 3 is the maximum number of running drones you will be able to set for a single application.
• 10 drones max: If you’re on an Individual Plan, you will be able to run a total of 10 drones maximum. If you need more than that, we reccommend you upgrade to a Business Plan
• One-day retention for Metrics: These changes will go into effect on the same day as our new metrics for CPU, Memory, and HTTP. Customers running on an Individual Plan will have a default retention of one-day.
• One-day retention for Logs: As we improve our logging facilities, we will be increasing the retention around our logs for Business Plans. Individual plans, however, will have one-day of logs retention.
  • Designated support hours: The support provided in #nodejitsu on irc.freenode.net will be staffed by Nodejitsu Support Engineers from 9am - 8pm ET, Monday - Friday. There may be staffers or community support available at other times, but the above hours are when you can rely on live help from Nodejitsu.

While these changes bring limitations in the short term, they allow us operate more responsibly long term. This new pricing and the subsequent changes to our business allow us to provide the same great hosting and to focus on making it better; not only taking the time to field incoming errors but to fix underlying issues. These changes won’t be for everyone, but we look forward to bringing you a better product, and serving many of you for a long time to come.

We’re excited about the future of our Platform-as-a-Service product. By making these changes, difficult as they may be, it will make our product sustainable as we continue to grow and deliver on our promise of the best Node.js Platform-as-a-Service available.

The majority of OpsMezzo is now available as Open-Source. A related Press Release can be found at nodejitsu.com.

Cloud computing has matured dramatically since Amazon Web Services launched in 2006. A dizzying number of vendors have emerged in the market to solve a wide spectrum of specific problems.

These products have matured over the years, but they are built on a fundamentally flawed thesis: the parts are more important than the whole. In others words: organizations are on their own to integrate individual products into a complete system. Yes, there is a large amount of source material available through blogs, tutorials, and training, but the manual integration work remains.

At it's core, this means that organizations running in "the cloud" are dealing with too many vendors to solve simple problems. Vendors using multiple languages, databases, operating systems and other operational internals. While we live in a polyglot world, the operational calculus of running these multi-vendor systems is challenging to even seasoned professionals.

At this point you are probably wondering: "What about PaaS solutions?" PaaS is great for individuals and organizations who do not have (or do not want to have) an operational competency. This concept of "NoOps", while attractive to some, is also fundamentally flawed:

1. Too simple for larger organizations: In order to present users with a set of APIs that are simple enough even for novices massive operational details are hidden. Larger organizations (especially those running their own datacenters) already have an operational competency, so the PaaS abstraction is just too simple.
2. Still too many vendors: Heroku pioneered the concept of "add -ons" and since then every PaaS company has been quick on their heels to copy that model. This leaves their customers paying too much to too many vendors with little-to-no visibility behind the scenes.

We saw these problems two years ago in the early days of Nodejitsu and set out to make OpsMezzo a complete solution built entirely on a single platform: Node.js + CouchDB.

Keep reading to learn more about how OpsMezzo can streamline and simplify your devops workflow. Or if you're interested in trying out OpsMezzo in your organization send an email to sales@nodejitsu to get a free evaluation!

TL;DR?

• Why OpsMezzo?
• A system of actors
• Want more about OpsMezzo?

Why OpsMezzo?

What do we mean by a complete solution?

• Configuration management with quill and composer.
• Provisioning and orchestration with baton and conservatory.
• Monitoring and auto-scaling built on godot.
• Industrial Load balancing built on node-http-proxy.

And more importantly why should you use OpsMezzo?

• Every line of code is battle-tested by powering the nodejitsu.com PaaS
• Language, platform and cloud agnostic
• Deploy anything.
• Write your deployment scripts in any language.
• Deploy to any IaaS cloud using pkgcloud.
• Run on any operating system using system.json
• quill and the backend service it communicates with, composer, are Open-Source.

A system of actors

The name OpsMezzo comes from the musical team intermezzo:

Intermezzo: a short movement or interlude connecting the main parts of the composition.

All of the actors in the OpsMezzo system are designed to work together. In line with our commitment to Open Source every actor is Open Source or built directly on-top of MIT licensed Open Source software.

Examining the diagram above, OpsMezzo is composed of four actors:

• Composer: Modern configuration management backed by system.json.
• Conservatory: Flexible provisioning and state management for your servers on any cloud provider.
• Conductor: Industrial grade load-balancing with WebSocket support.
• Instrument: Powerful monitoring, metrics, and auto-scaling backed by godot and compatible with riemann.

Along with two developer tools:

• baton: A provisioning and distributed SSH tool that talks to conservatory and your cloud provider (AWS, Rackspace, OpenStack, etc).
• quill: A configuration management tool that talks to composer and understands system.json.

By developing all of these tools together OpsMezzo eliminates the months of work necessary to integrate other off-the-shelf software.

Want more about OpsMezzo?

This is just the beginning for OpsMezzo. We couldn't be more excited to have released 12 repositories as Open Source in a single day.

Over the next few weeks we will be releasing all of the documentation, even for actors that are not Open Source). Right now all of the documentation for Open Source libraries (such as quill and composer) can be found at docs.opsmezzo.com.

Interested in trying out OpsMezzo in your organization? Send an email to sales@nodejitsu to get a free evaluation.

In the time in between our outage and today we have overhauled our entire load balancing architecture, also known as Conductor. Below is a detailed look at our old load balancing system and our newly implemented solution.

Previous implementation

The previous implementation illustrated above suffers from several problems:

1. Long startup time: populating the entire cache from CouchDB can take upwards of several minutes.
2. No retry on failure: incoming requests are simply killed with 500 ECONNREFUSED.
3. Lack of consistency: since each balancer maintains it's own _changes listener the state maintained is independent of other balancers.

New implementation

The proposed implementation addresses the problems outlined in the Current implementation by:

1. Reducing startup time: There is no cache built at startup, so balancers will be immediately available to take requests.
2. Retry on failure: requests will be retried when they fail up to a certain limit.
3. Consistent central broker: the Cache Manager is now the authoritative and consistent source for host information and proactively invalidates the in-memory LRU cache on balancers as changes come in from CouchDB.

Put more plainly, our new implementation of Conductor makes us less reliant on memory caching and the risks that come with it. It also moves us to a model where failure to connect with your app initiates a series of automatic retries and self-corrects. This will limit instances of ECONNREFUSED to times when the issue lies within a customers application, rather than our platform. Our team has needed to shelve some things in order to make this a reality, but our reliability is ultimately our top priority, and we believe this will bring a significant improvement to our core infrastructure.

Thanks for hanging in there. We hope customers across the platform will be seeing a more reliable hosting experience.

From a young age, I've been fascinated with making systems do what I want. I fiddled with electronics and I learnt to program at age 7. I left VB for Perl, Perl for Python. I taught myself Linux, discovered the UNIX philosophy of simple, composable parts forming a powerful whole. I found node.js two years ago and loved how working with streams and composing small things was so very natural. Unfortunately, node wasn't as mature and thus I stuck with Python. About half a year ago I rediscovered node.js, the freedom of JavaScript and found the wealth of easily composable small modules in npm. This time, it stuck. (I do my best to help out with npm by triaging GitHub issues, patching small things and merging pull requests, occasionally frightening isaacs in the process.)

My other passion is RepRap, a project aiming to build self-replicating 3D printers. Perhaps our future robot overlords. Two years ago, I hopped into the RepRap IRC channel, with overly ambitious plans for cheaper RepRaps. I've been a regular ever since, helping other people the best I could with the knowledge I had acquired. My enthusiasm got me invited to the first RepRap Prusa Mendel (iteration 1) workshop to help out. I was gifted a set of RepRap parts. I've helped with a number of workshops since and helped develop the Prusa Mendel iteration 3.

Trying to deploy my first serious node.js app, after checking out some of the polyglot clouds and other node.js clouds, I found Nodejitsu. jitsu deploy. Bam. Sensible cloud hosting. That too, stuck. I had a few issues and I was quickly helped out on IRC. I ended up staying there too, helping out people when possible. About three months ago, Maciej said I was doing a great job helping people and talking to servers and asked me if I wanted a job doing this.

I just got past my three month support internship, on board, helping people kick ass on our cloud and working on getting Node 0.10 into the cloud. See you in #nodejitsu!

\o

At this point, we'd like to take the time to explain what went wrong, and give some insight into some of the things we've been doing to make sure these issues don't repeat themselves.

Context

Our unfortunate tale begins last Wednesday, March 6th. In the early hours of the morning, our DevOps team deployed a fairly straightforward change to our load balancing layer. We use CouchDB to store app state, and the CouchDB _changes feed to keep our load balancers up-to-date when apps are created, destroyed, or deployed to new drone servers. We had, up until this point, been using custom code to follow the _changes feed with Node's HTTP API, but had decided to switch to using follow instead. We made the changes, we reviewed the changes, and the changes spent time in our staging environment being tested. Everyone, myself included, was confident that this change would both eliminate the few occasional hiccups we'd been seeing with the _changes feed from time to time, and would also be too straightforward to introduce new problems. So, in the wee hours of Wednesday, we deployed it to production.

The Problems

As anyone who's ever tried to write a serious benchmark of a network service can tell you, it's not easy to replicate large amounts of authentic internet traffic for testing. It's easy to make one computer, or several, make lots and lots of requests to the same host. The real-world experience of having millions of people all over the world making requests to your site or service, however - requests from different operating systems, different hardware, all travelling over completely different network paths - is extremely difficult to replicate accurately. Needless to say, the testing our balancers got in our staging environment did not turn out to be a reasonable facsimile of our usual production load.

In and amongst all of the events which unfolded, there was required travel taking place on Thursday and Friday for two key members of our DevOps team, including my own flight for my move from San Francisco back home to New York.

Wednesday, and most of Thursday, stayed quiet. The trouble with a subtle bug that produces an incorrect state is that it starts out as an occasional, sporadic issue - if thousands of apps are working fine, but one user is seeing a problem, simple deduction suggests that the issue, while not necessarily the user's fault, lies with some aspect of their app that sets it apart from the rest of the apps in question. Since this looked like a few odd users seeing something strange, and not like a large systemic problem, our support engineers began investigating the apps in question rather than our infrastructure.

In the meantime, they offered a simple workaround: if a user runs jitsu apps start on their app, the app will be deployed to a new drone, the database will be updated, and the balancers will receive the new network location for the app. In almost all cases, this returns the app in question to a working state, and gives us time to look at what happened without giving our users serious downtime. Sometime around dawn UTC on Friday, a few of our users who were affected by the current problems cleverly decided to make scripts for themselves, that would run jitsu apps start over and over and over. While I don't fault our users for taking measures to ensure their uptime, this very quickly escalated into a situation of "deploy spam" - a significant number of users got fancy with Node's child_process module, and had several concurrent instances of jitsu in memory, all trying to start the same app over and over.

If you think that sounds a lot like an accidental DDoS, you're right.

By the time my flight was departing from San Francisco early Friday, it had become apparent that this was a subtle and systemic problem - it was still only affecting a very small percentage of apps, but it was affecting "too simple to fail" apps (simple tests and the like) and, most importantly, different load balancers would respond differently, indicating a more pervasive issue. We gathered our Ops team, as best we were able, given the travel circumstances, and began investigating in earnest.

By mid morning, our deployment infrastructure was under a level of load that surpassed what we saw during the last hour of Node Knockout this past year. We were up around 20-30x normal deploy traffic from the aggressive workarounds, and our CouchDB started feeling the strain. The balancers started to get even more out-of-sync with one another, and deploys started to fail from CouchDB timeouts and other related issues.

And that brings me to the other bug.

We operate with a pool of free drone servers, under normal circumstances - when it gets below a certain threshold, auto-provisioning algorithms kick in and start provisioning new VMs to add extra capacity to our cloud. This process is sometimes manual, but it generally requires very little attention from DevOps.

Under a massive sudden load spike, however, and with serious ongoing database issues, the free drone pool got very low, and the high error rate led to more inconsistent state. This time, several attempts to jitsu apps start would all wind up receiving a success message, and all wind up telling the database - and therefore the balancers - that they were on the same IP address.

Our standard drone VMs are small - 256MB of RAM - and we only provision one app per small VM. They really just don't work well otherwise.

The problem, then, is what happens, for example, when three or four Express apps, all using the default port 3000, all hit the same IP address. Only one app - usually the last one to start - would be running successfully, and the other users would visit their URL, and see a completely different site.

Recovery

By the time this happened, we realized it was time to admit defeat, roll back our balancer changes, and start looking into ways to lessen the load. We put our API into a short emergency maintenance, let everything calm down a little, added a lot of new capacity to our drone pool, and reverted to the pre-Wednesday version of our load balancers, prior to the switch to follow.

While this was going on, Maciej Malecki, our self-styled Hotfixer-in-Chief, attacked our provisioning service head-on, and successfully found and fixed the edge case that allowed apps to incorrectly be deployed to already-used drones when the pool was empty. With the balancers reverted, and this edge case fixed, the cloud rapidly returned to normal.

Where We Go From Here

We're determined that these issues - or any issues of this size and scope - not happen again, and we’ve taken the following measures towards prevention:

• We have initiated a complete overhaul of both our staging environment and our staging practices, and reaching for new heights of OCD-level Continuous Integration and QA.
• Our staging environment has been completely replaced. As of this writing, every VM in question was deleted and reprovisioned from scratch.
• We're phasing out most of our manual QA in favor of an automatic deployment bot that will send amusing, demeaning emails to the team whenever app deploys don't work in staging.
• Implemented an improved approach to internal load testing in staging, seeking to better answer the complex question of how to accurately simulate real traffic from the wild internet, and avoid any more sudden surprises when new code meets production load.
• We are greatly expanding and decentralizing our CouchDB, to prevent any database growing pains as we continue to scale.
• In the future, scheduled deploys won’t happen if several senior members of the DevOps team aren’t available due to travel or other circumstances.

In closing, I apologize profusely. Issues like this are unacceptable in any hosting provider, and our commitment to being the best Node.js host out there won't let us let things like this slide.

Today we're pleased to announce the release of my latest open source project, versions.

Versions allows you to create a small scale content delivery network using Node.js. It gets its performance by caching file lookups in memory and setting aggressive caching headers. The fastest request is one that you don't have to make! The server comes with a wide range of features that most public paid CDN's also support:

• Origin pull, the ability to fetch resources from remote servers and cache them on the server.
• Cache Headers, versions will automatically configure cache headers for you.
• Advanced & automatic GZIP, It gzips your resources automatically for browsers that support it (except IE5/6) and detects obfuscated gzip headers as researched by the Yahoo Performance team.
• REST API, providing a REST API to manage the server.
• Clusters, when ran in clusters it can synchronize configuration and other details between servers using a Redis engine.

Creating the server is really simple:

// Require the versions module
var versions = require('versions');

// Setup some origin servers where we want to pull the content from and tag them
// with an id, optionally setup some syncing to be ready to roll.
versions.set('origin servers', [
    { url: 'https://www.nodejitsu.com', id: 'home' },
    { url: 'https://webops.nodejitsu.com', id: 'webops' },
    { url: 'https://gravatar.com/userimage/', id: 'gravatar' }
  ])
  .set('sync', true)
  .set('redis', { host: 'localhost', port: '6379', auth: 'pass' })
  .listen(8080);

As you can see in the snippet above, we are setting 3 origin servers where we will pull our assets from. This allows us, for example, to fetch our base CSS from https://www.nodejitsu.com and another CSS file from https://webops.nodejitsu.com and cache our Gravatar look ups:

• http://example.com/css/base.css would only load the base.css from our home origin as it's the only server that contains this file.
• http://example.com/css/foo.css would be loaded from the WebOps origin as it doesn't exist on the home server.
• http://example.com/15003245/847041ab43d45ec001bd9ef611f2184c.jpg?size=40 would load an avatar from Gravatar as both WebOps and home doesn't have this file.

Cache busting

The problem with aggresive caching is that it's harder to serve your users fresh content when you make changes to your assets. If left unattended users will see the cached version of their browser. The solution? Implement a cache busting scheme. As query strings are known to cause cache issues with reverse proxies we decided to prefix our files using a custom path instead. This would allow us to "bust" browser cache by referencing the file from a different path. In versions you can prefix your assets with a special /versions:<cache bust string> string, for example http://example.com/versions:daf3daf87Adf/base.css would be internally rewritten to http://example.com/base.css

This introduces consistency problems since you might have different applications that use the same CDN and assets. To ensure all your applications are referencing files with identical cache busters we have created an API client. The API can sync version strings between different connected clients.

// Require the versions module.
var versions = require('versions');

// Enable syncing using the .set(config, value) API and connect to your versions
// instance using the .connect method.
versions.set('sync', true)
  .set('redis', { host: 'localhost', port: '6379', auth: 'pass' })
  .connect('http://example.com');

The client will sync its configuration with the server using the provided Redis configuration and setup a pub/sub connection for constant version changes. In order to "tag" your assets with the synced cache buster you can use the versions.tag('path'); method. If you supply it with the path /foo/bar.js it will return http://example.com/versions:0.0.0/foo/bar.js.

Clusters & multiple origins

Versions has support for clusters out of the box -- at Nodejitsu we run it on multiple drones. You only need to set the sync configuration and supply it with your redis details and it's ready to go.

var versions = require('versions');

// Setup syncing like before
versions.set('sync', true)
  .set('redis', { host: 'localhost', port: '6379', auth: 'pass' })
  .connect('http://example.com');

// Setup more alias servers so you can spread resources across different servers
// or to increase parallel download of resources
versions.alias('http://example.org')
        .alias('http://example.net');

// Tag your assets on the fly using:
versions.tag('/filepath.js'); // http://example.com/versions:0.0.0/filepath.js
versions.tag('/bar.css'); // http://example.net/versions:0.0.0/bar.css

In addition to running clustered the API client also accepts multiple origin servers. It will balance your assets between supplied servers using the node-hashring module. Ensuring load and cache size are distributed equally. This comes with the benefit of parallel downloading as each host can only download 4-8 assets at the same time.

So if you ever need a high performance static server for your Node.js application feel free to take http://github.com/3rd-Eden/versions for a spin.

The CDN in Action

Both Nodejitsu's website and handbook are powered by our CDN. Over the last weeks we integrated our documentation with the website. Formerly, our documentation was accessible on a separate domain outside the website. We changed our handbook module to a dependency of the website, to provide a more consistent look and feel. In addition, we changed handbook from a blacksmith powered static file server to a Markdown document collection.

Beside we think it looks better, thats nothing special right? But how are we gonna serve images in the Markdown content without storing them remotely or relying on some third party storage. This is where versions comes into play. We can simply call https://versions.nodejitsu.com/resources/img.png to acquire the image. If it is not yet available on our CDN it will be fetched from the configured external resources. This allows us to keep content centralized. Also handbook is explicitly about serving Markdown content. Versions will keep images in cache and ensure our website will serve documents blazing fast.

The old handbook site loaded our WebOps documentation in a un-cached state in 5.1 seconds. The new documentation does this in 3.67 seconds. With primed cache you were still looking at 3.7 seconds for the old site and only 1.7 seconds for the new version. We think this isn't bad for a page full of big images. So we saved about 2 seconds of load time by using versions.

Try it out for yourself! Want to see our CDN in action? Visit nodejitsu.com or nodejitsu.com/documentation.

Today, we’re very excited to announce the launch of our Business Plans, fueled by a new partnership with Telefonica. Business Plans deliver greater reliability and selection for small to medium companies who are looking to host mission critical applications.

What makes Business Plans different?

Flexible VM sizes for increased reliability

Business Plans give you the ability to select the size of the VM that you’re deploying to. Business Plan customers will be able to choose between 256MB, 512MB, and 1024MB VMs. This significantly increases reliability, and reduces the complexity and degree of horizontal scaling required to handle increased load.

It'll look a little something like this:

$ jitsu deploy joyent us-sw-1 --drones 2
info:    Welcome to Nodejitsu jitsuka
info:    jitsu v0.12.0, node v0.8.19
info:    It worked if it ends with Nodejitsu ok
info:    Executing command deploy joyent us-sw-1
info:    Fetching app nodejitsuftw
info:    App currently deployed to
data:    {
data:        datacenter: 'us-east-1',
data:        ram: 256,
data:        drones: 1,
data:        provider: 'joyent'
data:    }
info:    Deploying application in cloud with:
data:    {
data:        datacenter: 'us-sw-1',
data:        ram: 256,
data:        drones: 2,
data:        provider: 'joyent'
data:    }
info:    App example is now started
info:    http://nodejitsuftw.jyt.us.sw1.nodejitsu.com on Port 80
info:    Nodejitsu ok

Choose your infrastructure provider, and datacenter

You now have the choice of Telefonica and Joyent, and a variety of datacenters to deploy to. As of today, we have support for Joyent’s datacenters in Virginia, Las Vegas and Amsterdam, and Telefonica’s datacenter in London.

For example:

$ jitsu cloud list
info:    Welcome to Nodejitsu jitsuka
info:    jitsu v0.12.0, node v0.8.18
info:    It worked if it ends with Nodejitsu ok
info:    Executing command cloud list
info:    You can use one of the following providers
data:    jitsu cloud joyent us-east-1
data:    jitsu cloud joyent us-sw-1
data:    jitsu cloud joyent eu-ams-1
data:    jitsu cloud telefonica eu-london-1
info:    Nodejitsu ok

This means you can choose your location, and reduce your app’s latency to your customers that matter.

$ jitsu cloud joyent us-sw-1
info:    Welcome to Nodejitsu jitsuka
info:    jitsu v0.12.0, node v0.8.19
info:    It worked if it ends with Nodejitsu ok
info:    Executing command cloud joyent us-sw-1
info:    Fetching app example
info:    App currently deployed to
data:    {
data:        datacenter: 'us-east-1',
data:        ram: 256,
data:        drones: 1,
data:        provider: 'joyent'
data:    }
info:    Deploying application in cloud with:
data:    {
data:        datacenter: 'us-sw-1',
data:        ram: 256,
data:        drones: 1,
data:        provider: 'joyent'
data:    }
info:    App example is now started
info:    http://example.jyt.us.sw1.nodejitsu.com on Port 80
info:    Nodejitsu ok

High Performance Webops

All of these new features are also available in our Webops dashboard which has gotten a complete overhaul to engine.io.

Full support for Europe!

These new features, and our partnership with Telefonica mean one more thing: we now have full support for Europe. Telefonica, one of the world’s largest Telecommunications companies, has recently launched its Instant Servers, offering Joyent’s cutting-edge cloud technology to users from its own, growing selection of datacenters. Combined with Nodejitsu’s unparalleled Node.js PaaS, this will enable Node-based sites and apps to server users in the European market like never before.plans

Ready to take your hosting to the next level? Let's do this!

Wait...did you think that was it?

Just in case that’s not enough for you, we also launched a new, improved version of our frontend website, Nodejitsu.com. Our rebranded site is meant to make it easier to understand how our product works, and who we are. We also think it looks a whole lot slicker.

Thanks for hosting with us! We’re glad to have you on board.

• Able to handle thousands of messages per second.
• Scaling to accommodate the dynamic needs of thousands of applications.
• Flexibility to make decisions based on various criteria.

was challenging. In fact, it is something we've tried to address in several implementations. Over the past two years, it was clear the team that we needed something new. Today we are happy to announce that we are releasing this new solution, godot as Open Source software under the MIT license.

Prior Art

There are an endless number of monitoring solutions out there. There was one that caught our eye when considering our needs: Riemann. Riemann stood out because it is designed as a complex event processor. In their own words:

Riemann provides low-latency, transient shared state for systems with many moving parts.

The events the Riemann processes are simple, with a single metric property representing a single numeric value for the event. Godot uses this same event format for the structure of events.

{
  host:         "A hostname, e.g. 'api1', 'foo.com'"
  service:      "e.g. 'API port 8000 reqs/sec'",
  state:        "Any string less than 255 bytes, e.g. 'ok', 'warning', 'critical'",
  time:         "The time of the event, in unix epoch seconds",
  description:  "Freeform text",
  tags:         "Freeform list of strings, e.g. ['rate', 'fooproduct', 'transient']"
  metric:       "A number associated with this event, e.g. the number of reqs/sec."
  ttl:          "A floating-point time, in seconds, that this event is valid for."
}

When we dug into the Reimann codebase, obviously being predispositioned to node.js, the overwhelming feeling was that it felt to heavy-weight and a little bit obtuse. That and the strong emphasis on streams made this type of system an ideal candidate for a node.js incantation.

Enter godot.

Introducing Godot

Lets jump right in. A common use-case for monitoring is ensuring that everything is still running using heartbeats. In godot you just need to declare a client to send heartbeats and a server to ensure heartbeats are received.

Both of these operations can be done in a single line of Javascript. First lets focus on the server:

server.js

var godot = require('godot');

//
// Reactor server which will email `user@host.com`
// whenever any service matching /.*\/health\/heartbeat/
// fails to check in after 60 seconds.
//
godot.createServer({
  //
  // Defaults to tcp
  //
  type: 'tcp',
  reactors: [
    godot.reactor()
      .where('service', '*/health/heartbeat')
      .expire(1000 * 60)
      .email({ to: 'user@host.com' })
  ]
}).listen(9876);

Now that the server is listening for connections from heartbeat clients, lets create a client to start sending messages.

client.js

var godot = require('godot');

//
// Producer client which sends events for the service
// `app.server/health/heartbeat` every 15 seconds.
//
godot.createClient({
  //
  // Defaults to TCP
  //
  type: 'tcp',
  producers: [
    godot.producer({
      host: 'app.server.com'
      service: 'app.server/health/heartbeat',
      ttl: 1000 * 15
    })
  ]
}).connect(9876);

Both the .createClient and the .createServer methods of godot accept multiple producers and reactors (respectively) allowing you to compose complex behavior from multiple simple streams.

Figure 1: High-level Godot Architecture

Lets examine the dataflow of a simple reactor that sends ops@host.com an email whenever the CPU load of a server goes above 50%.

godot.reactor()
  .where('service', 'cpu')
  .over(50)
  .email({ to: 'ops@host.com' });

Figure 2: Data-flow for sending email on high CPU load

This is obviously very naive and you should probably perform somekind of exponentially decaying moving average to avoid false-positives on short-lived spikes (godot has EWMA built-in via the window-stream module).

var windowStream = require('window-stream'),
    godot = require('godot');

var M1_ALPHA = 1 - Math.exp(-5/60);

godot.reactor()
  .where('service', 'cpu')
  .movingAverage({
    average: {
      type: 'exponential',
      alpha: M1_ALPHA
    },
    window: new windowStream.EventWindow({ size: 10 })
  })
  .over(50)
  .email({ to: 'ops@host.com' });
  

Performance

Before writing this, I noticed a recent post from aphyr (the author of Riemann) discussing in intimate detail how he has achieved 200k messages per second in Riemann. This intrigued me to see just how this little node program stacked up:

node test/perf/pummel.js -c 5
Starting performance test with:
  network protocol  tcp
  concurrency:      5
  sampling interval 10s
  duration:         10s
  ttl:              0
  port:             10557
 
Starting reactor 1
Starting producer 1
Starting producer 2
Starting producer 3
Starting producer 4
Starting producer 5
 
Now receiving messages...
 
Received:
  1069482 total messages
  106948.2 per second

So with little or no attention paid to performance we're already processing 100k messages per second in node. This makes sense because this sort of IO bound application is exactly what node.js was designed for. I suspect (but have not yet had the time to investigate) that this 100k / second benchmark could be faster because it is deficient in several places:

1. Single reactor process: the underlying sockets are not shared between multiple processes.
2. Not distributed: this benchmark was both producing and consuming data from the same machine, so there would be plenty of spare CPU if the messages were produced on a second machine.
3. No framing: As aphyr points out, combing multiple messages into a single TCP packet can greatly increase network throughput, I'm eager to try this out.

So what is Orchestrion?

If you've been following Nodejitsu then you've probably read about our monitoring product, Orchestrion. Orchestrion is built on-top of Godot adding primitives for orchestrating cloud applications:

• Network monitoring: Concerned about network or HTTP latency? We've got you covered.
• Server and application provisioning: Wouldn't you like to just scale your app instead of telling your ops team to scale your app?
• SSH: Run arbitrary processes on demand to respond to incidents.
• Process monitoring: Notify your team through alerting tools when crashes occur.

Lets consider how the above example, emailing your ops team on high CPU load, could be handled with Orchestrion:

Figure 3: Data-flow for scaling your app on high CPU load

Automating these common tasks frees up your developers and operations engineers to handle higher-level more important problems like building your application and your business.

Interested in learning more? Why don't you request a quote for our Enterprise Product. If you're using our Public Cloud product today then stay tuned: we'll be integrating many of these higher level features of Orchestrion into the platform soon.

We first discussed this when we launched our Paid Individual Plans, back in October. Now we've streamlined the process!

Meet opensource.jit.su:

As we say on the page, just fill in your info, and our team will get back to you soon. Big thanks to Julian Duque and Karolina Szczur for the design and code. We're looking forward to hearing from you!

My other passion is Education, I'm involved in the Coderise project where we teach the fundamentals of web programming with JavaScript, HTML and CSS to high school teens ages 13 to 17 years old.

Also I'm the co-founder of two amazing projects, one is MedellinJS, the local JavaScript user group, where we have one reunion monthly and about 240 members, the other is Espuma Lab, a hackerspace where we will teach basic programming and electronics using mainly Raspberry Pi and Node.js.

I felt in love with Node.js at first sight and I'm coming from a Java / Ruby background working mainly developing mobile applications APIs and Cloud Computing Services, now I'm focusing my efforts 100% to the cloud and Node.js projects.

That's why I'm working at this amazing company, Nodejitsu, as a Support Engineer by night and Community Evangelist by day. I'm very happy because I'm working with top-notch members of the Node.js community. This is a great team! I will be working closely with the community so I hope we will see you again.

Pleased to meet you!

Say hello to Nodejitsu's new Status Page!

On the Status Page you can get info at a glance on how the cloud is doing. We wanted to give everyone a central place to go where they could check on the status of Nodejitsu services. After all, no one wants to spend valuable time debugging an issue that's actually caused by problems that Nodejitsu is experiencing. Is there any better way to blame someone else than through a beautiful web page? I'm joking. Kind of.

The Status Page has four sections, each representing how a different aspect of our cloud is doing.

• The first of these is API which refers to the status of our main Nodejitsu API - if this is having issues, deploys and account-related actions won't be working, but running apps will be unaffected.

• Next are Deploys, which shows the status of deploying new versions of your application. Deploys will be down if the API or Database is experiencing issues - or we might occasionally have scheduled maintenance.

• Then there's the Database, which refers to the database we use internally to store application and user data. As with an API outage, this will prevent deploys and account-related actions from working, but running apps will not be affected.

• Finally, we have the Balancers, which refers to the status of our load balancers.

If you're looking for the Status Page on the Nodejitsu website, scroll to the bottom navigation; it'll be the last link under 'PAAS'.

Of course, the status page is a Flatiron project. On the database side of things, Redis was used, just to mix it up. As always, if you experience any issues with the status page, or anything else on Nodejitsu, please let us know either in IRC at #nodejitsu on irc.freenode.net or through email at support@nodejitsu.com.

Today we are announcing Deployment Status Badges. When you're using Continuous Deployment on Nodejitsu, these let you know instantly if the deployment is in progress, has been successful, or has failed.

If you want to use them it's super simple: If your username in Github was yawnt and the repo was called myblog all you need to do is to add the following Markdown to your readme.md file:

[![Nodejitsu Deploy Status](https://webhooks.nodejitsu.com/yawnt/myblog.png)](https://webops.nodejitsu.com#yawnt/myblog)

We've got info on our badges and a whole lot more in the new and improved Nodejitsu Handbook

P.S. If you really love the badges don't forget to follow @karolinaszczur and tell her all about it!

Today, we would like to announce that from now on you can deploy your open-source Github repositories directly to Nodejitsu.

How does this work? By implementing a Github service for Nodejitsu, you will now be able to deploy your open-source application anytime you push a commit to Github.

But that's not good enough Continuous Deployment you say? We have also added an integration with TravisCI so that you can also deploy your open-source application only after a successful test run.

Open-source is a core value at Nodejitsu. We all maintain several open-source modules that make the node.js community a better place, and promote high profile projects like flatiron and http-proxy so we want to make sure that Open Source developers have the best experience available for deploying their application.

Keep reading to learn all the details about getting setup with Continuous Deployment on Nodejitsu!

• Getting Started
• Monitoring Deployments
• Working with Travis
• What's next?

Getting started

To setup continuous deployment with Github and Nodejitsu, first, access the Admin section on your Open Source node.js Github repository. Click Service Hooks and then Nodejitsu. You will be presented with a form with four fields:

• Username, which defaults to your Github username
• Password, your password or a valid Nodejitsu API authentication token
• Branch, where you can define the branch you wish to deploy and defaults to master
• Endpoint, which defaults to https://webhooks.nodejitsu.com

Select Active and hit Update Settings. From now on every-time you commit to github (in the designated deployment branch) we will deploy that application for you. That simple.

But wait, I have API keys I can't commit to Github as open source?!

Don't worry, you can use jitsu set env to set environment variables that you can access with process.env. Check our handbook for more information. Environment variables set this way persist across deployments and are also available in our webops application.

If this is your first time getting this setup, then you probably want to set those environment variables first in a new nodejitsu application before deploying from Github. Otherwise, your first deployment will fail because the environment variables are not yet set.

  jitsu apps create your-app
  jitsu env set <key> <value>

Monitoring deployments

There's several ways to access the deployment status in the Nodejitsu Webhook API, and you can find the complete documentation at webhooks.nodejitsu.com.

The most fun way to monitor your deployment is with the realtime status changes feed.

  #
  # if your username is foo and password is bar this would be 
  # https://foo:bar@webhooks.nodejitsu.com/1/status/foo/changes
  #
  curl https://username:password@webhooks.nodejitsu.com/1/status/username/changes?include_docs=auto

This will create an HTTP keep alive connection that pushes the status to you in realtime every time someone invokes our API:

  {
    "id": "https://webhooks.nodejitsu.com/1/status/dscape/changes/2b3de47c2ce04a9dda4d31aac5000a8d",
    "app_name": "hello-world-api-flatiron",
    "uuid": "5p45jm1332436661256",
    "status": "waiting",
    "provider": "github",
    "commit": "dscape/hello-world-flatiron-api/96410ed970f6224a4cd3c450150c5d65bbc77fcd"
  }
  {
    "id": "https://webhooks.nodejitsu.com/1/status/dscape/changes/2b3de47c2ce04a9dda4d31aac5000bab",
    "app_name": "hello-world-api-flatiron",
    "uuid": "aa6e9b1332436698771",
    "status": "ok",
    "provider": "travis",
    "commit": "dscape/hello-world-flatiron-api/96410ed970f6224a4cd3c450150c5d65bbc77fcd"
  }

Each request to our API is logged with a unique uuid, so you can use it to refer possible issues to our support staff.

We are now ready to deploy, go back to Github and click Test Hook. You should be up and running shortly.

Working with Travis

What about continuous integration? We added Travis-CI so you can feel safe about your deployments. Simply add something like this in your .travis.yml file.

notifications:
  webhooks: 
    urls:
      - http://webhooks.nodejitsu.com/1/deploy
    on_success: always
    on_failure: never

Internally our API will try to see if you have Travis configured like this, and if you do it will put the deployment request from Github on a hold until Travis informs us all tests have passed.

If tests failed we won't deploy. Simple.

What's next?

If you don't have a Nodejitsu account sign up for one at nodejitsu.com. Feel free to take a peek at our Getting Started guide, and also to check out webops.nodejitsu.com to see the status of your applications.

Join #nodejitsu on irc.freenode.net and get immediate assistance from our support team! Alternatively you can email support@nodejitsu.com.

As great as the old handbook was it had some problems:

• It was just one big Markdown file.
• The build system was archaic.
• There was a lot of duplicated content.

The best part about the handbook was that it was and still is totally open source on Github. Now that it is also a blacksmith powered website, we wanted to make sure that stayed the same by adding a "View on Github" button to every page.

So now it's time to go practice your nodejitsu!

Pkgcloud is a node.js module that allows interaction with multiple cloud service providers. With the help of this module, you're able to manage a variety of cloud resources from within your application. You can provision compute, storage or database instances with ease.

IaaS providers have each chosen their own vocabulary for describing the resources they provide, leading to inconsistencies in terminology from one to the next. One of the objectives of pkgcloud is to maintain a unified vocabulary and use a regular terminology for applications without users needing to know which term was chosen by which provider. This is part of our ongoing efforts to be Infrastructure Agnostic.

Complete Docs

You can check out the full pkgcloud documentation on Github right here.

Looking for an overview? Keep scrolling.

Vocabulary

Here's how the unified vocabulary for pkgcloud breaks down.

Compute

Storage

Getting Started

First install the library from NPM

npm install pkgcloud

Once it is installed, you can use it from your code.

var pkgcloud = require('pkgcloud');

At present pkgcloud supports: compute, storage and database as available resources. You can create a client to provision and control resources from your favorite provider. For client initialization, you will need the authentication info used by your provider, like username and password.

Compute

var joyentOpts = {
  provider: 'joyent',
  username: 'daniel',
  password: '1234'
  // Also you can use key/keyId as authentication
};

var rackspaceOpts = {
  provider: 'rackspace',
  auth: 'yourTokenForRackspace'
};

var amazonOpts = {
  provider: 'amazon',
  keyId: 'YourKeyId',
  key: 'yourkey'
};

//
// Now you can create your client with your config
//

var joyentClient = pkgcloud.compute.createClient(joyentOpts), // Joyent CloudAPI client
    rackspaceClient = pkgcloud.compute.createClient(rackspaceOpts), // Rackspace Cloud Servers client
    amazonClient = pkgcloud.compute.createClient(amazonOpts); // AWS EC2 client

Storage

You could also use the same config for storage resources

var amazonClient = pkgcloud.storage.createClient(amazonOpts), // AWS S3 client
    rackspaceClient = pkgcloud.storage.createClient(rackspaceOpts); // Rackspace Cloud Files client

After creating the client, depending on the chosen provider, pkgcloud will offer some methods for interaction. For detailed information about those methods, please see the complete documentation on github.

Databases

Database Providers supported:

• IrisCouch (Couchdb and redis databases)
• MongoLab (Mongodb databases)
• MongoHQ (Mongodb databases)
• RedisToGo (Redis databases)
• Rackspace (MySQL databases)

As in the previous examples, you will just need to define a few options that include the provider's name in the provider field and the necessary authentication fields.

var client = pkgcloud.database.createClient(options);

Pleease refer to github for further information about which methods are available for which provider.

A realistic example

Using the newly created joyentClient, we will provision some compute instances.

var flavor, image;

joyentClient.getFlavors(function (err, flavors) {
  // Choose the flavor (memory, disk, swap)
  flavor = flavors.shift();
  joyentClient.getImages(function (err, images) {
    // Choose the pre-build image
    image = images.shift();
    // Provision a new machine called 'test-server'
    joyentClient.createServer({
      flavor: flavor,
      image: image,
      name: 'test-server'
    }, function (err, server) {
      // Now look the details of your new provisioned server
      console.log(server);
    });
  });
});

Roadmap

1. Backport latest fixes from node-cloudfiles and node-cloudservers
2. Include CDN and DNS services.
3. Implement fs compatible file API.
4. Support additional service providers.

Thinking about Contributing?

We welcome contributions to pkgcloud by any and all individuals or organizations.

Before contributing, please take a look at the Contribution Guidelines. We are pretty flexible about these guidelines, but the more closely you follow them, the more likely we are to merge your pull-request.

SO COME ON, WHAT ARE YOU WAITING FOR?

Don't want to hack alone?

Take a look at the NKO locations page for hosted backspaces around the world.

If you're in NY, the Nodejitsu team has partnered with Levo League to provide a chill, comfy space with healthy meals and drinks just south of Union Square. RSVP here.

Node Knockout Deployment Instructions!

Introduction to Jitsu deployment

TL;DR

$ git clone https://github.com/nko3/[slug] myapp
$ cd myapp
$ sudo npm install -g jitsu
$ jitsu login ## username and password provided on your NKO team page
$ jitsu deploy

Installing

To install Jitsu, all you have to do is run: npm install -g jitsu

Hint: If npm throws an error, try using sudo npm install -g jitsu.

Setting up Jitsu

Once you install Jitsu, you will need to login with the password that was provided to your team by NKO. This login, and only this login, will need to be used for each team member. NKO will not accept submissions that are under another Nodejitsu account.

To login, use: jitsu login

Hint: If you get an error when logging in, try using one of the following:

• jitsu conf set protocol http
• jitsu conf set host api.nodejitsu.com

Deploying

Before developing your app, you should clone the repository provided to your team by NKO. This will follow the pattern of nko3/[slug]. For example, if your team is named "Node Bandits", your repository would be nko3/node-bandits.

If you do not have access to this repository, let someone know immediately. This repository provides a skeleton of what NKO is expecting you to submit when you deploy your application.

Once you have this repository checked out, you can code your application however you see fit. Once you're done, deploy your application to our cloud by running jitsu deploy in the directory containing the package.json file that was provided by your nko3/[slug] repository.

Warning: Make sure you keep version numbers synced across team members. If a team member trying to deploy an application with version 0.0.1 while another team member has deployed 0.0.2, the server will reject the deploy.

Remember:

• nko3/[slug] should be nko3/your-team-name. For example, a team named "Node Bandits" would be nko3/node-bandits.
• You get one login for Nodejitsu per team. Use it, because your submissions won't be accepted from another account.
• Keep version numbers synced amongst your team. Using different versions could, and probably will, cause unintended effects.

Going deeper

Jitsu has a lot of features including deploying old snapshots, getting running logs, setting environment variables, and even creating databases. More info can be found by just typing jitsu at the command line or in the Nodejitsu Handbook.

Help

You can visit us online at our homepage. You can find out more information about Jitsu on Github. As always, you can also find us on IRC on Freenode on ##nodejitsu.

Nodejitsu offers newer versions of node, great customer support, features like WebSockets and multi-cloud options (coming soon!) all on the best operating system to run your node.js application: SmartOS Joyent SmartMachines.

If you want to ensure that your application keeps running continuously, you should follow our upgrade instructions before October 30th.

Here’s what you’ll need to know.

node.js versions

Nodejitsu only installs latest versions of node: v0.6 and v0.8.

At this point node v0.4 should not be considered production quality. Many bug and security fixes have been introduced in latest releases and didn't find their way into latest node v0.4 release.

If you need help with upgrading your application, here are some resources for you:

• API changes between v0.4 and v0.6
• API changes between v0.6 and v0.8

Also, feel free to drop by our IRC channel to get support from our team and fellow coders.

Upgrading

Installing jitsu

To deploy to Nodejitsu, you need our CLI tool, jitsu. You can install it reqlly easily with npm.

[sudo] npm -g install jitsu

Now, jitsu command should be globally accessible.

Signing up for a nodejitsu account

There are 2 ways to sign up for a Nodejitsu account: through the website and with jitsu.

If you decide to signup through the website, just click on the "Try Nodejitsu For Free" button and fill in some details.

jitsu command responsible for signing up is jitsu signup. jitsu will prompt you for username, email address and password and create an account for you.

In both cases, you'll receive an email with instructions on what to do next.

First deployment

The major difference between no.de and Nodejitsu deployment is the method of sending code to servers. no.de uses git-based deployment, while at Nodejitsu we simply pack your code into a tarball using npm.

Therefore, to deploy you don't need to commit your changes into version control. At any point in time you can execute jitsu deploy command and have your changes reflected in the code running on a remote server (obviously, you should still use version control; it's just no longer required for deployment).

So, how your typical deploy will look like:

[maciej@pc08 test-app]$ jitsu deploy
info:    Welcome to Nodejitsu nodejitsu
info:    jitsu v0.9.7
info:    It worked if it ends with Nodejitsu ok
info:    Executing command deploy
info:    Analyzing your application dependencies in app.js
info:    Checking app availability test-app
info:    Creating app test-app
info:    Creating snapshot 0.3.5-46
info:    Updating app test-app
info:    Activating snapshot 0.3.5-46 for test-app
info:    Starting app test-app
info:    App test-app is now started
info:    http://test-app-21421.jit.su on Port 80
info:    Nodejitsu ok

Accessing logs

Another difference is the way of accessing server logs. no.de offered SSH access. Accessing logs is a bit different at Nodejitsu - you can fetch them using jitsu.

For example, to get 5 most recent lines of logs from app home:

[maciej@pc08 ~]$ jitsu logs app home 5
info:    Welcome to Nodejitsu nodejitsu
info:    jitsu v0.9.7
info:    It worked if it ends with Nodejitsu ok
info:    Executing command logs app home 5
info:    Listing logs for home
[08/29 23:09:12 GMT+0200] [served] /css/normalize.css
[08/29 23:09:11 GMT+0200] [served] /
[08/29 23:08:59 GMT+0200] [served] /paas/tools.html
[08/29 23:08:15 GMT+0200] [served] /img/visuals/visual-2.png
[08/29 23:08:15 GMT+0200] [served] /img/visuals/visual-3.png
info:    Nodejitsu ok

You can more help about that command by executing jitsu logs in your terminal.

Custom domains

If you were using a custom domain with no.de, you need to apply some changes to your DNS set up. Instructions on how to do that can be found at dns.jit.su.

Something wrong?

We'd love to help and make the transition as pleasant as possible! Talk to us on #nodejitsu at Freenode IRC or at support@nodejitsu.com.

When Nodejitsu entered public beta, we shared our approach to Platform-as-a-Service pricing, and our commitment to changing the way that Platform-as-a-Service companies incentivize developers. Today's announcement reflects that commitment and we're pleased to have you try Nodejitsu today for free in our Development Sandbox. And when you're ready you can upgrade to one of our Individual Plans starting at just $3.

(Please note: at this time we are only introducing Paid Individual Plans. Our Business Plans will be available for purchase soon!)

If you're hosting on Nodejitsu, you still have one free month left

If you're using Nodejitsu today you will automatically be enrolled in our Development Sandbox next week which will give you one free drone at Nodejitsu for one month. After that time you can upgrade to one of our Paid Individual Plans starting at just $3.

Wait! I'm running more than one app already!

If you are running more than one app you will have one week (until Oct. 17th) before additional apps are turned off. Your app will not be deleted. You can always keep as many stopped apps as you want on Nodejitsu. It will remain stopped until you begin paying for the drones to run it.

We're here to help

Problems? Questions? Reach out to us on Twitter or in #nodejitsu on IRC. We couldn't be more excited to be working with you.

Shanghai, China - HuJS

When I was invited to HuJS by Glenn Block (from the Microsoft Azure team), I was actually a little surprised that there was enough interest to have this kind of regional Javascript conference in China. The CHINA MARKET meme had become used so much, I had doubts that there would really be anything of substance there. I couldn't be more wrong.

The conference itself was great; I wish I spoke Chinese because over half of the talks were in the native language. I was blown away by the talent and overall prolific output of these Chinese Javascript developers. I was also quite impressed with not just the Javascript community, but the burgeoning startup community in Shanghai. Special thanks to the team over at Factual for being gracious hosts and showing me around Shanghai!

Lisbon, Portugal - LXJS

I had been looking forward to LXJS for months. Not just because it was shaping up to be an awesome conference, but because it was one of those rare moments when more than five members of the Nodejitsu team were in the same room. In fact, between staff and speakers there were eight nodejitsuka at LXJS:

• Paolo Fragomeni - Thinking Distributed
• Karolina Szczur - The Pursuit of Simplicity
• Maciej Malecki - Node.js Internals
• Elijah Insua - Hardware with Javascript and CNCs
• Arnout Kazemier - Building the Real-Time Web
• Charlie Robbins - Open Source LIKE A BOSS

Not to mention that the event organizers were three of Nodejitsu's own Nuno Job, Pedro Teixeira and Ana Hevesi. They did a great job organizing the best destination conference of the year. Personally I really enjoyed the closing keynote by Mikeal Rogers, where he shared some staggering statistics about the Node.js community and how we approach infrastructure:

On Infrastructure

All of the papers that people write, and all the blog posts that are saying "this is what a clean infrastructure looks like": they're justifying the old infrastructure after the fact. They built that because they couldn't build it the way that they wanted to; and now it's gospel.

On Node.js Growth

Today there are over 15,000 packages in the npm repository. (...) Let's do a quick comparison: RubyForge has [about] 9,000 packages in it. (...) RubyGems has less than 3,000 packages. (...) So [node.js] already more than that. Python has 24,000 packages in it, but it's been around since 1989; [node.js] has been around three years so we'll close in on them by the end of next year.

It's not over yet!

In the past two years, the Javascript community has raised the bar for technology conferences by, as Chris Williams (creator of JSConf) says, "growing out, not up". That means we're having more conferences not bigger conferences. This allows the same intimate and friendly atmosphere to be fostered at every event while still serving the needs of a vastly distributed community. What does this mean? It means conference season is just getting warmed up. Here's a rundown of awesome events and a look at where the Nodejitsu team will be.

October

• Reject.js Berlin, Germany : October 4th (Karolina Szczur, Paolo Fragomeni)
• NodeCopter.js Berlin, Germany : October 5th (Karolina Szczur, Paolo Fragomeni)
• JSConfEU Berlin, Germany : October 5th and 7th (Karolina Szczur, Paolo Fragomeni)
• NodeDublin Dublin, Ireland : October 18th and 19th (Paolo Fragomeni, Charlie McConnell, Maciej Malecki)
• EmpireJS - New York, NY : October 22nd (Charlie Robbins, Ana Hevesi, Charlie McConnell)
• RedisConf - Portland, OR : October 22nd (Arnout Kazemier)
• Keeping it Realtime - Portland, OR : October 23rd and 24th

November

• CascadiaJS - Seattle, WA : November 8th and 9th
• Nodejsconf.it - Brescia, Italy : November 10th (Maciej Malecki)
• Node.js Fest Japan - Tokyo, Japan : November 18th (Charlie Robbins)
• dotJS.eu - Paris, France : November 30th (Charlie Robbins)

You should also definitely be subscribed to ConfNotice, a newsletter about awesome events curated by JSConf's own Chris Williams.

Over the last two years, as the business and team have grown, the founders’ ideas about management and policy grew to be at odds. Starting a company takes passion and strong personalities, but eventually company disagreements spilled over into being too personal. In order to maintain a healthy, respectful work environment, we felt we needed to separate ourselves.

To clarify, Flatiron is a Nodejitsu initiative. While Marak has been one of its key architects and proponents, his departure from the company does not mean the end of Flatiron. Quite to the contrary, the Flatiron framework and associated tools are used to power many key parts of Nodejitsu’s cloud offering. Flatiron will remain open-source, and our commitment to free open source software with non-restrictive licenses remains steadfast. We will continue to collaborate with the node.js community to build and innovate Flatiron.js.

Marak’s contributions to Nodejitsu, as well as node.js in general, have been innumerable. His evangelism in the early days was instrumental to helping many people find and embrace node.js - including several key members of our team. Perhaps most importantly, his unique and outgoing sense of humor has been an important part of our team morale, as well as the spirit of the node community in general.

This has been a very difficult decision, but one which had to be made so that we could continue to build and grow Nodejitsu into what we believe it can be. We will always be grateful for Marak’s contributions and we hope the node.js community will continue to benefit from his talents.

In computer science, reflection is the ability of a computer program to examine and modify the structure and behavior (specifically the values, meta-data, properties and functions) of an object at runtime - Wikipedia

The Flatiron framework is designed to help with the rapid development of robust isomorphic JavaScript applications. Flatiron is built on the node.js philosophy of small, distinct, usable components. Every component of the framework can be used independently, for it's own distinct purpose.

Resource-View-Presenter

While developers have the choice to assemble these Flatiron components in anyway they like, Flatiron encourages the use of the RVP (Resource-View-Presenter) pattern.

Using the RVP pattern allows developers to move their applications towards a domain driven design. This rather utopian computer science idea is usually muddled by several layers of UML diagrams and poorly instrumented tool chains.

Flatiron's approach towards attaining such programming nirvana is a bit more simple and realistic.

Resources are JavaScript Objects

A Resource is a JavaScript object. You've probably worked with them before.

Let's define a Creature resource using the resourceful library.

var resourceful = require('resourceful');
var Creature  = resourceful.define('creature');

Creature is a constructor function for creating new Creatures. It's easy I promise.

var bob = new Creature({ 
  id   : 'Bob',
  type : 'dragon'
});

You can totally still export Creature in a Node.js module.

exports.Creature = Creature;

This newly created Creature instance Bob comes with some built-in methods to help you out with performing basic CRUD operations.

By default, resources will use an in-memory storage engine provided by resourceful. The storage engine itself is pluggable and supports several modern database engines.

//
// save, create, destroy, all, find, etc...
//
bob.save({ type: "giant moth" }, function(err, result){

});

Here are several code examples of using these CRUD methods.

Woah, I don't want this CRUD

Easy! You can, noop them out, override their definitions at will, or just igore them. A resource is a just a JavaScript object and create, save, etc are just functions!

over-ride built-in resource methods

// No more delete
delete Creature.destroy;

// Custom save
Create.save = function(id, data, callback){
  callback(null, 'The googles do nothing');
});

Extending Bob the Dragon

Now we have a new Creature, Bob the Dragon. Bob can pretty much do anything a respectful CRUD dragon could do.

Well, almost! We are also going to add a feed action to Creatures, because Bob will probably get hungry at some point.

Creature.feed = function (_id, options, callback) {
  var self = this;
  self.get(_id, function(err, creature){
    if(err) {
      return callback(err);
    }
    var life = creature.life + 1;
    self.update(_id, { life: life }, function(err, result){
      callback(null, 'I have been fed my life is: ' + result.life);
    });
  });
  self.remote = true;
}

Note: The remote property is used to indicate the method should be reflected via public remote interfaces. Any methods which are not remote, will not be exposed to public interfaces.*

Let's get reflecting

I thought this was a post about reflection? Well it is!

• We defined our resource, Creature
• We gave Creature a custom feed method
• We created Bob the Dragon

Now we are going to reflect our Creature across across several interfaces!

Flatiron currently supports resource reflection across:

• RESTful HTTP Server / Clients
• HTML Forms
• socket.io websocket Server / Clients
• Command Line Interfaces

simple restful server:

restful.createServer([Creature]).listen(8000, function () {
  console.log(' > restful server started on port 8000');
});

individual code examples can be found in each reflection library

This means that Bob the Dragon can be expressed as a fully functional HTTP / websocket isomorphic application with a command line interface.

This sounds like a giant clusterf*ck of auto-generated code

No way! First off, there is zero auto-generated or scaffolded code in the reflection process. I've done that before, and it's stupid.

Flatiron breaks down the problem of uber-reflection into separate and distinct modules that have no direct dependency or relation to each other.

The core dependency ( or the R in RVP) is the resourceful library. Resourceful allows you to define resources. The Flatiron framework then has several distinct projects for reflecting resources across well-defined and well tested interfaces.

Customization already sounds impossibly complicated

No way! Any interface not directly reflected can just as easily be implemented using the appropriate backing library. All Flatiron modules are broken down into distinct and re-usable projects.

For example, if you used restful to reflect a RESTful router server-side in node.js, you will then have access to a Director.router instance. This router is created by the heavily used Director library.

If you need to override a generated route, or create an ad-hoc route, or make any customization, the API is exactly the same as the Director API.

customize a reflected router interface:

app.router.get('/', function(){
  this.res.end('home page');
});

//
// Overrides `/creature/larry` but won't override,
// any other `/creature/:id` captures.
//
app.router.get('/creatures/larry', function(){
  this.res.end('larry is special!');
});

Like most of Flatiron's reflection libraries, restful is built to solve 90% of use-cases. If you hit a case where restful is causing a problem, you can simply drop into the Director API.

Resource reflection is highly encouraged, but most definitely optional.

TL;DR; All of Flatiron's reflection libraries are optional and have a well-defined library backing it's interface.

Wait, what about schema? What about validations?

Resourceful provides support for custom validation functions and an optional JSON-Schema implementation through revalidator.

You can arbitrarily create validation functions or use JSON-Schema on any property or resource and all interfaces will respect these validations at run-time. It's slightly magical.

validation on individual properties

// Number between 0 and 10
Creature.property("life", Number, { min: 0 max: 10 });

// Email validation
Creature.property("email", String, { format: 'email' });

// Custom validation conforms
Creature.property("even", Number, { 
  conform: function (value) {
    return value % 2 === 0;
  } 
});

use before and after hooks for resource validations on resource actions

Creature.before('create', function(options, callback){
  if(options.foo !== 'bar') {
    return callback(new Error('foo is not bar!'));
  }
  console.log('about to create a new creature!', options);
  callback(null, options);
});

Creature.after('create', function(err, result){
  console.log('Creature was created!', result);
});

Fine, this might work. What about relational data?

Easy! Resourceful supports relationships between resources ( one-one, one-many, many-many ). Relationships are defined in resources through the Resource.parent API.

Each reflection library will respect these relationships and act accordingly. You can find additional code examples for setting up relational data in resourceful here or in the README for the specific reflection library you are using.

defining a simple resource relationship

var Album = resourceful.define('album');
var Song  = resourceful.define('song');
Song.parent('album');

Everything else will be handled for you by Flatiron. It's that easy.

Okay, I believe. What now?

Get over to http://github.com/flatiron and get started!

Flatiron is already the most contributed to Node.js Framework on Github. With the upcoming 1.0 version, we are about to enter a new age of isomorphic JavaScript frameworks and web-development. Join the JavaScript party!

Giddy-up!

This is Now

The app is a mosaic of images which gives you a window into the lives of people in cities around the world. You check out NY and find photos of sky scrapers, Brooklyn street corners, and people's pets, while the London feed still reveals remnants of the 2012 Olympics. I caught up with Marcio who explained that this project was Lexical Gap's first foray into node.js. The Lexical Gap team used Socket.io to communicate with the Instagram API via Instagram-node-lib to deliver a smooth real-time experience.

This Is Now also turned out to be a great opportunity for our team to test what happens when apps running on our small drone servers get hammered with traffic (i.e. 500 concurrent Socket.io connections per drone). We were able to quickly scale up the app by running it on multiple drones, seamlessly handling the burst of interest in the app, spurred on by several articles in the press - the more popular cities, like New York and Los Angeles, were peaking at well over 2000 concurrent websocket connections. While there were a few regressions, we jumped in to handle them quickly, and everything went great.

To quote Marcio "We received a private beta account from Nodejitsu on the 21st of April, and we have never left. The platform provided a robust suite of functionality that assisted in the development, management, and deployment of our Node.js applications. The deployment tools are definitely the most user-friendly in the industry. Everything highlighted above would be already enough reason for us to be convinced to use it, but our respect/admiration really grew when we met the guys behind Nodejitsu."

Thanks. :) Congrats to the Lexical Gap guys on the success. We can't wait to see what community turns out next.

Zero downtime deploys!

I am thrilled to announce that your application no longer requires stopping in order to start running new code.

Essentially, that means two things:

• no downtime during regular deployment
• if deploy fails, your application stays up, running the last snapshot

This feature involves a semantic change to our API. POST /apps/:username/:appname/start first starts a new drone, then kills the old one. You will never again see a "Cannot start running application" error while accessing this endpoint.

This is different from POST /apps/:username/:appname/restart, which tells our application server to restart the application (as in, kill your process and start it again). Because the restart endpoint affects the already-running instance, zero-downtime does not apply.

Fetching snapshots

Many of our users were asking about a way to fetch an existing snapshot from our cloud, and here it is! The API endpoint for downloading a snapshot is GET /apps/:username/:appname/snapshots/:id.tgz. The API will respond with a tarball.

var fs = require('fs'),
    request = require('request');

request('http://user:pass@api.nodejitsu.com/apps/user/app/snapshots/v0.0.1.tgz')
  .pipe(fs.createWriteStream('user-app-v0.0.1.tgz'));

We also introduced a new command in jitsu: jitsu snapshots fetch. This will download a tarball of your app to your current working directory.

jitsu hooks

Another feature which was requested quite often was pre- and post-deploy hooks for jitsu. Obviously, we implemented them!

In some cases, you might want to perform some action before or after deploying an application. That's where these hooks come in handy.

Example:

{
  "name": "test-app",
  "subdomain": "test-app",
  "scripts": {
    "predeploy": "echo This will be run before deploying the app",
    "postdeploy": "echo This will be run after deploying the app",
    "start": "app.js"
  },
  "engines": {
    "node": "0.6.x"
  },
  "version": "0.0.0"
}

Results in the following output from jitsu:

[maciej@pc08 test-app]$ jitsu deploy
info:    Welcome to Nodejitsu nodejitsu
info:    It worked if it ends with Nodejitsu ok
info:    Executing command deploy
info:    Analyzing your application dependencies in app.js
info:    Checking app availability test-app
info:    Creating app test-app
This will be run before deploying the app
info:    Creating snapshot 0.0.0
info:    Updating app test-app
info:    Activating snapshot 0.0.0 for test-app
info:    Starting app test-app
info:    App test-app is now started
info:    http://test-app.jit.su on Port 80
This will be run after deploying the app
info:    Nodejitsu ok

There are various use-cases, including build or compile steps, tagging releases, and so on.

That's it!

Thanks for all the feedback and support - and if there are other features you think would be awesome, let us know! We're always looking to make our platform even better than it already is.

If you have any questions, feel free to reach out to us at #nodejitsu on Freenode IRC.

What's js13kgames anyway?

js13kGames is a coding challenge for HTML5 game developers founded by Andrzej Mazur (a game dev himself). The main rule is to keep your file size below 13kB. There's no official theme for entries, so be creative! You'll want to check out the full list of guidelines here.

While the contest is focused on JavaScript we strongly encourage all participants to try Node.js for your back end needs! In case of any problems you can always hang out in the #nodejitsu channel on irc.freenode.net to get some answers.

Prizes

About those prizes...Nodejitsu is giving away one six month Regular plan for free! You'll be able to deploy and manage your apps and games with all the goodness our platform has to offer: WebSockets, CoffeeScript, ImageMagick and libxml support, custom domains, and free SSL on *.nodejitsu.com. Sounds awesome, right?

Our partner - The Node Firm is sponsoring one ticket to the open Node.js training of your choice. If you're not yet familiar with the technology but want to learn from top-notch developers (core contributors and module creators) that's a perfect occassion to do that!

Other prizes include three licenses for Impact.js, the JavaScript game engine created by Dominic Szablewski, a ticket to Bocoup training, two books from O'Reilly media and three copies of Foundation HTML5 Canvas for Games and Entertainment by Rob Hawkes. That makes a great set both for beginners and seasoned developers.

Submitting your game

All entries should be public repositories on Github. In order to participate you need to fill out the form on the js13kGames website with a link to the repository, the app itself (hosted at Nodejitsu), and a few details about yourself.

Deployment to Nodejitsu

Since Nodejitsu is already in public beta, every participant can register for a Developer Sandbox account for free.

After successful registration with jitsu, all you have to do to deploy your application is run the following in your app's top level directory:

jitsu deploy

This will use npm to bundle your application, send it to nodejitsu, and deploy it!

Because jitsu uses npm to bundle your application, dependencies and app configuration are managed through your package.json. You can read more about the package.json format at http://package.json.jit.su. Jitsu will automatically prompt you for a number of required fields as needed, such as your desired application subdomain (ex: myfungame for http://myfungame.jit.su), desired node.js version, and anything else we need. You can read more about how nodejitsu works from our handbook.

Every time you want to deploy a new version of your application, just run jitsu deploy. If you have any problems, just copy and paste all of jitsu's output into a gist and show it to our support team in #nodejitsu on freenode.net.

Good luck!

Nodejitsu is now available for all. You can sign up for Nodejitsu and will immediately be placed in the Development Sandbox without having to wait for an activation.

If you already have a Nodejitsu account and you've been waiting for activation…

It's your lucky day. You should be receiving an email including your confirmation code. So long as you have node.js and our CLI tool, jitsu installed, you're good to go; all you have to do is run the confirmation command! Check out the Nodejitsu Handbook if you need a refresher on how to deploy.

If you're a beta tester…

You don't need to do anything right now. You're already in. Hope you've been enjoying yourself.

Billing and the Developer Sandbox

All users at this stage are in the "Development Sandbox", which is free. Our billing plans will roll out in the next couple of weeks and at that point you can choose to upgrade to a higher level of service.

You can check out our soon-to-be-live pricing plan here.

What took so long?

We're aware that many of you have been waiting a very long time for access to Nodejitsu. There are a few reasons why we chose to stay in private beta and to limit access for this long. Firstly, We're deeply committed to the quality of our service. We want to provide first class, hands-on support by people who actually love what they do.

We are also building a product, not just a public cloud. We've been building a cohesive suite of tools that can be deployed to your data center so you can build your own Node.js clouds, on your own hardware. We want to help node succeed within established businesses as well as with individual developers.

Finally, at Nodejitsu, our success is directly proportional to the success of Node.js itself. Most of all, we want to ensure that the node community is provided with the best solution possible.

Some of you may remember the early wild west days of node when a new release meant everything you had built had broken in one way or another. Back then (in fact for most of our first year of business) our focus was to ensure that users (and us!) had the right set of tools to build applications. We knew that without this ecosystem of Open Source tools Node.js (and therefore Nodejitsu) would never succeed. The second year was greatly focused on building out our infrastructure. Now the time is right to present you with the culmination of this work.

It's been a privilege so far to work with such an awesome community. We're both humbled and excited to be opening our doors. Let's hack!

Maintaining software is no walk in the park: writing modules is at once the most rewarding and most difficult thing you can do. I'm often reminded of what isaacs once said on the nodejs mailing list:

It's like having children. Seems like a lot of pointless busy-work, and then you do it, probably without meaning or wanting to, and then suddenly fall so in love, you can't understand why everyone doesn't go as nuts over this little food-to-poop converter as you do. If you're not careful, you'll end up writing 3 or 4 more.

This article represents the lessons learned from writing node modules since node@0.1.103.

Maintaining Software is Hard

• Understand the Commitment
• Define What You're Going to do and do it
• Get Some Attention
• Keep your Dependencies Lean and Fresh
• Don't be Afraid to Refactor
• Be Wary of Compiled Modules
• Support Your Users
• Assess Pull-Requests Carefully
• Unpublish Versions you won't Support
• Know When to Throw in the Towel

Understand the Commitment

It bears repeating: maintaining software is hard. If you're not up to the challenge of responding to issues, publishing hotfixes, and generally making your module a priority in your life then you probably want to consider looking for something that already exists.

Define What You're Going to do and do it

By the time you've decided to commit to writing a module make sure you completely understand the problem at hand. Whether it's an API library, a utility module, or a framework be clear on the expectations early on. Too many times I've seen libraries that are doing too much or making choices that not all of their user-base will appreciate.

Get Some Attention

If you're new to Open Source, you need to understand that your modules don't just market themselves. You are defining a brand and you need to get some attention for it. More attention means more users. More users means better issues and more pull-requests. It's a positive feedback loop that is going to do more for your module than you could possibly imagine.

Keep your Dependencies Lean and Fresh

One of the toughest things about maintaining modules is maintaining your dependencies. When you're building an application you can shrinkwrap your dependencies to a hard version, but when you're building a module you need to be flexible. This means that your underlying dependencies can change without an explicit change from you.

Generally, setting a dependency version to 0.N.x (where N is a hard number) is safe, but be sure to stay on top of things or you're in for some unexpected surprises. It also means you should be careful what dependencies you take on. The more dependencies you don't maintain, the more unexpected complexity you may encounter.

Don't be Afraid to Refactor

There is nothing worse than coming back to a module after several months and realizing you made some bad initial decisions, or even worse your public API needs to change. Read up on Semantic Versioning and if you're module is pre-1.0.0, then don't be afraid to refactor. Refactoring can breathe new life into your module and your enthusiasm for it. Just be sure to update the version of your module appropriately and maintain a changelog.

In forever@0.10.0, we refactored the core programmatic Monitor functionality out into a new module: forever-monitor. This separation of concerns is important: forever is a CLI application for running forever-monitor with daemonization.

Be Wary of Compiled Dependencies

This is a subtle but brutally important lesson about writing Node.js modules. Taking on any compiled dependencies, whether as a node addon or 3rd party library, is going to increase the complexity of your install process by an order of magnitude. It is also going to (probably) make your module incompatible with your Windows users because they may not have a full compiler tool-chain.

In forever we learned this lesson the hard way. Until detached child processes were implemented in Node.js and libuv in 0.8.0 we were using a compiled add-on called daemon.node. At least 10% of all issues opened on forever (and a lot on npm itself) were related to this compiled dependency. Poorly configured Node.js installations, incompatible versions of Python (used by node-waf and node-gyp), and rebuilding between Node.js versions are just a small sample of the common errors your users will encounter if you go with compiled modules.

Chances are what you're looking for already has a pure Javascript implementation. If not, then just be aware what you're getting yourself into.

Support Your Users

When working with your users you have to divorce your ego. Be helpful, friendly, and responsive. Any attitude is poison to your success. Your first users will become your greatest champions in the Open Source community at large.

That being said, know when to say "no" and "not now". If your module gets successful you're going to get a lot of feature requests. It is not your responsibility to implement all of these features, nor to implement them immediately. Have a roadmap and stick to it.

Assess Pull-Requests Carefully

Once your module reaches a critical mass, you're going to get pull-requests. This is awesome. In the face of this awesomeness, however, don't lose sight of what's important. Make sure your own coding style, test coverage, and vision for your module do not suffer in your eagerness to merge a pull-request.

Unpublish Versions you won't Support

This lesson will probably draw some contention from the community at large. In fact, at Node Knockout last year, a team made an npm registry clone for ensuring this doesn't happen.

That being said, making this decision will help you explicitly manage the expectations of your users when they open issues on your module. By removing it from npm you're saying implicitly that you will no longer maintain that version. If you've ever tried to keep multiple version branches in-line with hotfixes to master you understand how difficult this is and why it's not always worth it.

With forever, older versions were unpublished when we decided to no longer support the lowest Node.js version supported by that module.

Know When to Throw in the Towel

This is a sad, but true, fact of maintaining Open Source modules. If you don't have the time to keep your module going issues are going to pile up, feature requests will go unanswered and users are going to know. This immediately forces your users to fill the void and start publishing their versions to npm. Having multiple forks of your project being used is only going to fracture whatever community you've managed to build up.

All is not lost however. Many of the most successful Open Source projects are no longer maintained by their creator. When you realize that your module is falling by the wayside reach out to your most active users. Who opened the most pull-requests? Who is the most active on your mailing list? Who helps users with issues? Any user who matches one or two of those questions is likely your best candidate for a new maintainer.

A Grain of Salt

These lessons come from years of experience writing and maintaining Node.js modules, but it is important to remember these experiences are my own. You may decided to do things differently and that is a good thing. Experiment, be creative and keep hacking!

The challenge:

We kicked off the fun-but-grueling event at Pivotal Labs in NY over beer and artichoke pizza. I stepped up to rehash the rules, Charlie gave a live demo, and Jon Gottfried of Twilio presented the Portal 2 Replica Gun (!!) being offered as a side prize for best Twilio hack.

After that, everyone broke and rushed off to hole themselves up in various conference rooms to code like mad for the next 17 hours. Some formed teams while others were 1 person armies. We made coffee. We drank Red Bull. We cracked jokes. Ideas were spun up and tossed aside. The hours wore on and we all started to get a bit bleary eyed.

The winner:

At 1:30pm the next day, we officially ended hack time. Charlie gathered everyone around for demos and I ran the conference call with our handpicked judging panel: Daniel Shaw, Carter Rabasa, Christian Sanz, Nuno Job (of Nodejitsu), Joe McCann and Visnu Pitiyanuvath. Judging was based on Originality, Completeness, Design, and Fun.

The judges selected You Laugh You Lose (requires either Canary or Opera), created by Francis Gulotta and Michael Shapiro. YLYL used OpenCV for facial recognition and flashes a bunch of funny pictures from internet memes. If you laugh, you’re disqualified. For added fun, you can play against others remotely.

It was great to have everyone involved, and it was especially cool that some folks who were new to node came out and used the event as an excuse to get themselves up and running. More photos from Hack Your Way to NodeConf can be found here.

Next stop, Portland!

I graduated from the University of Alaska Fairbanks in 2009. I stuck around for a few years and became a developer/programmer for the University of Alaska. About a year ago, I decided I wanted to move away; out of the cold, closer to family, and to expand my career horizon. I packed up 8 years of worth of collected belongings between my wife and I into our car and at 1:30pm June 18, 2011, I started my journey out of Fairbanks, AK to Seattle, WA.

I had a few interviews lined up when I arrived, but at the time the market was looking for senior developers. In the interim of looking for something to pay the bills, I got introduced to node.js and all its' awesomeness. Long story short, a friend and I decided to step out of our comfort zones and sign up for a competition in a language we knew nothing about (other than a little JavaScript/jQuery knowledge). We signed up for the Node Knockout 2011 competition. We had a trial run and did everything wrong and 26 hours in decided to start drinking, when we woke up 52 hours later we had an app called YUMDUM—Don't ask, we don't even know... As for how we did at the competition, we didn't place but we had fun leaning more about node.js, GTFS, and attempting to make a simple public transit routing application on the web.

Shortly after the competition I started a Support Engineer role at a local company in Seattle. There I got to learn what support was like at the first tier of a software company and all about graveyard shifts. In other words, I completed my Jedi training (in reverse) and became a Master of Support.

Fast forwarding to the past few weeks, I was on my normal graveyard shift when I decided to ping Josh and see what was up with him and Nodejitsu. We got to talking about how awesome his job was, how he could really use an awesome Support Engineer, and how together we could pave the way to glory for Nodejitsu with node.js. And then that talk became reality!

For my first project at Nodejitsu, I have been working on a bot that will make node.js code compliant with the version that is being released very soon. Actually, the change was first introduced in node.js v0.3.0 but it hasn't been an issue until node v0.8.0. this is because the “sys” module throws if you try to require it in an application.

The migratorBot has several modes, but the most interesting are "repo" and "npm". The "repo" command will take either a location to a folder or a git hub link to a repository. If you provide it with a Github repository it will take your provided username,fork the repository, clone it locally, check out a new branch, make any changes, push that commit back to Github, then it will submit a Pull Request to the original repository owner with a friendly message about why the changes were made.

The other interesting mode is "npm". This mode will query ALL of the packages in npm that have links to Github repositories and will process all 10,000+ repositories, this might be why your reading this blog if the migratorBot has left you with a pull request on Github. You’re Welcome!

This project has been fun and challenging as its been about six months since I have developed any serious node.js applications. Coming back onto the scene and being greeted with flatiron, winston, and various other highly useful frameworks and tools has been highly rewarding.

Hacking our way to NodeConf

Of course, we're heading to NodeConf this July and showing up in force. But we've got one extra ticket for the sold out conference, and we've figured out a fun way to give it away. This weekend we're hosting a hackathon in NY along with folks at Pivotal Labs and Twilio. Whoever puts together the best hack will win the last ticket to NodeConf, airfare, and space in the hotel with the Nodejitsu team.

Meet.js

On the other side of the world, Paolo recently travelled to Germany for Berlin Buzzwords, and being so close to Poland, he decided to pop in and pay Maciej and me a visit. This was a great excuse for me to invite him to JavaScript meetups I've recently started organizing - meet.js, a series of barcamps held in various Polish cities, originally founded by Damian Wielgosik.

We managed to get an interesting lineup in place for our most recent gathering on Monday, June 11th. Besides node.js basics (covered by Paolo) other topics were: behavior driven development, ECMAScript.next and functional programming. The event had more than 60 attendees and apparently vast majority was really curious about server-side of things.

Obviously we were onto something. I decided to quickly team up with gang of passionate developers at Future Simple known for organizing HackKRK hackathons and do something more to meet the needs of passionate Polish JS community.

Node.js workshops

We've decided to do a workshop conducted by Paolo. The interest simply went beyond our expectations. We've gathered one workshop group and had another on the waiting list. We were contacted by people from other countries as well - either willing to attend or help to organize such workshop in different locations. Apparently there are developers willing to travel across the country to listen to talks about Node. Of course this is a pattern - in the past conferences like NodeConf or jsConf sold out literally within seconds.

Spain.js

Another event we'll be present on is small conference in Madrid, Spain. I'll be speaking about The Pursuit of Simplicity; embracing the importance of sane decisions during design and development processes, the case of various libraries or frameworks and many more. So if you happen to be in sunny Spain make sure to drop me a line.

It has been a long time coming. We have been working on making Nodejitsu the best-of-breed Node.js platform with outstanding cloud orchestration and infrastructure tools to back it up for over two years. In that time, the Node.js community has grown organically into a vibrant group of dedicated professionals from all over the world. They are focusing on the right problems: creating and improving software and tools that set the standard for innovation.

Our commitment to the community remains steadfast and the pricing for Nodejitsu is designed to only further the goals of the community and our customers.

Our plans are designed to service both Individuals building non-critical applications or experiments and Businesses who need dedicated resources for hosting mission critical applications.

Individuals

Businesses

It may be surprising to you that Nodejitsu is not offering an unlimited free tier. This is intentional. The choice represents our deeply held belief that competing Platform-as-a-Service pricing models are doing damage to developer communities everywhere. We do not want to be part of the same problem.

And what is the problem at hand? Let me preface this by saying I am an technologist. I have been building things with technology as long as I can remember. So believe me that I say this with all the love in my heart: developers feel too entitled. Far too frequently in the Open Source community, I see the wasted potential of those focused on making demands instead of spending their energy on creating value. Don't forget, there are no refunds on Open Source projects.

It is, however, not completely their fault. Developers have been subject to pervasive incentives for years. In our Information economy those who understand and actively shape the flow of information (i.e. developers) hold a heavy sway over public opinion:

• Our families and friends value our opinions of what technology to buy.
• Our labors contribute to the consumer products reaching millions of people.
• The opinions of our communities -- both in the mainstream and the dissents -- shape the futures of multi-billion dollar corporations, whose executives pay close attention to what developers have to say.

Thus to leading tech companies, in both the consumer and enterprise markets (read: Google, Microsoft, Apple, etc) the mantra has become "make developers happy and ye shall succeed."

Impact on Cloud Computing

Now consider cloud computing. In their drive to make developers happy early Platform-as-a-Service companies introduce an unlimited "free tier" usually comprised of one or two compute units of varying size. This may be an energetic effort to increase product distribution, but it is the fundamental problem with Platform-as-a-Service pricing.

You may be asking yourself: "Huh? Isn't an unlimited free tier a good thing?" In reality, this pricing system distorts developers’ incentives, to the detriment of their customers and themselves. Investors with whom I’ve spoken who have placed bets with some of Nodejitsu’s competitors have told me that if they were to "do it over again, they would never have created a free tier."

1. It ignores the varying needs of customer segments

Very early in the life of Nodejitsu, based on feedback and user data from our private beta, two distinct customer segments became obvious to us:

Individuals: Developers or independent freelancers who wish to host personal applications which they expect small to moderate amounts of traffic. If these apps experience higher than average latency or less than a 24x7 SLA the disruption has a minimal effect on the customer.

Businesses: Small to medium sized businesses that want to host one or more applications in a real production environment. These applications need minimal latency and maximum uptime; disruptions could mean a real impact on their revenue or their reputation with their customers.

What both segments have in common is their wish to offload completely their developer operations to Nodejitsu (That said, if you already have a devops team why don't you checkout some of our Enterprise solutions?).

By creating two different pricing plans for both Individuals and Businesses we at Nodejitsu seek to find a middle ground for everyone.

2. It discourages giving back

Economists like to evaluate incentives by considering rational behavior and utility theory. To sum it up:

Rational behavior is a decision-making process that is based on making choices that result in the most optimal level of benefit or utility for the individual.

How might an unlimited free tier influence the rational behavior of developers? If the desired utility can be obtained for free, then there is no incentive for developers to contribute beyond improving a product and workflow they use regularly. Although this secondary incentive will elicit contributions from some developers there is a huge set of developers for whom this is not enough.

We are working on new and innovative ways to motivate developers to contribute to the Open Source community through the Nodejitsu platform.

3. It encourages "beating the system"

To save money on the free tier many Platform-as-a-Service companies will turn off applications while they are not in use. In theory, this may create economy for the platform companies, but by offering an unlimited free tier these companies have told developers "if you keep this application active it will stay up." I can't tell you how many times I've heard a ludicrously simple way of getting around these idle shutdown systems. This is why many of our competitors do not support long-held connections or WebSockets.

Nodejitsu was the first platform to support WebSockets and we remain committed to providing the best WebSocket and soft real-time experience on the web.

4. It causes price inflation for average users

So if an unlimited free tier encourages abuse of resources, who pays the price for the waste? Platform-as-a-Service companies really have two choices. The first is to operate at a loss. This has been the model for large technology companies with existing revenue models that are attempting to break into the market space. Operating at a loss is, obviously, not sustainable. As more platform companies announce pricing plans over the coming months, we expect many of them to transition to the second choice for dealing with this cost inefficiency: passing this cost on to the paying user. This is already a widespread practice in the platform startup space, and somehow it seems to be tolerated. For what should be an affordable resource, the markup on 300MB of RAM has really gotten out of control.

I’ll say it again: paying customers (many of them individual developers and freelancers) are subsidizing the warped incentive for developers to work around platform companies’ payment plan restrictions.

Our Commitment to You

These ideas all underpin our decision at Nodejitsu to limit the free tier in our Development Sandbox. We want you to try Nodejitsu today for free. We will do our best to ensure that your trial period lasts as long as possible. When that trial period is over we have paid plans starting at just $4 per month. We will also be offering many of our Enterprise Products as a service in addition to our existing Enterprise software model.

It may be difficult to accept, but it is our belief that a limited free tier will encourage developers to look beyond the free services of the platform and to consider how they can best use Nodejitsu's tools to continue to build a better community. We developers need to realize that we have the power to actively shape our surroundings through self-motivated creation instead of waiting for the next version of your favorite library. Fix a bug, fork a project, add a feature, you're going to make someone else's day. Stay tuned and keep creating!

A year and a half ago, Winston was created in an attempt to unify all potential logging transports. While winston serves its job well, another project needed to be added to the equation to help solve the problem mentioned above.

Winston's success has been defined by it's ability to instantiate loggers and add multiple different transports to each one.

History

It was a cold New York night in early 2010, and we were all huddled around a projector watching people present their new javascript libraries. I do not recall all of the speakers that night at NYC.js, but I do know that is where I met Marak Squires after he gave a talk on some outlandishly awesome webhook thing (now known as hook.io). He was dressed in a full on suit, sneakers, and a fitted NY baseball cap cocked slightly to the side. Being an inquisitive person, I decided to figure out what made this guy tick.

So I pulled him aside, as the meeting was breaking up, and introduced myself. After a short talk with him I realized he was at least as fanatical about javascript and node.js as I was. A group of us went back to the apartment he shared with Charlie Robins for beers and a "Javascript Party".

Things clicked pretty well between Charlie, Marak, and myself, and I would head over there nearly daily to hang out and hack on various node projects with Marak. Charlie was still a Microsoft dude by day and ruby guy by night. Over the course of a month or so, after enthusiastic debates and poking bits of fun, we actually got Charlie writing javascript under node. It did not take long until he had the bug, just as Marak and I had after seeing Ryan Dahl's jsconf.eu talk in late 2009. It was perfect timing to throw Paolo Fragomeni into the mix.

Paolo and I had been talking about node and jsdom over irc while he was living in Texas. Apparently he could sense what was going down in NYC and decided to make the move. At one point he asked me if I had a couch or knew of one he could crash on while he found a place. I explained that I lived an hour out of the way by subway, but I mentioned that Marak was more centrally located and there was potentially a couch there for him to crash on. It worked out and we gained another javascript/node fanatic!

Fun times, cut short

Shortly thereafter, my employer at the time took on a huge contract which required me and a couple others to carry the load of a 15+ person IT staff that was relieved of duty at a moments notice. It was a bloated legacy mess of enterprise java, and it was our job to keep the site running and eventually rebuilt it. There were many long nights, and my "Javascript Party" time was limited greatly.

Unfortunately, I missed the birth of nodejitsu, but I tried to see the nodejitsu guys as much as time would allow. A year later I decided to take my leave of NY and move to Arizona. I've spent nearly a year consulting, building open source projects, tinkering with hardware, enjoying nature, and trying to relax.

Working at nodejitsu

I am extremely excited to be working with the nodejitsu of today! They have built a solid and compelling product, and I simply cannot wait to help get them to the next level.

Now, this isn't necessarily a problem when it comes to building tools. In fact, I think it's a great demonstration of flatiron's flexibility. Each project can use exactly the libraries and organization that make sense for their applications, even for apps as different as haibu and jitsu. It can be confusing for newcomers, though, that are used to frameworks that enforce stronger conventions, or those that are trying to figure out how to get from "hello world" to something more profound.

I thought what would be helpful for flatiron newcomers is an in-depth tutorial, which iterates on a "hello world" flatiron server (like those in the project's examples) into a relatively complex webservice that uses some of flatiron's more complex functionality.

In this tutorial, I'm going to create a webservice for browserify, a popular client-side code framework for node.js. All of the code is on github, and every section of this tutorial corresponds to a code commit. In early parts of this tutorial I will show all the code, but as it gets longer I will typically explain the code with links to the relevant files/commits on github.

Flatiron HTTP Hello World

To get started, let's write a simple "hello world" app using flatiron, in ./app.js:

// Require flatiron and grab the app object.
var flatiron = require('flatiron'),
    app = flatiron.app;

// Use the http plugin. This makes flatiron act as an http server with a
// router on `app.router`.
app.use(flatiron.plugins.http);

// Route handler for http GET on the root path
app.router.get('/', function () {
  // The request and response objects are attached to `this`.
  var req = this.req,
      res = this.res;

  // Flatiron comes with a logging object already attached!
  app.log.info('Saying hello!');

  // Handle the response as you would normally.
  res.writeHead(200, { 'content-type': 'text/plain'});
  res.end('hello!');
});

// Start the server!
app.start(8080, function (err) {
  if (err) {
    // This would be a server initialization error. If we have one of these,
    // the server is probably not going to work.
    throw err;
  }

  // Log the listening address/port of the app.
  var addr = app.server.address();
  app.log.info('Listening on http://' + addr.address + ':' + addr.port);
});

Writing a basic flatiron server is pretty simple. First, we require flatiron and grab the 'app' object. Then, we assign a route handler to '/' which responds to request with the text, 'hello!'. This server also takes advantage of flatiron's built-in logging to print information to the screen.

Here's the accompanying ./package.json:

{
  "name": "flatiron-helloworld",
  "version": "0.0.0",
  "dependencies": {
    "flatiron": "0.1.16",
    "union": "0.3.0"
  }
}

The most important thing to notice here is that union is also a dependency. Union is what's called a "peer dependency". This means that flatiron doesn't itself list union as a dependency, but the http plugin expects to be able to require it. This is because flatiron has uses outside of http servers where it doesn't make sense to install union.

Luckily, if you forget, flatiron will return a reasonable error message:

$ node app.js 
flatiron.plugins.http requires the `union` module from npm
install using `npm install union`.
Trace: 
    at Object.<anonymous> (/home/josh/dev/nodejitsu/flatiron-tutorial/hello/node_modules/flatiron/lib/flatiron/plugins/http.js:25:11)
    at Module._compile (module.js:441:26)
    at Object..js (module.js:459:10)
    at Module.load (module.js:348:31)
    at Function._load (module.js:308:12)
    at Module.require (module.js:354:17)
    at require (module.js:370:17)
    at Object.http (/home/josh/dev/nodejitsu/flatiron-tutorial/hello/node_modules/flatiron/node_modules/broadway/node_modules/utile/lib/index.js:261:14)
    at Object.<anonymous> (/home/josh/dev/nodejitsu/flatiron-tutorial/hello/app.js:7:25)
    at Module._compile (module.js:441:26)

Once flatiron and union are both installed, of course, our app works as expected:

$ node app.js 
info: Listening on http://0.0.0.0:8080

Now we can make a request to our server and receive a kind message:

$ curl localhost:8080
hello!

Furthermore, we can see evidence of our request in the server's logs:

$ node app.js 
info: Listening on http://0.0.0.0:8080
info: Saying hello!

"Hello World" With A Twist: Configuration and POST

Let's make our "Hello World" app slightly more sophisticated:

var path = require('path');

var flatiron = require('flatiron'),
    app = flatiron.app;

// Set up app.config to use ./config.json to get and set configuration settings.
app.config.file({ file: path.join(__dirname, 'config.json') });

app.use(flatiron.plugins.http);

// This router syntax allows you to define multiple handlers for one path based
// on http method.
app.router.path('/', function () {

  // This is the same functionality as previously.
  this.get(function () {
    this.res.writeHead(200, { 'content-type': 'text/plain' });
    this.res.end('hello!');
  });

  // Now, when you post a body to the server, it will reply with a JSON
  // representation of the same body.
  this.post(function () {
    this.res.json(200, this.req.body);
  });
});

// Now we're using app.config to set the port, with a default of 8080.
app.start(app.config.get('port') || 8080, function (err) {
  if (err) {
    throw err;
  }

  var addr = app.server.address();
  app.log.info('Listening on http://' + addr.address + ':' + addr.port);
});

The accompanying package.json has a few minor changes as well:

{
  "name": "browserify-cdn",
  "version": "0.0.0",
  "scripts": {
    "start": "node ./app.js"
  }
  "dependencies": {
    "flatiron": "0.1.16",
    "union": "0.3.0"
  }
}

Besides a name change, our app also now has a "scripts.start" field. This means two things:

• 'npm start' will start your app.
• 'jitsu deploy' will successfully start your app.

Note that this app also has configuration now, using app.config! We're using this to load configuration settings from ./config.json:

{
  "port": "3600"
}

So, if you run this server and GET /, the response is the same as before:

$ curl localhost:3600
hello!

However, if we POST some data:

$ curl -X POST -d 'foo=foo&bar=bar' localhost:3600
{"foo":"foo","bar":"bar"}

We can also post JSON if we remember to set the content-type header properly:

$ curl -X POST -d '{ "foo": "foo", "bar": "bar" }' -H 'Content-Type: application/json' localhost:3600
{"foo":"foo","bar":"bar"}

This all Just Works is because flatiron has built-in tools for buffering and parsing request bodies, making the development of JSON APIs dead easy.

Now we're ready to do something a bit more interesting.

Basic Browserifying Server & Separation of Concerns 101

Now we're going to make our app browserify posted code snippets. Once it's done, the interface should work like so:

$ curl -X POST -d 'var traverse = require("traverse");' localhost:3600
var require = function (file, cwd) {
    var resolved = require.resolve(file, cwd || '/');
    var mod = require.modules[resolved];
    if (!mod) throw new Error(
        'Failed to resolve module ' + file + ', tried ' + resolved
    );
    var res = mod._cached ? mod._cached : mod();
    return res;
}

require.paths = [];
require.modules = {};
require.extensions = [".js",".coffee"];

require._core = {
    'assert': true,
    'events': true,
    'fs': true,

**snip**

I didn't include the entire bundle here, but the general idea is this: You post browserify-able javascript code to the service, and it returns the browserified bundle.

By this point, our app is getting big enough that we want to break functionality up into different pieces. The way I approach this is like so:

1. Write a library (or a few) that implement the basic pieces of functionality you need.
2. Call that library from ./app.js, only handling http details in the router.

The idea here is to separate the concerns so that there isn't routing logic in the bundler or bundling logic in the router. In larger apps, this approach gets broken down further into large-scale design patterns like MVC. In this case our app is pretty small though, so simply separating the bundling into its own library will suffice.

The bundler, which lives in ./bundler.js for this iteration, is a constructor that inherits from EventEmitter2:

var EventEmitter2 = require('eventemitter2').EventEmitter2,
    browserify = require('browserify'),
    detective = require('detective'),
    npm = require('npm'),
    crypto = require('crypto'),
    util = require('utile');

// I wrote the Bundler as a constructor with prototype methods.
// I find that it's a good fit for stateful problems.
var Bundler = module.exports = function (opts) {

  // This lets you create a new Bundler without the 'new' keyword.
  if (!(this instanceof Bundler)) {
    return new Bundler;
  }

  var bundler = this;

  // Set the bundler's persistent options here. Also handle defaults.
  bundler.options = opts || {};
  bundler.options.npm = bundler.options.npm || {};

  // Overrides falsy values such as `undefined`
  if (bundler.options.cache !== false) {
    bundler.options.cache = true;
  }

  // Bundler inherits from an EE2 with wildcards and the :: delimiter.
  EventEmitter2.call(this, {
    wildcard: true,
    delimiter: '::',
    maxListeners: 0
  })

  // Bundler requires a loaded npm in order to work properly.
  // The rest of the code in the constructor sets this up and emits events
  // to signal when it's ready.
  bundler.ready = false;

  npm.load({}, function (err) {
    if (err) {
      // What would be really cool is if I checked to see if there was an error
      // event listener or not.
      bundler.emit('error', err);
    }

    bundler.ready = true;
    bundler.emit('bundler::ready');
  });
};

util.inherits(Bundler, EventEmitter2);

The bulk of the constructor handles setting default options for our object and configuring the EventEmitter2 aspects of the object. The interesting piece here calls npm.load on the npm singleton object, emits events when it succeeds or fails, and sets a this.ready flag. This allows us to create our object and then listen for when we may use it.

Next, we'll define the prototype method that actually bundles our code:

// The 'bundle' method is what actually attempts to bundle your project.
Bundler.prototype.bundle = function (src, cb) {
  var bundler = this,
      modules = detective(src);

  // We used 'detective' to get a list of needed modules, so that we can make
  // sure they're installed before trying to browserify.
  npm.commands.install(modules, function (err) {
    if (err) {
      cb(err);
    }

    var bundle;

    // Attempt to browserify the passed-in javascript source
    try {
      bundle = browserify(this.options)
        .addEntry('index.js', { body: src || '' })
        .bundle();

      // Hit the callback with our complete bundle object.
      cb(null, {
        src: src,
        md5: crypto.createHash('md5').update(src).digest('base64'),
        bundle: bundle
      });
    }
    catch (err) {
      cb(err);
    }
  });
};

This method handles making sure that required modules are installed, and then attempts to browserify the code. It then calls back with an object representing the code bundle.

Finally, let's add .attach and .init methods, which will let our new module act as a broadway plugin:

// Bundler can be used as a broadway plugin.
Bundler.attach = function (opts) {
  this.bundler = new Bundler(opts);
}

// The major win we get from making Bundler attachable is that we can integrate
// its initialization step with our app.
Bundler.init = function (done) {
  var npm = 'npm'.red.inverse;

  if (this.bundler.ready) {
    console.log('init: %s is ready.', npm);
    return done();
  } else {
    console.log('init: Waiting for %s.', npm);
    this.bundler.once('bundler::ready', function () {
      console.log('init: %s is ready.', npm);
      return done();
    });
  }
}

Now that the bundler library is done, all we have to do is integrate it with our ./app.js.

First, let's talk about how the bundler was integrated into the app. After requiring it, we use bundler like so:

// This is a new library used for running browserify bundle jobs for us.
var bundler = require('./bundler');


// Note that 'bundler' is written to work as a flatiron plugin.
app.use(flatiron.plugins.http);
app.use(bundler);

This creates a property called app.bundler, and adds an initialization step so that when we call app.start npm is also loaded.

Later, we use app.bundler in our POST handler like so:

this.post(function () {

  // If the request body doesn't have the property we expect, it's assumed
  // to be raw javascript. Note that the raw unparsed body is buffered into
  // req.chunks (as a Buffer).
  var req = this.req,
      res = this.res,
      js = req.body.js ? req.body.js : req.chunks.toString();

  // 'app.bundler' was created when we attached 'bundler'.
  app.bundler.bundle(js, function (err, data) {
    if (err) {
      return res.json(500, {
        success: false,
        reason: err.message || 'unknown'
      });
    }
    res.writeHead(200, { 'content-type': 'text/javascript' });
    res.end(data.bundle);
  });
});

Note that most of the code here is about figuring out what the user wants to browserify and how to respond in return, and that all the bundling takes place in a single call.

Finally, here's the ./package.json for completeness:

{
  "name": "browserify-cdn",
  "version": "0.0.0",
  "scripts": {
    "start": "node ./app.js"
  },
  "dependencies": {
    "flatiron": "0.1.16",
    "union": "0.3.0",
    "utile": "0.0.10",
    "eventemitter2": "0.4.x",
    "browserify": "1.10.6",
    "detective": "0.1.x",
    "npm": "1.1.x"
  }
}

The only important changes are in the dependencies field, since we now have a few more of them!

Finally, we can run our service with npm start, and when a user runs the curl command from earlier (curl -X POST -d 'var traverse = require("traverse");' localhost:3600) they will receive a bundle and our logs will look like so:

$ npm start
npm info it worked if it ends with ok
npm info using npm@1.1.14
npm info using node@v0.6.14
npm info prestart browserify-cdn@0.0.0
npm info start browserify-cdn@0.0.0

> browserify-cdn@0.0.0 start /home/josh/dev/nodejitsu/flatiron-tutorial
> node ./app.js

init: Waiting for npm.
init: npm is ready.
info: Browserify-CDN Listening on http://0.0.0.0:3600
npm http GET https://registry.npmjs.org/traverse
npm http 304 https://registry.npmjs.org/traverse
npm info into /home/josh/dev/nodejitsu/flatiron-tutorial traverse@0.6.1
npm info installOne traverse@0.6.1
npm info unbuild /home/josh/dev/nodejitsu/flatiron-tutorial/node_modules/traverse
npm info preuninstall traverse@0.6.1
npm info uninstall traverse@0.6.1
npm info postuninstall traverse@0.6.1
npm info preinstall traverse@0.6.1
npm info build /home/josh/dev/nodejitsu/flatiron-tutorial/node_modules/traverse
npm info linkStuff traverse@0.6.1
npm info install traverse@0.6.1
npm info postinstall traverse@0.6.1
traverse@0.6.1 ./node_modules/traverse

Congratulations, you now have a working browserify webservice!

Add Caching with Resourceful

So, our browserify service works, but there's a problem: Our browserify webservice has to bundle the project every time, even if it's done it before! We can minimize this with caching.

To facilitate this, I reorganized the project a little bit:

$ ls
app.js  config.json  lib  node_modules  package.json  post
$ ls -R ./lib
./lib:
bundler  common.js

./lib/bundler:
bundler.js  cache.js  index.js

As you can see, I moved bundler.js into ./lib/bundler and added a cache.js and an index.js.

You'll also notice that I removed config.json from the git repository. This is because it now includes credentials to a couchdb. Mine looks like this:

{
  "port": "3600",
  "couch": {
    "https": true,
    "uri": "**********.iriscouch.com/browserify",
    "port": "6984",
    "auth": {
      "username": "**********",
      "password": "**********"
    }
  }
}

In order to add caching, the approach I took was to create a new constructor which inherits from Bundler, called CachingBundler, in ./lib/bundler/cache.js. This new constructor contains a new prototype method, .get, which checks our cache and only calls .bundle if required. This sort of inheritance pattern is appropriate for cases where your new object is "an x that also does y"---for example, Bundler is an EE2 that also bundles, and CachingBundler is a Bundler that also caches.

Our new library starts with a long list of requires:

var Bundler = require('./bundler'),
    common = require('../common'),
    resourceful = require('resourceful'),
    errs = require('errs'),
    colors = require('colors'),
    utile = require('utile');

Bundler and common are internal libraries, while resourceful, errs and utile are all flatiron components. This caching layer uses resourceful to interact with a storage backend (in this case, couchdb). Resourceful is an "Object Document Mapper," or ODM, which means that it internally maps javascript objects and prototypes with documents in a document store like couch. This saves developers the step of wrapping their couchdb http calls in a custom abstraction, and is typically used to put the "model" in MVC.

Here's what creating a resource with resourceful looks like:

// Cached bundle resources.
var CachedBundle = resourceful.define('cachedBundle', function () {
  this.string('md5');
  this.string('source');
  this.string('bundle');
});

This resource is quite simple. It's called a 'cachedBundle', and is expected to at least have three strings (md5, source and bundle).

Now, we can create our constructor:

// A version of the bundler that caches. Use `.get(text)`.
var CachingBundler = module.exports = function (opts) {
  // This lets you create a new caching Bundler without the 'new' keyword.
  if (!(this instanceof CachingBundler)) {
    return new CachingBundler;
  }

  opts = opts || {};
  Bundler.call(this, opts);
  // Set up resourceful to use couchdb.
  resourceful.use('couchdb', opts);
}

utile.inherits(CachingBundler, Bundler);

Generally, this constructor just repeats the actions of the original Bundler constructor and then inherits from it. However, it also sets up resourceful to use our couchdb, using the options passed into it.

The meat of our new functionality is in the get prototype method:

CachingBundler.prototype.get = function (src, cb) {
  var self = this,
      md5 = common.md5(src);

  // This logging function simply adds some extra text to the message and
  // then emits it with the event 'log::lvl' (ex: 'log::info').
  function log (lvl, msg) {
    var restOf = [].slice.call(arguments, 2),
        data = utile.format.apply(
          null,
          [ '(md5 `%s`) '.cyan + msg, md5 ].concat(restOf)
        );

    self.emit('log::' + lvl, data);
  }

  log('info', 'Trying to find bundle...');

  // Check to see if the file already exists in the cache, based on md5 hash of
  // the source text. This logic accounts for the possibility of hash collisions
  // and multiple versions of the same bundle.
  CachedBundle.find({ md5: md5 }, function (err, bundles) {
    if (err) {
      return cb(realError(err));
    }

    var matches;

    if (!bundles.length) {
      // No cached bundles found. We should bundle.
      log('warn', 'Not found.');
      return bundle(src);
    }

    // Log the document id's for found bundles. If there is more than one
    // bundle, it means there was either an md5 hash collision or multiple
    // documents for the same bundle exist.
    log('info', 'Found bundle(s): %j', bundles.map(function (doc) {
      return doc._id;
    }));

    // Try to find one that has the same text.
    matches = bundles.filter(function (bundle) {
      return bundle.source == src;
    });

    // Any answer other than 1 means an actual md5 hash collison.
    switch (matches.length) {
      case 1:
        log('info', 'Match found.');
        finish(null, matches[0]);
        break;
      case 0:
        log('info', 'Hash collision with existing document(s) %j', matches);
        bundle(src);
        break;
      default:
        log('Multiple documents describing this document exist.');
        log('Showing the first one.');
        finish(null, matches[0]);
        break;
    }
  });

  // This function is called when we need to bundle. It addition to logging, it
  // also attempts to cache the resulting bundle.
  function bundle (src) {
    log('info', 'Bundling.');
    self.bundle(src, function (err, doc) {
      if (err) {
        return cb(err);
      }

      // Here, we create a new CachedBundle, which maps to our data store.
      var bundle = new CachedBundle(doc);

      // This is how you save resources with resourceful.
      log('info', 'Caching bundle for next time...')
      CachedBundle.save(bundle, function (err, bundle) {
        if (err) {
          log('error', 'Error while caching bundle');
        }
        return finish(realError(err), bundle);
      });

    });
  }

  // This is pretty much the last step, whether the doc came from the cache
  // or from a bundling.
  function finish (err, bundle) {
    if (err) {
      return cb(err);
    }

    log('info', 'Returning bundle.');
    cb(null, bundle);
  }

};

Most of the logic you can see involves grabbing the document, and making sure it's the right one, and logging the behavior using the EventEmitter2 api (remember, Bundler inherited from EventEmitter2). Interaction with our datastore is simplified to the creation of a resource type, a call to .find and a call to .save.

Like Bundler, our caching bundler also has .attach and .init methods for integration with flatiron:

// Attach the caching bundler, passing in our config options for the couch.
// Also emits logging events to the app.
CachingBundler.attach = function (options) {
  var app = this;

  this.bundler = new CachingBundler(utile.mixin(
    options,
    this.config.get('couch')
  ));

  // Little known fact about flatiron: The logging plugin will log messages
  // emitted on the `log` namespace of the core app, which is an instance of
  // EventEmitter2.
  app.bundler.on('log::**', function (msg) {
    app.emit(this.event, msg);
  });

};

CachingBundler.init = Bundler.init;

The init method is exactly the same. The new behaviors are in our attach method. One of these behaviors is to pass our app's couchdb configuration to the object constructor. The other behavior is to re-emit logging events on the app object, whose log plugin will handle them.

I also created a ./lib/bundler/index.js which looks like so:

module.exports = require('./cache');

and allows us to simply require our new caching bundler in ./app.js like so:

var bundler = require('./lib/bundler');

Now, let's run our app and do another curl -X POST -d 'var traverse = require("traverse");' localhost:3600:

$ npm start
npm info it worked if it ends with ok
npm info using npm@1.1.14
npm info using node@v0.6.14
npm info prestart browserify-cdn@0.0.0-3
npm info start browserify-cdn@0.0.0-3

> browserify-cdn@0.0.0-3 start /home/josh/dev/nodejitsu/flatiron-tutorial
> node ./app.js

init: Waiting for npm.
init: npm is ready.
info: Browserify-CDN Listening on http://0.0.0.0:3600
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Trying to find bundle... event=log::info
warn: (md5 `RxLf0LntDlue3quYVoaVtw==`) Not found. event=log::warn
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Bundling. event=log::info
npm http GET https://registry.npmjs.org/traverse
npm http 304 https://registry.npmjs.org/traverse
npm info into /home/josh/dev/nodejitsu/flatiron-tutorial traverse@0.6.1
npm info installOne traverse@0.6.1
npm info unbuild /home/josh/dev/nodejitsu/flatiron-tutorial/node_modules/traverse
npm info preuninstall traverse@0.6.1
npm info uninstall traverse@0.6.1
npm info postuninstall traverse@0.6.1
npm info preinstall traverse@0.6.1
npm info build /home/josh/dev/nodejitsu/flatiron-tutorial/node_modules/traverse
npm info linkStuff traverse@0.6.1
npm info install traverse@0.6.1
npm info postinstall traverse@0.6.1
traverse@0.6.1 ./node_modules/traverse
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Caching bundle for next time... event=log::info
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Returning bundle. event=log::info

Naturally, this request wasn't cached so our app had to bundle it. If you check your couch with futon, you'll see that our document has been stored by resourceful. So, if we curl our api again:

info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Caching bundle for next time... event=log::info
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Returning bundle. event=log::info
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Trying to find bundle... event=log::info
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Found bundle(s): ["f4328d0a958746dfa8b45856a900134e"] event=log::info
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Match found. event=log::info
info: (md5 `RxLf0LntDlue3quYVoaVtw==`) Returning bundle. event=log::info

Since we've already bundled that particular source snippet, we may simply return it! Now we're cooking.

What Now?

In this quick tutorial, I showed you how I built the foundation for the new browserify-cdn project, which is hosted at http://browserify.nodejitsu.com, by starting with a "hello world" flatiron example and then iterating on it until we had a fully-working api server. Along the way, we learned a little about flatiron core's capabilities and many of its supporting libraries, as well as touching on concepts for organizing medium-sized apps.

The master branch has a number of tweaks and improvements, the most noticeable of which is serving an html landing page and static content using my static server middleware and adding more colors to the text output. Of course, now that you have your own browserify-cdn, you can change it however you want!

If you do, don't forget to send me a pull request.

We are also co-hosting two events in April: The Node Philly Conference and the NYC MongoDB Hackaton and speaking at three more!

Here's our schedule: be sure to say hi and hangout.

JSConf (&NotConf)

It's JSConf! Enough said! Pre conference will be notconf and it is looking amazing!

Web-5 Conference (Béziers, April 4-6)

Maciej and Charlie will be attending the Web-5 conference in Béziers, France. The conference revolves around Web Technologies and has some great speakers. Great cheese and great weather are also givens. You can register for the conference here.

Dyncon (Stockholm, April 21-22)

Dyncon is an fantastic event. Charlie spoke last year and had a blast. Nuno will be doing the honors for Nodejitsu this year, and we expect great talks from a great selection of speakers.

Node Philly (Philadelphia, April 23)

Node Philly (node.ph) is the first ever node.js conference in Philly. Charlie and Paolo are attending, as well as some great minds from the Node.js community like James Halliday and Mikeal Rogers. The event is part of a larger initiative called Philly Tech Week. Tickets are available for a nominal fee.

MongoDB Hackaton NYC (NYC, April 27)

We are co-hosting the MongoDB hackaton with 10gen. Charlie will be on location to help you out with your applications, and get you some free Nodejitsu accounts. This event is free and will be in the 10gen NYC headquarters.

Now, after three years of toiling away on Github, we have now expanded our engineering team to over fourteen strong and every Nodejitsu developer is an active contributor to the open-source community.

• How to become an open-source developer?
• How do you get recruited as an open-source developer?

How to become an open-source developer?

Becoming an "open-source developer" is actually quite simple. You really only need to do one thing: contribute.

Contributing can be done on several levels of participation. One of the easiest ways to contribute to a project is to use it, find an issue, and report the problem. If you've found an actual problem that can be resolved, hopefully one of the maintainers will respond by fixing the issue or providing guidance on how to resolve the issue.

There are also several other useful things you can do such as: unit-tests, benchmark suites, documentation, multi-platform support, third-party integrations, and new functionality.

How do you get recruited as an open-source developer?

If you are contributing ( as described in the previous paragraph ), you've already done the hardest part.

Good work always speaks for itself.

If you have worked on projects which are significant to companies who are hiring developers, they probably already know your name. It's much more likely a company will hire a developer who has already made an improvement to a piece of their codebase versus someone who has little knowledge of any company projects. At Nodejitsu, we primarily hire developers who have already contributed to major company supported projects like: jitsu, haibu, flatiron, and hook.io.

How to recruit open-source developers?

Recruiting open-source developers is not difficult, but it can be very difficult for certain companies.

First off, don't cold email open-source developers. If an open-source developer has posted their email in a public place ( like on Github ), they probably put it up for technical support emails, not recruiter spam.

Sponsoring hackathons with open-source incentives is always a great idea.

Allow company time for open-source projects.

This isn't about setting an arbitrary percentage of time like 10% or 20% of hours. If an employee is working on a project in their free time that could help improve open-source software in a significant way, you should encourage them to work on it. Hopefully, the projects they are working on in their free time align with the goals of your company.

Use algorithmic recruitment to find developers.

http://www.githits.me/ is fun.

http://coderwall.com/leaderboard is also good.

http://develop.github.com/ is great.

The data is out there. Do some googling and get creative.

Upcoming jobs at Nodejitsu

At Nodejitsu, our developers are top-class and their open-source contributions stand on their own. Our developers are constantly spammed for job offers and choose to work for us because we try to let them do the work they would be doing anyway.

Over the next few months, we will be aggressively expanding our development team and hiring new people who can step up in any of the following roles.

• Support Engineering
• Development Operations
• Front-End Development
• User-Experience
• Design and Graphic Art

We are looking for people who are self-motivated and are always willing to take the extra step needed to get the job done. We are looking for people who can not only provide results, but can also proactively take the initiative to get results.

Location: On-Site in San Francisco or New York City, Telecommute

• Compensation is competitive and based on experience
• Health insurance (including dental and vision)
• Equity opportunities available for senior roles

If you are interested in working for Nodejitsu, send an email to hiring@nodejitsu.com or better yet...

Get a pull request merged on Github.

Now get coding.

Asynchronous functions

An asynchronous function is one that usually returns before having finished its job and takes a function it should call when the job is done. that action is the callback, a function to be executed at that point in time.

// async
var fs = require('fs');
// callback
function logContents(err,contents) { console.log(contents); }
var contents = fs.readFile('a.txt', 'utf8', logContents); 
console.log('1');
// outputs 1, then the contents of the file when the file is read

vs.

// sync
var fs = require('fs');
var contents = fs.readFileSync('a.txt', 'utf8'); 
console.log(contents);
console.log('1');
// outputs the contents of the file, then 1.

Iterating in javascript with asynchronous functions

The problem

When someone is starting with node i always recommend they watch pedro teixeira's video nodetut on asynchronoys iteration patterns

In this nodetuts video Pedro shows how to iterate asynchronously in node.js. here is the naive approach code he shows which doesn't work:

// simulating an asynchronous call, e.g. getting something from a database
var asyncFn = function(data, callback) {
  var timeout = Math.ceil(Math.random() * 3000);
  // run callback after timeout miliseconds
  setTimeout(function() { callback(null, data); }, timeout);
};

var insertAll = function(coll, callback) {
  coll.forEach(function (nr) {
    asyncFn(nr, function(err,data) {
      console.log('‣', data);
    });
  });
  callback();
};

insertAll([1, 2], function() {
  console.log('done'); });

This will not work as you probably think it does:

done
‣ 1
‣ 2

The reason is that the insertAll function returns before the async function calls the callback so done gets logged before the function is complete.

Solving the problem with a module

If you think implementing your own asynchronous iteration function is too much hassle there's plenty of modules out there that solve this problem. my favorite is async.

In async the solution for this problem would be something like:

async.forEach([1, 2], asyncFn, function(err, results){
   console.log('done');
});

To iterate is human, to recurse is divine

When looking to the solution to this problem without the help of a module it struct me as obvious how much all these algorithms work like pattern matching.

• if the queue exists run some generic processing the queue function
• if the queue is empty run the callback

What is pattern matching?

Pattern matching is a recursive way to develop your programs where you define patterns for the program to identify and match those patterns with actions that should be executed.

Let's consider the very well understood map function that takes a collection and transforms each element by applying a function:

[1,2,3].map(function duplicate(x) { return x*2; });

If you solved this problem using pattern matching the execution of this function would be:

transformed_map = [];
map:
  l = [1,2,3]
    is l an empty array?
      no.
    is l anything?
      yes.
      head = 1
      tail = [2,3]
      transformed_map.shift(duplicate(1));
      map([2,3])
transformed_map = [2];
map:
  l = [2,3]
    is l an empty array?
      no.
    is l anything?
      yes.
      head = 2
      tail = [3]
      transformed_map.shift(duplicate(2));
      map([3])
transformed_map = [2,4];
map:
  l = [3]
    is l an empty array?
      no.
    is l anything?
      yes.
      head = 3
      tail = []
      transformed_map.shift(duplicate(3));
      map([])
transformed_map = [2,4,6];
map:
  l = []
    is l an empty array?
      yes.
      return transformed_map
[2,4,6]

Solving the problem with pattern

I decided try to implement pattern matching in javascript as my js1k entry called functional love.

The rules were simple: implement functions that allow you to do pattern matching in javascript without changing the language itself.

This is what i came up to for the node tuts episode:

insert_all([], function done() { console.log('done'); });
insert_all(_, function all(l) {
  insert_element(l.shift(), function (err, elem) {
    console.log('‣ ', elem);
    insert_all(l);
  });
});

• if the array is empty, log the message done.
• if its not empty, insert the element and when the callback is complete recursively call the insert_all function with the rest of the list.

You can now call this function like this:

insert_all([1, 2, 3, 4, 5, 6, 7, 8, 9, 10]);

How it works

You probably noticed i use the same syntax to declare the insert all function and to call the function itself. what happens is the first time you call pattern it sets the arity of the function. is this case the first pattern has two arguments, meaning i should expect a function with one argument. whenever i call insert_all with one argument i'm calling the function.

When you registered the first two patterns i created a fact table like:

Now when you call the insert_all function we need to iterate this fact table and see which rule applies. then call that function. simple. if the function wants to recurse it simply need to invoke the insert_all function.

Mad science corner

Ok enough with the easy stuff. let's use pattern to implement a function that reverses an javascript array. asynchronously of course:

reverse([5,4,3,2,1], function (list) { console.error(list); });

We can do this using the reduce function that takes a list l and an accumulator z and returns something that was constructed by applying a function f to each element of the list x and the accumulator z.

// pretending asynchronous behavior
// image this is some kind of echo server
function echo(z, x, cb) {
  setTimeout(function() {
    z.unshift(x); cb(z);
  }, Math.ceil(Math.random() * 100)); 
}

function reverse (l, cb) { reduce(echo, [], l, function(z) { cb(z); }); }

Now all we need is to implement the reduce function with pattern:

var _;
reduce(_, _, [], _, function (f, z, l, cb) { cb(z); });
reduce(_, _, _, _, function (f, z, l, cb) {
  f(z, l.shift(), function (new_z) {
    reduce(f, new_z, l, cb);
  });
});

Get the full code for this example here.

Js1k

pattern is my js1k submission. it was a lot of fun to write. if you are wondering if you should submit an entry yourself you totally should.

I like broadway a lot, so I'd like to take this opportunity to explain how it works and why you should be using it.

Here are a few phrases and bits of jargon that tend to pop up when discussing broadway:

• Inversion of Control
• Dependency Injection / Feature reflection
• Application extensibility

In this post, I'm going to explain what the mean, how they work in broadway, and why they're a good thing.

More after the break!

What's Inversion of Control?

Inversion of control is basically just a fancy term for plugins. For example, connect and express middlewares implement a sort of inversion of control:

connect.use(function invertedControl (req, res, next) {
  // Function now has control of our request and response

  // Code that may or may not manipulate the req and res objects.

  next();
});

The idea of "inversion of control" is usually contrasted with an older "procedural" paradigm. When writing code in a procedural style, you would have a central piece of code and change its behavior directly. Here's an example with python 2.7's BaseHTTPServer:

from BaseHTTPServer import HTTPServer, BaseHTTPRequestHandler

class RequestHandler(BaseHTTPRequestHandler):
  def do_GET(s):
    s.send_response(200)
    s.end_headers()
    s.wfile.write('Pong!')

server = HTTPServer(('', 8080), RequestHandler)


server.serve_forever();

This snippet runs a synchronous http server and uses classes for handlers. There isn't an equivalent here for node, since node's http server is asynchronous.

Even though it sounds complicated, it simply describes a family of design patterns where an object is manipulated directly by a called function.

In broadway, a basic example of inversion of control looks like this:

var app = new broadway.App;

// This will throw an error.
//app.say('Oh, what a world.');

// This plugin is often a separate module.
var sayPlugin = {
  attach: function attachSay () {
    this.say = console.log;
  }
}

app.use(sayPlugin);

// Will print 'Hi world!' as we expect.
app.say('Hi world!');

In this example, when our app .uses the sayPlugin, the attachSay function gains control of the app object (referred to in the scope of attachSay as this). Now all we need to do to give our broadway app a working .say is use the sayPlugin plugin!

This of course hardly scratches the surface of broadway's capabilities.

What are Dependency Injection and Feature Reflection?

Really, dependency injection is a particular flavor of these "inversion of control" patterns. Yes, I said the 'p-word'. But don't worry! All this means is that dependency injection is a 'thing'.

Dependency injection implies 3 things:

• An injection container that uses plugins
• plugins that modify the behavior of the container
• A way to for some plugins to depend on other previously-loaded plugins

The Wikipedia article on the subject says as much, but focuses on the use of dependency injection in static languages like Java. For instance, the Spring framework for Java implements this approach. You already saw an example of the same style of "IoC container" in this article. The Spring framework's version, perhaps unsurprisingly, requires you to write as much XML as Java (perhaps more). I don't say this to dig on Java or Spring; I say this to point out how much more lightweight it can be in a dynamically typed language such as JavaScript.

In the previous broadway example, the app was our injection container and it used the plugin sayPlugin. But what about dependencies?

I'm fond of comparing building applications with broadway to assembling a megazord because of the way you can use multiple plugins that depend on each other to build a whole. For an example, suppose I'd like to give my app a head that lets it say things:

// head.js

exports.name = 'head';

exports.attach = function attachHead (opts) {

  var head = this.head = {};

  head.say = function speak () {
    var words = Array.prototype.slice.call(arguments).join(' ');

    console.log('The stranger looks at you and says, "' + words + '"');
  };

};

Now suppose I want to give my future bot a sweet hat:

// hat.js

exports.name = 'hat';

exports.attach = function attachHat (opts) {

  // This plugin *depends* on the "head" plugin being loaded.
  if (!this.head) {
    throw new Error('You can\'t wear a `hat` without a `head`!');
  }

  var stranger = this;

  stranger.hat = {};
  stranger.hat.tip = function hatTip () {

    stranger.head.say('Howdy stranger.');
    console.log('He tips his hat in that half-Tom-Selleck, half-Frank-Sinatra manner.');
  };

};

In this example, if the app has a head, it can wear a hat and gains the ability to .tip its .hat. By checking for this.head, our hat plugin requires that we've already .used the head plugin.

Here's what this looks like together:

// index.js

var broadway = require('broadway');

var hat = require('./hat'),
    head = require('./head');

var stranger = new broadway.App;

// Attach a head!
stranger.use(head);

// Now you can wear a hat!
stranger.use(hat);

// Tip dat hat
stranger.hat.tip();

Once we give our stranger a head, it can wear a hat and do a pretty good cowboy impression:

% node index.js 
The stranger looks at you and says, "Howdy stranger."
He tips his hat in that half-Tom-Selleck, half-Frank-Sinatra manner.

What About Application Extensibility?

We at Nodejitsu are excited about broadway because it allows for something we're calling "application extensibility". What does that mean?

Suppose that you would like to make a hooks-based plugin system for your broadway app, where the main module looks like so:

// index.html

var broadway = require('broadway'),
    hooks = require('./hooks'),
    app = new broadway.App,
    fooHook = {};

// Example of a hook we may want to attach,
// depending on an event hook interface
fooHook.attach = function attachFooHook () {
  this.hooks.foo = function () {
    console.log('You just called `this.hooks.foo`!');
  };
}

app.use(hooks);
app.use(fooHook);

// Initialize your app!
app.init(function (err) {
  if (err) {
    throw err;
  }

  // When we emit 'app::foo', we want `app.hooks.foo` to get called.
  app.emit('app::foo');
});

What we want is a nice way to attach 'hooks' that get called when app emits app::* events. Remember, broadway.App inherits from EventEmitter2.

The output of this example will look like this:

% node index.js
You just emitted app::foo!
You just called `this.hooks.foo`!

As it turns out, writing this plugin is actually relatively straightforward:

// hooks.js

exports.name = 'event-hooks';

exports.attach = function (opts) {
  var self = this;

  this.hooks = {};
}

exports.init = function (done) {
  var self = this;

  self.on('app::*', function () {
    console.log('You just emitted `app::foo`!');

    var plugin = self.hooks[this.event.split(self.delimiter)[1]];

    if (plugin && typeof plugin === 'function') {
      plugin.call(self, arguments);
    };
  });

  done();
}

First, in the plugin's attach method, we create the app.hook namespace for attaching new hooks. Then, later, the init method adds the relevant event listeners.

Cool! But even cooler: You can reuse this "hook" plugin for building apps over and over again, and then write further plugins that depend on the hook plugin.

A Real-World Example of Application Extensibility

Just last week, we released a new flatiron plugin called flatiron-cli-config. Here's a version of the example on the project's front page, to illustrate its use with flatiron, whose app is an extended version of the base broadway container:

// index.js

var flatiron = require('flatiron'),
    app = flatiron.app;

app.name = 'app.js';

// The flatiron app has loaded a basic `config` plugin.
app.config.file({ file: 'test-config.json' });

// Use flatiron's `cli` plugin
app.use(flatiron.plugins.cli, {
  usage: 'A simple CLI app using flatiron-cli-config'
});

// Use `flatiron-cli-config`, which depends on then `config` and `cli` plugins.
app.use(require('flatiron-cli-config'));

// Note that 'app.start' automatically initializes your broadway app.
app.start();

Based on the previous broadway examples, this should be straightforward enough. But, what the heck did the flatiron-cli-config plugin do? Basically, it gives you a complete CLI-based configuration API---the same as the one used by jitsu. For example, this should work:

% node index.js config set foo bar

Just like we gave our stranger a head and a hat earlier, we just gave this application a complete CLI interface including configuration with an index of only 9 lines.

Conclusions

Here's the tl;dr on broadway:

• It's a plugin system.
• It lets plugins do almost anything to the host app.
• Broadway plugins are like your app's gear and power-ups.

Broadway is quite possibly my favorite component of all of flatiron. I hope you find the time to give it a spin! I think you'll find it worthwhile.

Flatiron is an adaptable decoupled framework for both node.js and the browser. An adaptable framework enables users to build multiple types of applications. In Flatiron we are focusing on:

• Web applications (http and websockets)
• Command line (CLI) applications
• Front-end applications

This article will focus on the tools at your disposal when using Flatiron to build CLI applications as well as how many existing Flatiron CLI applications are structured.

• Getting started
• To the code!
• Configuring your Command-line Application
• Composable Command-line applications
• Flatiron Command-line apps in the wild

Getting started

Writing a Command-line application with Flatiron starts out like writing every other flatiron application:

  $ flatiron create simple-app cli
  info:   Creating application simple-app
  info:   Using cli scaffold.
  info:   Creating directory config
  info:   Creating directory lib
  info:   Creating directory commands
  info:   Creating directory test
  info:   Writing package.json
  info:   Writing app.js
  info:   Writing lib/index.js
  info:   Application simple-app is now ready

Lets examine the directory structure created by running flatiron create simple-app cli:

  simple-app/
    app.js
    config/
    lib/
      index.js
      commands/
    package.json
    test/

With this simple scaffold the application has already been setup with:

• Commands lazy-loaded from the file or directory specified by options.source.
• app.cmd: Helper function for creating CLI routes.
• app.argv: Parsed CLI arguments using optimist.
• app.config: Hierarchical configuration sourced from: environment variables, CLI arguments and a JSON file in that order.
• app.log: Multi-transport logging using winston.
• app.alias: Helper function to alias shallow commands (e.g. app config list to app conf).
• app.inspect: Robust object inspection and formatting using cliff.
• app.prompt: CLI prompt using the Flatiron prompt module.

There are other options which make writing CLI applications quick and easy. Leave the tough stuff to us.

To the code!

Lets examine each line of the code generated by flatiron create simple-app cli:

var flatiron = require('flatiron'),
    path = require('path'),
    app = flatiron.app;

app.config.file({ file: path.join(__dirname, 'config', 'config.json') });

app.use(flatiron.plugins.cli, {
  source: path.join(__dirname, 'lib', 'commands'),
  usage: 'Empty Flatiron Application, please fill out commands'
});

app.start();

Setup up Configuration

  app.config.file({ file: path.join(__dirname, 'config', 'config.json') });

Add a store to back the configuration for this application. It will automatically load :pwd/config/config.json from disk.

Attach the CLI plugin

  app.use(flatiron.plugins.cli, {
    source: path.join(__dirname, 'lib', 'commands'),
    usage: 'Empty Flatiron Application, please fill out commands'
  });

Attach Command-line functionality to the application. See Configuring your Command-line Application for more detailed documentation.

Start the Application

  app.start();

Initializes the application and then dispatches to the director router using app.argv._. For example when running app config get foo, app.argv._ will be ['app', 'config', 'get', 'foo'].

Configuring your Command-line Application

There is a simple convention for passing options to both flatiron.plugins.cli and flatiron.plugins.http: the settings for the named property added to the application has the same name in the options passed to .use(). For example:

//
// Configuring app.argv
//

app.use(flatiron.plugins.cli, {
  //
  // Settings used by `require('optimist').options()
  //
  argv: {
    login: {
      alias: 'l',
      description: 'Login to use',
      string: true
    }
  }
});

//
// `app.argv.l` is now available
//
console.log(app.argv.l);

There are some important settings you need to be aware of:

• source {string}: File or directory on disk to lazy load commands from. In the case of a directory, the names of the files map to the first parameter in app.argv._.
• usage {string|Array}: Help text to display when using app help
• argv {Object}: Settings for CLI options parsing.
• prompt {Object}: Settings for Flatiron prompt module.
• version: {boolean}: Automatically sets up response for app --version and app -v
• async: {boolean}: If true configures the CLI router attached to the application to be asynchronous. See the director documentation for more information.

Composable Command-line applications

Flatiron enables developing adaptable applications through a lightweight dependency injection system called broadway. Writing broadway plugins is easy and can be used to easily compose applications out of reusable components. This was the motivation behind a new CLI-plugin for Flatiron released today: flatiron-cli-config. This plugin extends the application by adding handlers for app config *. It also adds help for all of these commands available through app help config *:

• app config list: Lists all key-value pairs known to the application.
• app config get <key>: Displays the value for to the user.
• app config set <key> <value>: Sets the to the specified value.
• app config delete <key>: Clears the value for the specified .

Using this plugin is simple:

  app.use(require('flatiron-cli-config'));

That's it. Your application is now fully setup to use edit configuration files from the command-line.

Flatiron Command-line applications in the wild

Since we released Flatiron there have been a number of interesting applications released and refactored:

• issues: Github Issues from the CLI
• box: Powerful key -> value storage for the CLI.
• todo: Todos in the CLI like what.
• npmcount: Silly program that counts number of npm packages from one or more users.
• jitsu: Flawless command line deployment of your Node.js apps to the cloud.

Well that about wraps it up. Can't wait to see what crazy things I'll be able to do from the CLI soon!

The main goal of SPDY is reducing the latency of web pages by multiplexing multiple http streams into one TCP connection and compressing http headers.

So instead of creating a TCP connection for each http request (like styles, scripts, images, or even ajax), SPDY-enabled browser will hold only one connection with the server.

source: http://www.igvita.com/2011/10/20/faster-web-vs-tcp-slow-start

Your server can tell the browser that it supports SPDY in two ways: setting the ‘Alternate-Protocol’ header, or by using a TLS handshake extension NPN (Next Protocol Negotiation). In the second case the server will send the supported protocols (i.e., SPDY/2 HTTP/1.1 HTTP/1.0) to browser before any data transfer will be performed, and the browser will choose the protocol that is preferable for the client.

You can learn more about SPDY by examing the protocol specification: http://www.chromium.org/spdy/spdy-protocol/spdy-protocol-draft2

Node.js and SPDY

The growing interest in both the SPDY protocol and Node.js created the need for a stable Node.js SPDY module when the first version was released last year. There were a number of issues with this first implementation: it was buggy, it did not support the default node.js builds, and it was incompatible with popular frameworks (only partial express support, and no socket.io support at all). Despite all of these issues it still attracted considerable some attention from the Node.js community.

Before continuing development on node-spdy it was clear that node.js core needed to be improved: the tls module was lacking NPN support, and there was no zlib functionality. While adding NPN callbacks to node was quite simple, I found that almost every OS has old SSL libraries without support for it. That’s why latest node version bundles openssl in it’s deps/ folder. Thanks to Isaac Z. Schlueter, zlib module was added in 0.6.0 version of node.js, and required only a little patching: it was missing stream dictionary support.

With all that working out of the box (only in 0.7.0-pre version of node now) I decided to start developing a completely new stable node-spdy version. You can take a look at what I’ve done so far at https://github.com/indutny/node-spdy . Note that to test it you’ll need to build v0.7.0-pre of node.js (currently at github’s master branch):

  git clone git://github.com/joyent/node.git
  cd node
  [sudo] ./configure
  [sudo] make
  [sudo] make install

I have one running SPDY server running at https://spdy-twitlog.indutny.com/. It’s demonstrating socket.io + express support of node-spdy module.

Features & Roadmap

Implemented:

• Node.js compatible interface (same as for require(‘http’).Server )
• Opening/closing streams
• Fallback to https
• Basic flow management (no WINDOW management so far, that means .pause() and .resume() will just buffer data internally instead of sending WINDOW_UPDATE to client)
• Protocol errors handling
• Stream scheduler (for priority management)
• Push streams

Roadmap:

• WINDOW_UPDATE and INITIAL_WINDOW_SIZE
• Fixing bugs

Blacksmith is a static site generator. It uses weld, jsdom and marked to turn json, markdown and 100% genuine HTML and CSS, on the filesystem, into awesome sites like the one you're looking at right now. Moreover, it includes a command line interface for quickly building, editing and testing blacksmith-built websites. Here's a short list of features:

• Easy to use
• Can be hosted anywhere, since it generates static HTML/CSS
• Write and Edit articles on the file system using github-flavored markdown
• Based on JSDOM and Weld
• Easily create custom themes using plain HTML and CSS ( no micro-templating ! )
• Default theme is a port of scribbish by Jeffrey Allan Hardy
• Ships with a robust node.js static server suitable for production

Blacksmith may remind you a bit of jekyll and octopress, and in fact it is influenced by these projects. However, whereas jekyll is "blog-aware" and uses liquid for templating, blacksmith uses the DOM for its templating needs and can easily generate not just blogs, but documentation sites and more. Plus, you don't need ruby or rake since it's all written in node.

Why Static?

We knew from the start that we wanted our blog platform to generate a static site. Why? Static sites are fast, foolproof, and get the job done. Why dynamically generate a page each time if you only have to do it once?

This might seem counterintuitive, considering all of the real-time, client-side code in Nodejitsu. However, while websockets and hash-based routing fit really well for many problems, others, like our blog, simply don't need it.

Why Weld?

We also knew from the start that we wanted to avoid microtemplates. Hij1nx puts it pretty well, but to reiterate: Templates are 100% HTML and css, and that is awesome.

Given this question, you may ask "why not plates?" Truth be told, we began this project before plates was ready for prime time. There is, however, a good reason not to switch: Having access to a full dom allows for doing such things as using jquery server-side, for the utmost in flexibility.

Try it out!

Just visit blacksmith's github repository, install it and have fun!

In software application development we usually store application state in volatile and non-volatile random access memory. We also use network devices to communicate state between our application and a central provider or server. We store state in things like databases, the file-system and cloud hosting providers. This approach is recommended for most applications, but when we start to think about the implications of building large, distributed, and decentralized applications, we begin to rethink about how these applications are going to store and maintain state.

In some way or another, almost anything can store state. Any device that can be written to, then read from later can be used to store state.

Think of it like this:

There are hundreds of thousands of endpoints on the web right now that you can post some sort of data to, and read it back later. Why not make a database out of that?

Basic strategies in building a decentralized and anonymous database

1. No central servers. A central server is a central point of failure.
2. No central authorities. A central authority over control of the database indicates another ( and much worse ) central point of failure.
3. Ability to encrypt and obscure data.
4. Ability to run in any combination of local / cloud / hosted. The database should be able to run locally, on hosted servers, and in the cloud.
5. Ability to easily share and replicate data. The data should be able to easily transfer and replicate between multiple systems and users.

The goals of building a decentralized and anonymous database are fairly straight forward. The data needs to have no central point of failure ( technological or political ). The data needs to be able to be secured. The data needs to be accessible and sharable.

Introducing hnet

hnet is a decentralized and anonymous database built in node.js. It works by spreading small amounts of information across several nodes using a variety of non-traditional storage engines.

http://github.com/hookio/hnet/

What is an hnet node?

An hnet node is anything that we can store a small amount of data which we can eventually read back later. The hnet client communicates with many hnet nodes in order to establish a dataset.

What is an hnet storage engine?

A storage engine can be considering a wrapper for any service on the web that can store data, and read it back later. Storage engines are small and pluggable, so it should be very easy to author new engines. Some examples of engines are: gist, pastebin, imgur, google groups, reddit, twitter, irc, etc... The only real requirement for a storage engine is that it can store state, the rest is up to your imagination.

Why hnet?

hnet was originally built as an easy way to provide distributed and semi-anonymous tables of server ips and ports for hook.io hooks. hnet helps provide hook.io a way of auto-discovery other hooks over a Wide Area Network. Note: hook.io already supports auto-discovery over the Local Area Network via mdns.

How does hnet work?

Many dumb hnet nodes, one smart client.

In order to query hnet, first we connect to a few "top-level" nodes

If you are not maintaining your own hnet, the hnet client will fall-back to a few semi-moderated top level nodes. If you don't feel comfortable having someone else moderate your data, it's trivial to setup your own piece of hnet.

You will notice that hnet uses Iriscouch for many of it's top-level nodes and that most nodes eventually link back to http://hnet.iriscouch.com/public. Iriscouch provides free hosted CouchDB as a service. It allows you to signup for a free CouchDB instance instantly and with minimal registration. IrisCouch was chosen for both it's high quality of service, and the fact the lead developer of the service Jason Smith is an active open-source and open-data advocate.

Note: Our node.js deployment tool jitsu also ships with a nifty command jitsu databases create couch, which will instantly give you a CouchDB instance through IrisCouch.

The hnet protocol is JSON

the hnet protocol supports arbitrary JSON data, and optional JSON-RPC commands

Here is an example of what a JSON fragment returned from an hnet node might look like:

example JSON returned from hnet node:

[
  { "foo": "bar", "tar": "val" },
  { "foo": "boo", "something": ["a","b","c"] },
  { "foo": "bar", "tar": "val" },
]

optional JSON-RPC commands can be embedded

hnet optionally parses these JSON-RPC commands.

[
  { "foo": "boo", "something": ["a","b","c"] },
  { "method": "link", params: [ 
      { "type": "couch", "uri": "http://hnet.iriscouch.com/public/0" }
    ] 
  },
  { "foo": "bar", "tar": "val" }
]

If we look at the array item from the previous JSON fragment, we can see:

{ "method": "link", params: [ { "type": "couch", "uri": "http://hnet.iriscouch.com/public/0"} ] },

The link method is particularly important.

It indicates that we should lazily link this document from a remote dataset into the current position in the item array.

This allows us to create large datasets from many small hnet nodes.

hnet client receives data from many nodes

After hnet is able to query a few top-level nodes, it begins to crawl several other linked hnet nodes and merges the data locally.

Saving data to hnet

hnet could be considered an append-only database. hnet is designed to create new nodes instead of attempting to edit existing nodes. Every time a new hnet node is created, it links back to at least two existing nodes.

Since every new node created links back to at least two other known hnet nodes, all newly created hnet nodes will eventually link back to the first nodes we started querying.

Establishing a TTL ( Time To Live ) for circular link resolution

By default, the hnet client will not attempt to resolve circular JSON-RPC links. You can enable circular linking by specifying the ttl parameter in the hnet constructor.

Ex:

var Hnet = require('../lib/hnet').Hnet;

var hnet = new Hnet({
  ttl: 5000
});

hnet.load();

The ttl parameter indicates that the hnet client will resolve any circular links it encounters, after a delay of 5,000 milliseconds. If a ttl is specified, the hnet client will continue to crawl hnet forever, or until the circular link becomes broken due to specific hnet nodes becoming unavailable.

A quick note on Steganography and Cryptography

hnet exposes simple Hnet.get and Hnet.save methods. It's fairly trivial to perform whichever cryptography you want on the data before you save it, and when you retrieve it. hnet also provides an optional interface for specifying un-encrypted metadata with encrypted data. This approach allows you to use hnet with most existing cryptography standards.

Steganography is the art and science of writing hidden messages in such a way that only the sender and intended recipient are aware of the existence of the message. It can be considered a form of security through obscurity. Many hnet storage engines use some form of steganography. The advantage of using steganography in conjunctions with cryptography, is that messages do not attract attention to themselves. Cryptography can protect the contents of a message, but steganography can obscure the fact that a message even exists.

At Nodejitsu, our node.js hosting platform deals with 1000s of live servers spanned across multiple data-centers. As scale increases, minute statistical probabilities become very real problems. The network is unreliable, disks become unwritable, streams break, unexpected input is unexpected, and entire data-centers can go down.

Think of it like this:

If you have 1,000 servers each individually rated at 99.9% uptime, on average, one of those machines is always failing.

So we research...

Fault tolerant system
Fallacies of Distributed Computing
Properties of distributed systems
High availability Percentage calculation

and we research more...

Shared nothing architecture
Byzantine fault tolerance
Paxos

Eventually, we hit the The Halting Problem:

Given a description of a computer program, decide whether the program finishes running or continues to run forever. In 1936, Alan Turing proved that a general algorithm to solve the halting problem for all possible program-input pairs cannot exist.

to summarize all of these links in the context of this blog post...

It's mathematically impossible your distributed application is not going to fuck up. Deal with it.

Basic strategies in building fault tolerant applications in node

1. The operating system's RAM, processes, and file descriptors are relatively cheap. Use them as needed.
2. Use JSON to pass data between nodes. ( see: nssocket or dnode ).
3. Embrace crash-only design. Nodes should be able to disconnect, restart and reconnect as fast as possible with minimal impact.
4. Never assume any one node process will never go down. Assume that any process can go down for any reason at any time.

The last point is particularly important. If your application is unable to recover from a "central" node going down, it should not be considered fault tolerant.

Hook.io

hook.io is a distributed input/output framework for node.js. It allows you to seamlessly link together several processes and start sending messages between them. hook.io also provides a rich network of stand-alone hook libraries for adding additional i/o sources. In a sense, hook.io can be described as a next generation enterprise service bus.

• Email List: http://groups.google.com/group/hookio
• Video Lessons: http://youtube.com/maraksquires ( mirror )
• Wiki Pages https://github.com/hookio/hook.io/wiki/_pages
• hook.io for dummies
• Distribute Node.js Apps with hook.io:
• #nodejitsu on irc.freenode.net

Basic Paxos

Paxos is a family of protocols for solving consensus in a network of unreliable processors. Consensus is the process of agreeing on one result among a group of participants. This problem becomes difficult when the participants or their communication medium may experience failures.

As of v0.8.6, hook.io now has the ability to do a very basic consensus among all hooks to ensure recovery if any hooks die.

In previous versions of hook.io, if the "master" or "central" server hook went down, all other hooks would stop communicating.

Now, hook.io is able able to recover from the "central" server hook going down.

Previously, when the hub went missing, none of the spokes knew where to connect anymore.

Building fault-tolerant node.js application with hook.io

hook.io can be used programmatically or as stand-alone binary. For simplicity, we will assume in this example that you are using hookio as a binary application, and with all it's default settings.

For more examples of using hook.io programmatically check out the examples or test folders.

To install hook.io with npm

 npm install hook.io -g

Now, we are going to start up three instances of hookio in our terminal.

Terminal 1

 hookio --autoheal

Terminal 2

 hookio --autoheal

Terminal 3

 hookio --autoheal

Using hook.io's default auto-discovery functionality the first terminal will take the role of server, and the second and third terminals will connect as clients to the first terminal.

Now, kill the first hook terminal with CTRL-C. The second hook will take over the role of the server. The third hook will now know to connect to the second hook instead of the first.

It might not seem very impressive, but it demonstrates a basic example of how to build fault-tolerant multi-process distributed applications in node.js

Applications built with hook.io now have no central point of failure.

But wait, there's more!!!

Mdns

Multicast DNS (mdns) is a way of using DNS programming interfaces, packet formats and operating semantics on a small network where no DNS server is running. The mDNS protocol is used by Apple's Bonjour and Linux Avahi service discovery systems. mdns is an easy way to help networked devices find each other without any prior configuration.

As of v0.8.6, hook.io now has alpha support for zero configuration networking via mdns. This means that if you hookio -m on two machines on the same Local Area Network...they will find each other and instantly start communicating without any configuration!

Computer A

 hookio -m

Computer B

 hookio -m

Now these two computers ( connected over a LAN, with no central DNS server ) will automatically discovery each other and begin to transmit messages.

Think of the possibilities!

Conclusion

With the new mdns and paxos features, hook.io is getting better everyday. The current goal is to stabilize this new functionality in the 0.8.x branch across all platforms.

The next steps will be p2p hook trackers and hosted hook services...

It helps you manage your node.js modules and dependencies.

Previously, we created a npm cheat sheet describing various NPM commands.

Now, here is a package.json cheat sheet for exploring various properties of npm's package.json format.

Enjoy the package.json cheatsheet!

These are Travis CI build status images. We started testing our code at Travis CI and we want to tell the whole world about it!

What is Travis CI?

Travis CI is a continuous integration system. Long story short: continuous integrations systems run your tests. How is it different from running them yourself?

• clean environment - your local development environment can be pretty weird and you know it! People who use your code don't necessarily have the same settings as you have
• easy creation of test environments
• it actually runs the tests - continuous integration system won't say 'this change is so small that it can't mess anything up'
• build logs are publicly visible - transparency is important, especially when it's about something other people rely on
• it's nearly realtime - we often push code and almost instantly get notified about test results

Why is it important for a CI to be realtime?

People make mistakes. Developers forget to npm update before npm test, keep their temporatory files and so on. You can probably write down a long list of silly mistakes you sometimes make. This can lead to pushing failing code.

Of course, Marak doesn't do that. At least nobody can confirm it.

Few days ago, thanks to Travis, we were able to find a regression in union. It turned out to be a dependency failure, which we were able to fix in approximately 5 minutes.

So, we delivered a product in 5 minutes. Sounds like a heaven for project managers.

OK, so how does this "realtime" thing look like? Many of us spend our time on #nodejitsu IRC channel at Freenode. Thus, a natural thing was to enable IRC notifications for builds.

I can definitely agree with Marak here. Point of this notifications is not only to tell you that you failed. We want people to know that their changes were good!

Versions!

We care about innovating. We care about our code being compatible with newer versions of software (node, npm and so on), but we have to care about people using older versions.

How can Travis help? Travis lets you to set up multiple test environments which target different runtime version. It's really easy to run your tests in node v0.6 and node v0.4 at the same time with just 3 lines in your .travis.yml file (detailed instructions for node.js are available here).

language: node_js
node_js:
  - 0.4
  - 0.6

Build you get looks like that.

Travis is easy!

Travis is a hosted service - you don't need a sysadmin to setup a Jenkins server for you. All you need is .travis.yml file and enabled GitHub post-receive hook (which Travis can do for you!).

Besides Node.js, travis-ci.org offers first class support for several other technologies:

• Clojure
• Erlang
• PHP
• Ruby

with more languages coming in the future. It only takes a couple of minutes to add your project to travis-ci.org, give it a try and if you find it useful, consider supporting the project.

Summary

We're having a great time with Travis. It helped us find a few regressions, bugs and compatibility issues. If you're starting or if you have an open source project, you should add it to Travis. Show people that you care about your test results!

Here is a quick cheatsheet of all my favorite npm commands:

• Installing npm
• Update npm
• Search for npm packages
• View details of a npm package
• Installing a npm package locally
• Installing a npm package into an application
• Understanding Global versus Local installs in npm
• Global Package Installation
• Uninstalling a package locally
• Uninstalling a package globally
• Installing a specific version of a package
• Cloning a module from Github
• Linking any npm package locally
• Linking local npm packages to multiple applications
• Unlinking a npm package from an application
• Unlinking a npm package from your system
• Create a new npm package
• Creating a new user account on npm
• Publishing a npm package
• Unpublishing a npm package
• Managing owners of packages

Installing npm

 curl http://npmjs.org/install.sh | sh

Update npm

There are several ways you can update npm. I prefer either:

 curl http://npmjs.org/install.sh | sh

      or

 npm install npm -g

Search for npm packages

 npm search hook.io

Protip: Try using http://search.npmjs.org

Double Protip: Know the name of the package you want before you start

View details of a npm package

 npm view hook.io

Installing a npm package locally

For the purpose of this demo, we will use http-server.

http-server is a package we've written which provides an easy to use wrapper around node's core http.Server class. This module makes for a good example, since it's API provides both a CLI binary and a requirable node.js module.

 npm install http-server

This performs a local install of http-server in our current working directory

You may also notice a new node_modules/ folder. You can ignore this for now.

Installing a npm package into an application

  mkdir mynewapp/
  cd mynewapp
  npm install http-server
  touch test.js

 node test.js

 mkdir anotherapp/
 cd anotherapp/
 touch test.js

 var HTTPServer = require('http-server');

 node test.js

node.js:134
        throw e; // process.nextTick error, or 'error' event on first tick
        ^
Error: Cannot find module 'http-server'
    at Function._resolveFilename (module.js:326:11)
    at Function._load (module.js:271:25)
    at require (module.js:355:19)
    at Object.<anonymous> (/Users/maraksquires/dev/nodeapps/anotherapp/test.js:1:80)
    at Module._compile (module.js:411:26)
    at Object..js (module.js:417:10)
    at Module.load (module.js:343:31)
    at Function._load (module.js:302:12)
    at Array.<anonymous> (module.js:430:10)
    at EventEmitter._tickCallback (node.js:126:26)

 cd anotherapp/
 npm install http-server

 npm install http-server -g

 npm install http-server -g

 http-server

cd mynewapp/
npm uninstall http-server

npm uninstall http-server -g

cd mynewapp/
npm install http-server@0.3.0

git clone git://github.com/nodeapps/http-server.git
cd http-server/
npm link

cd http-server/
npm link

mkdir newapp/
cd newapp/
npm link http-server

cd newapp/
npm unlink http-server

cd http-server/
npm unlink

mkdir mypackage/
cd mypackage/
npm init

npm adduser

cd mypackage/
npm publish

npm unpublish http-server

npm owner add marak http-server
npm owner rm marak http-server
npm owner ls http-server

So you're writing a module. Cool! Are you ready to document it? I'm here to help.

I'm no expert on readmes. I have, however, read a lot of readmes (both good and bad), written my fair share of readmes and documentation, and---most importantly---have formed strong opinions on what I should see in a readme.

This post focuses on the "first pass" for your readme. These PROTIPS are applicable, however, to readmes of all ages.

1. Write an Example, Right Now

The very first thing that should be written in a readme is a short example code snippet. This snippet should show what your library does, and as little else as possible. In the ideal case, people should be able to infer from your example how your library works.

For example, here's a stubbed-out readme for a small utility by SubStack:

mkdirp

Like mkdir -p, but in node.js!

Example

pow.js

var mkdirp = require('mkdirp');

mkdirp('/tmp/foo/bar/baz', 0755, function (err) {
    if (err) console.error(err)
    else console.log('pow!')
});

Output pow!

And now /tmp/foo/bar/baz exists, huzzah!

This probably isn't the greatest readme of all time. However, the example is a perfect case study. In four lines, a programmer more or less fluent in javascript can tell that mkdirp:

• Is a function.
• Takes 3 arguments: A file path, some octal permissions and an errorback.
• Probably makes directories.

Add in the mention of "like mkdir -p" and, without any other documentation, we can tell how it works.

Of course, mkdir -p is small enough that you can actually cover 100% of the API in one example. Not all projects are so lucky, but they can still strive to get a common use case across.

Here's another good example of an example, from node-tap. The example in this code saves what would otherwise be a terrible readme, to the point where node-tap has a solid contingent of loyal users despite its readme.

Why?

You want to get people to use your project, right? Supposing this is your goal, you have to understand that your readme is your library's best chance of "selling" your library, and you don't have that much time.

When a developer starts looking at libraries that might solve his or her problem, they're thinking two things:

1. Does this project solve my problem?
2. If so, how?

Chances are they're looking at your project's competition as well, and they're not very patient. You need to get the gist of this information across early enough that your audience pauses to read the rest of your readme.

A good example will tell a developer right away if the module does what they need, and any example will give the developer a taste of your API.

You also get the added benefit of having to write a simple case of your API, giving you the opportunity to "sanity check" your API. Does anything look bulky in your example? Could the API be made more obvious? Now you know.

2. Describe the install procedure.

A project that I think does a really good job of this is hook.io:

Getting Start / Demo

 npm install hook.io-helloworld -g

Now run:

 hookio-helloworld

Spawn up as many as you want. The first one becomes a server, the rest will become clients. Each helloworld hook emits a hello on an interval. Now watch the i/o party go!

Two. Lines. You now know how to get up and going with hook.io!

Node.js projects like hook.io are lucky in this regard, as npm makes most installations for node projects a one-liner. This is ideal, as ease of installation is another selling point for your project.

If you're not lucky enough to have an easy installation procedure, then this section becomes even more important. In the case of hook.io, I may have been able to put 2 and 2 together and went looking for it on npm. For a more complicated yet poorly-documented install procedure, developers will screw it up, at which point they will either complain to you or give up and move on.

Why?

Assuming your example made a sale, the second question a developer is going to ask is, "How do I install this thing?" Make it easy for them.

3. Stub out the API docs.

Now that you have a basic example and installation instructions, you can move on to documenting the basic API.

When I write API docs (and this is admittedly a weak point for me), I like to start with obvious entry points (your module only has one obvious entry point, right?) and work from there.

For example, here's a snippet from some API docs I wrote for union:

union.createServer(options)

The options object is required. Options include:

options.before

options.before is an array of middlewares, which are used to route and serve incoming requests. For instance, in the example, favicon is a middleware which handles requests for /favicon.ico.

Union's request handling is connect-compatible, meaning that all existing connect middlewares should work out-of-the-box with union.

In addition, the response object passed to middlewares listens for a "next" event, which is equivalent to calling next(). Flatiron middlewares are written in this manner, meaning they are not reverse-compatible with connect.

options.after

options.after is an array of stream filters, which are applied after the request handlers in options.before. Stream filters inherit from union.ResponseStream, which implements the Node.js core streams api with a bunch of other goodies.

The advantage to streaming middlewares is that they do not require buffering the entire stream in order to execute their function.

options.limit (optional)

This argument is passed to internal instantiations of union.BufferedStream.

In union, the obvious entry point is union.createServer. In this example, I tried to explain what you do with it, and what all the pieces mean.

Why?

Hopefully the idea of documenting your code isn't completely foreign. That said, you may be tempted to think that an example is "enough". I would suggest that this is true only in very rare cases. Even mkdirp could use a short paragraph explaining the arguments and behavior of the module. Inferrence is great, but shouldn't be completely relied upon. Be straightforward.

4. Tests

Your module may or may not have tests. If it does have tests (and tests are a wonderful idea!), you should describe how to run them. For example, here's a snippet that's very familiar to us at Nodejitsu:

Run Tests

Tests are written in vows and give complete coverage of all APIs and storage engines.

  $ npm test

This is pretty much ideal.

Why?

Like installation instructions, testing directions should be as straightforward as possible. If you're using node and npm, npm test is great because it distills testing into a one-liner.

(Choosing the right testing framework is another discussion entirely. I personally like node-tap).

5. Licensing and Contributors

Finally, tag on your license and contributors.

The content of this isn't actually too important in the context of a readme. For example, I usually just write the following:

License:

MIT/X11.

Why?

It's a good idea to tell other people how they can/may work on your code of course, but it's a secondary consideration. First, your commit logs will show who worked on the project, and arguably way better than your readme can. Second, the license is only important for people that want to hack on your code, and even then most people are willing to accept "It's MIT" when it comes to sending a pull request.

Licensing may become more important in the future, but in the short term you can afford to wait and scale it later.

6. But I had other stuff to say!

Write a blog post about it.

No, seriously.

Why?

A readme isn't always the best place to talk about your project. A readme's scope should really be pretty limited to:

• What is it?
• How do I use it?

Anything else---like, "Why I Wrote This Module" or "Addressing Criticisms Of My Module"---don't really fit. But, blog posts are a great way of getting this information out.

A good example of how a blog post can enhance a readme can be found on Nodejitsu's blog:

Scaling Isomorphic Javascript Code

In this blog post, Charlie Robbins discusses the motivations behind writing Flatiron, Nodejitsu's collective idea of what a web framework should look like.

For those interested in design patterns, it's a very good read. In this post, Charlie distills the meaning of MVC, how different existing frameworks implement it, and why (in his opinion) existing tools aren't enough. If that lights your fire, then I highly recommend this article.

That said, for the user that just wants to write a webapp and doesn't really care about which kind of MVC-ish paradigm they're using, such explanations in a readme will only get in the way.

TL;DR

If nothing else, write an example.

Successful open source libraries and initiatives require the same kind of attention to branding that many think is reserved for profit focused products. In other words, Open Source Doesn't Just Market Itself. Lets explore how discoverability, time to start (i.e. how long it takes to use what you've built), community, and some good old fashioned hustling can help build Open Source brands.

Overview

tl;dr? This is the place for you:

• "Don't Make Me Think": Understand your users' motivations.
• Divorce your Ego: Understand your motivations.
• Build a Community: Setting and managing user expectations.
• Open Source at Nodejitsu: A quick overview of Nodejitsu's Open Source projects and initiatives.
• Conclusion: Wrap it up.

"Don't Make Me Think"

A clear concise explanation of your Open Source offering is the best way to get users interested in what you're developing. Zach Holman from Github put it eloquently in his article Open Source Doesn't Just Market Itsef:

My favorite marketing of all time is documentation. We all loathe sleazy marketing, and by definition I like to think documentation can't be sleazy because it solves a real purpose: teaching everyone about the project.

Take a step back and think about discoverability and time to start from a user's point of view: they somehow stumbled upon your project. They probably don't know you or how awesome you are. They are looking for a solution to a problem that your project may or may not solve. The quicker you help them make this decision the better.

Documentation comes in all flavors, each of which serves a different purpose. Every type of documentation increases the momentum and reduces the perceived risk of your project so the more the better. Just like a startup, building momentum is a key metric to success: "Build enough momentum and they will come."

• README: This is Open Source MVP. Without a README (preferrably Markdown on Github), at least half the users that discover your project will simply move on. If you have a lot of Open Source projects be consistent. Use an outline; we do at Nodejitsu.
• Examples: If your library has several use cases, then it is absolutely critical to include example code and briefly discuss it in your README.
• Project Website: Even better. A custom project website allows you to create a true brand for your project; a logo, a catchy tag-line, whatever. Be sure to link back to Github or elsewhere.
• Blog Posts and Tutorials: This is the long tail of Open Source branding. Each blog post or descriptive tutorial you write will gather more momentum and lower the perceived risk of your Open Source offering.

Beyond documentation remember that this is Open Source: people will read your code. If you haven't finished part of your API or there is work left to be done be clear about it and expect feedback. Or, if you've decided on a particular way to write your code (such as promises or a particular coding style), realize many users will be immediately dismissive. This is particularly important in Javascript:

If your code uses a compiled fibers library, or something that transforms the code, then it's much harder for me to figure out what the heck is going on. The introspection is gone.

Divorce Your Ego

To me, the emotional investment in creating Open Source is best summed up by Isaac Schlueter:

It's like having children. Seems like a lot of pointless busy-work, and then you do it, probably without meaning or wanting to, and then suddenly fall so in love, you can't understand why everyone doesn't go as nuts over this little food-to-poop converter as you do. If you're not careful, you'll end up writing 3 or 4 more.

Divorcing yourself from your ego and these emotions is hard. You have probably put in countless hours of your own time just because you thought it was cool. But it is crucial to Open Source success: your users probably don't want your life story about why you created your library, or how it is going to solve a problem they don't have.

They also don't want to get an attitude when they open up an issue or reach out to you over Twitter, Email, or your Mailing list. A user may not completely understand the problem you're solving or is just plain misusing your API. An open-minded and understanding attitude is your greatest ally. This doesn't mean you have to divorce yourself from your opinions, just stay objective and never make it personal.

Build a Community

Beyond having a simple and concise explanation of what your Open Source offering is it is important to create a place where questions and FAQs can be easily found and responded to. Think of this as "beyond documentation."

When working with these community building resources it is crucial to realize your first interaction with a user may be your last. And, if you provide a great experience for your Open Source users they will become your community's greatest asset: responding to Github issues, helping new users, writing Open Source which expands on your own, authoring a critical mass of blog posts, and more. At Nodejitsu most of our new hires come out of this group of users who respond to common GitHub issues and mailing list threads.

So bury your ego and be as friendly as you can. That's not to say you have to do your users' work for them; just set and manage user expectations. We try to respond to every issue within 24 hours; ideally within 5 minutes.

• GitHub issues: You probably already use GitHub issues for your project. Are you checking them every day? Do you have a FAQ tag for commonly opened issues?
• Mailing list: Usually a Google group. This is a great place for feature discussions and commonly asked questions. Just look at the nodejs, expressjs, or hookio mailing lists.
• Twitter: Like it or not, Twitter is a very common place for developers to ask questions. Be prepared for some extra noise in your @ mentions. If you prefer not to use Twitter, be understanding and explain that to users. Also: retweets are one of your best marketing sources.
• Email: Personally I prefer not to answer questions about Open Source projects over email; I ask that a Github issue be opened. If you're feeling generous, however, it is a great way to build a rapport with one user.

When working with these community resources there are some basic expectations which my co-founder, Marak Squires, outlined in his JSConf.eu talk: How to be an Open Source Gangster. The tagline may sound silly, but as the Chief Evangelist at Nodejitsu Marak speaks from experience:

• Be available: If it takes a user a week to get a response to a pull-request, issue, or email, they probably aren't even a user anymore.
• Persistence, Tenacity, Follow-through: Stay focused. Every day is a new challenge. Like most developers Open Source is probably your side-project. Be explicit with yourself about how much time you are going to commit. If you go over that limit then find a way to share the workload with a contributor. Just giving up will undo any hard work you've put in.
• Ignoring Distractions: Don't get distracted by things that don't contribute to your core Open Source offering. "Let the things that don't matter truly slide."
• Recruitment: Realize that you are not a one man army and engage your contributors vigorously. In the long run, they are the greatest asset to your community.

Open Source at Nodejitsu

Last week at Nodejitsu we announced Flatiron. In addition to the libraries that we announced we took this opportunity to reorganize our personal Github repositories into organizations. We realized that as our Open Source offering grew organizing repositories across multiple Github organizations was the simplest path to high discoverability. This decision is not for everyone, and probably not the correct decision for most individual developers. Just like starting a company, these qualities help your Github organization tell a cohesive story. A good Open Source Github organization does a few things:

• Cohesive motivation: What problem is your Github organization solving? You don't get a README on your organization page so this should be inferred from the repositories or the homepage URL.
• Critical mass of repositories: Enough repositories to convince a passive user that the organization is legitimate. This can be extremely difficult if your company is new to Open Source; you may be better off contributing to an existing organization. Releasing several projects at once is a good way to avoid having less than a critical mass.
• Critical mass of members: Enough users to convince a passive user that the organization is legitimate. Githits.me is a great way to determine this; it ranks organizations by the sum of the Github followers of all users. This is not a perfect metric by any means, but it can help you determine this.

At Nodejitsu we have decided to delineate our core Open Source offering into several organizations each of which we feel tells a different story:

• Nodejitsu: Core Open Source libraries from our public Platform-as-a-Service, Nodejitsu.com including jitsu, haibu, and forever.
• Flatiron: Collection of decoupled Open Source libraries for building tools and web applications in Node.js.
• Hook.io: A full featured I/O framework for Node.js.
• Node Apps: A collection of sample applications designed to expedite time-to-start with Node.js on Nodejitsu.com.
• Node Training: Our training materials (when they're finished) along with tutorial-style projects which we use in our training sessions.

Moving and Renaming Personal Repositories

You've probably heard this saying about Computer Science before:

There are only two hard problems in Computer Science: cache invalidation and naming things. -- Phil Karlton

As the quantity of your Open Source libraries increases and they potentially move around to different organizations the first name you picked may no longer be appropriate. Such is the case for the routing component of Flatiron, which today we are renaming from sugarskull to director. We felt that the name director was more in-line with the Flatiron brand.

If you do decide to move personal Open Source repositories to an organization Github will not perform an 301 redirects so the best process is below. It is not ideal by any means, but it is the best Github can do today. If you are concerned about SEO think about this this way: you shouldn't be concerned with the 100 followers you have today; be concerned with the 1,000 followers you want in a year.

1. Move the repository to your organization from your personal Github
2. Immediately fork the moved repository back to your personal Github and add a redirection notice.

Conclusion

Hopefully not all of this is new to you; most of it is a distillation of the common wisdom that most developers follow today. A lot of this can be thought of as Open Source in the age of Github.

Prior to GitHub, word about open source projects travelled primarily through word-of-mouth and messenger bicyclists. The largest open source project had three users and two developers. I think it involved a lot of twine and glitter.

• Make your users first time as easy as possible: Minimize perceived risk to maximize momentum.
• Be an Open Source Gangster: Every day I'm Open Sourcin'.
• Engage your community and recruit actively: Github, Twitter, Mailing list(s); whatever works.
• Create a Github organization for large projects: Remember to tell a cohesive story.

This is the motivation behind flatiron. At it's core flatiron is two things:

An initiative to build a collection of decoupled tools with the same standard of quality and performance that you would expect from anything built by Nodejitsu.

A full-stack web application development framework which packages these tools together to make isomorphic and stream-based application development easier.

This article will introduce you to the components that make-up flatiron as well as how the core flatiron project.

A Sum of its Parts

At Nodejitsu, we have and continue to believe in a simple mantra: build the best tools and the best systems will follow. Tools and their distribution among members of an organization evolve organically overtime. To put it another way: if you use Node.js you are probably already using a component of flatiron. We have simply moved these tools to the Flatiron Github Organization

• api-easy: Fluent (i.e. chainable) syntax for generating vows tests against RESTful APIs.
• complete: Custom command line tab completion for nodejs CLI apps.
• nconf: Hierarchical node.js configuration with files, environment variables, command-line arguments, and atomic object merging.
• neuron: The simplest possible event driven job manager, FIFO queue, and 'task based cache' in node.js.
• node-prompt: A beautiful command-line prompt for node.js.
• sugarskull: A depth-first search based routing library for the browser and node.js.
• winston: A multi-transport async logging library for Node.js.

In addition to migrating existing libraries we have been working on a number of new libraries to round out these tools.

• broadway: Lightweight feature reflection and application composition for node.js and the browser.
• flatiron: An unobtrusive full-stack web framework for node.js and the browser.
• plates: Unobtrusive, thin and fast anti-templating
• revalidator: A cross-browser / node.js validator used by resourceful and flatiron.
• resourceful: A resource-oriented ODM with a high-level API for routing and validation.
• union: A hybrid buffered / streaming middleware kernel backwards compatible with connect.
• utile: A drop-in replacement for util with some additional advantageous functions.

A lot of effort has been made to ensure that each component stands on it's own and that each Github repository clearly explains the purpose, use case, and stability of the library. There are a few particularly interesting highlights among these libraries that we'll explore below.

A quick tour of Flatiron

Application composition

Thanks to Javascript you never have to worry about types again (mostly), but that doesn't mean that we can't learn something from design patterns more prevalent is statically typed languages. Namely: inversion of control and dependency injection. Don't be scared! Promise you won't have to register anything, ever.

Broadway exposes a simple "plugin" API which allows the application developer to extend the top-level application object easily. This model translates well between the browser and the server because developers can write different plugins to suite the needs of the different environments. Broadway includes several plugins by default which can be overridden if you don't agree with the particular module or library choice we've made:

Figure 1: app.use() in Broadway

Doesn't remind you of dependency injection? It shouldn't! Because of the loosely typed nature of Javascript it isn't necessary to bring along all of the heavyweight type registration APIs that are so common in most dependency injection frameworks. Instead, plugins must be responsible and properly implement three methods

• .attach(options): This synchronous method is called immediately on app.use(somePlugin).
• .init(done): This asynchronous method is called once when the application initializes and before it starts.
• .detach(): This synchronous method is called on app.unuse(somePlugin).

Figure 2: Plugin Lifecycle in Broadway

All of these functions are evaluated in the scope of the app (i.e. this inside of a plugin is the app itself). Lets take a look at a quick sample from broadway itself:

exports.attach = function (options) {
  //
  // Here setting `this.config` will expose the `.config` 
  // property on the application itself
  //
  options = options  || {};
  this.config = new nconf.Provider(options);
};

exports.init = function (done) {
  //
  // Later on we can initialize the configuration 
  // we attached earlier.
  //
  this.config.load(function (err) {
    return err ? done(err) : done();
  });
};

Ok. So what? Application extensibility allows us to reuse more components across different kinds of applications while writing less code. In almost every express application, you will see a server.js file which is essentially wiring up the same libraries to the same methods. If you write a couple of small applications it might not seem like a problem, but when building a product like Nodejitsu it simply isn't DRY enough.

Streaming or not streaming? No problem!

One of the current problems in node.js core is that there is no good way to buffer a stream when not closing the underlying file descriptor. Don't worry though; the geniuses behind node.js are on it and already have a solution on the drawing board. In the meantime though, something needs to fill the gap and that is union and sugarskull.

The problem is that 90% of the time you actually want to buffer a small amount of request data (< 10k) into memory then parse it and attach it to the http.ServerRequest object. Think: a POST with some JSON attached to it. Occasionally though, you may want to take advantage of a stream. Think: a large file upload to Amazon S3. In this case, buffering is cancer and .pipe() is king.

The solution is two-fold:

• Pipe the concrete http.ServerRequest to a BufferedStream implementation by default.
• Attach metadata to routes indicating if the route requires a stream or not. If a stream is not required, attempt to parse the entire body before continuing to the Router.

First, just a little about me... in July I joined Nodejitsu as COO. What's that mean? For the purposes of this discussion, not a whole lot. What's more important is that I'm an application developer and I'm new to Node. In this post, I'll discuss a lot of the things that attracted me to Node.

What is Node?

Node (aka Node.js, NodeJS) is Javascript. It's the same thing that's been running in your browser for years. Now it's better, faster, and stronger. Oh, and now it's everywhere. Node is what allows you to bring Javascript, a language traditionally run in the browser, to the server. And why does Node matter? I'm glad you asked that.

Javascript isn't going anywhere

There's no getting around having Javascript in your app. Javascript is THE language for making dynamic web interfaces bar none. If you're thinking, "Hey, but isn't that JQuery?" You're right. But JQuery is Javascript. Promise. And if you're thinking, "Come on, but what about Flash?" Well... what about it?

Javascript is going to be in your application somewhere. Node allows you to standardize on that from the top to the bottom. Some applications have Javascript on the front-end on top of a Python web application that speaks to APIs served by something more performant like Scala or Erlang. That's a lot of different competencies to hire for and to maintain and when something breaks, not everyone is going to be conversant with everything. Deferring specialization is a huge benefit of Node.

Wow, is it fast!

Node is an order of magnitude faster than Ruby and Python. This translates roughly to say that it's very easily the most responsive under load of any of the most popular high level languages (Javascript, Ruby, Python). Node and Javascript benefit from all the work that Google, Apple and Microsoft have poured into making their browsers more performant as well as more efficient to run on less resources (such as a netbook or iPad). Lucky us.

Realtime

Node uses an asynchronous event-driven model. This means that it can make more efficient use of resources to do more things at one time and is an excellent fit for the architectures of many Internet applications. This model also makes these applications easier, and more intuitive for engineers to create.

Facilitation of engineer AND code versatility

Javascript everywhere means that while there are still going to be developers that are more familiar with particular front or back end APIs or environments, it's a lot easier for the team to pitch in where help is needed. In a similar fashion, Javascript everywhere facilitates the reuse of code all through your stack which is a good thing as it cuts down on redundancy and helps maintainability of your application.

Yeah, we think it's pretty cool too

This is of course, just a taste. Here are some other excellent resources to get you started.

• Convincing the Boss
• Understanding Node.js
• "Data-Intensive Real-Time" Applications

Also, check us out on the Twitter (@Nodejitsu), read some of the other articles on our blog, or maybe attend some of our future training sessions. We also write a tremendous amount of open-source code that you can utilize in your applications.

This approach has been acceptable for other languages such as Java, Ruby, and Python it is simply not good enough for Node.js for one reason: Javascript is now an isomorphic language. By isomorphic we mean that any given line of code (with notable exceptions) can execute both on the client and the server. On the surface this seemingly innocuous property creates a number of challenges that are not solved by current MVC-based patterns. This article will explore some of these existing patterns, how both their implementation and concerns vary across languages and environments, and how they are not good enough for a truly isomorphic Javascript codebase. In conclusion, we will explore a new pattern: Resource-View-Presenter.

At a glance

Design patterns are the bread and butter of application development. They encapsulate and outline the concerns of the application and the environment of the in which it exists. Between the browser and the server these concerns can vary widely:

• Is the view ephemeral (e.g. on the server) or long-lived (e.g. in the browser)?
• Is the view reusable across different use-cases or scenarios?
• Should the view be annotated with application-specific tags or markup?
• Where should the bulk of the business logic reside? (in the model? in the controller?)
• How is the application state persisted or accessed?

Lets explore some of the existing patterns and how they answer these questions:

• Model-View-Controller
• Model2
• Model-View Presenter and Model-View-ViewModel
• Modern Javascript Implementations
• Real-time Implications
• tl;dr? Introducing Resource-View-Presenter
• Conclusion

MVC

Figure 1: Model-View-Controller

The traditional Model-View-Controller pattern assumes a long-lived view and a swappable Controller. For example, in a given view you may have different Controller logic depending on who is logged in. At a high-level no decision is made about how the view is rendered (i.e. what templating engine is used).

Given the assumption of a long-lived view and that user interaction is by definition coming into the view, traditional MVC is a very useful pattern for front-end development. As we will explore later, a slightly modified version of this pattern is in fact what Backbone.js uses.

Model2

Figure 2: Model2 Model-View-Controller

Don't be scared if you've never heard of Model2 before; it is a design pattern that dates back to 1999 when it was coined in an article by Govind Seshadri: Understanding JavaServer Pages Model 2 architecture. It can be argued that Model2 does not necessitate a fully-realized MVC pattern, but most modern implementations (such as Ruby on Rails) do formalize it in that way.

Figure 3: Rails Model2 Model-View-Controller

The common wisdom is that in Model2-like frameworks such as Ruby on Rails one should have "fat models and thin controllers". This is not the case for every application, but in practice it is generally what this author has seen. In the traditional MVC Controllers tend to be heavier (i.e. more business logic) due to their need to listen and react to input from the view so this decision seems to add up.

Given the stateless nature of HTTP the Model2 View is truly ephemeral: no state maintained in the view itself between requests. In most server-side frameworks any application state is stored via Session Cookies. This makes the decision of a one-way communication between the Controller and View very logical, but also unsuitable for any front-end development.

MVP and MVVM

Both the Model-View-Presenter and Model-View-ViewModel patterns are similar to the traditional MVC pattern with a few key differences:

• View does not have a direct reference to the Model
• Presenter (or ViewModel) has a reference to the view and updates it based on changes to the model

The MVP pattern is discussed at length by Martin Fowler (here, and here) and is generally discussed in the context of two implementations:

• Passive View: Design the view to be as naive as possible with the all but the absolute necessary presentation and business logic contained in the Presenter.
• Supervising Controller: Used in systems in which the declarative view can be expanded to encompass simple logic. In this incantation the Presenter should only take over when the declarative logic in the view cannot meet the system requirements.

Figure 4: Model-View-Presenter

Figure 5: Model-View-ViewModel

MVP and MVVM are almost indistinguishable with one key exception: MVVM assumes that changes in the ViewModel will be reflected in the view by a robust data-binding engine. Niraj Bhatt outlined the difference eloquently in his MVC vs. MVP vs. MVVM article: "For instance if View had a property IsChecked and Presenter was setting it in classic MVP, in MVVM ViewModel will have that IsChecked Property which View will sync up with."

The positive end of the MVP and MVVM is that the Presenter (or ViewModel) is easier to unit-test because the state of the View is by definition contained in methods invoked by (MVP) or properties set on (MVVM) by the Presenter or ViewModel respectively.

For front-end development either of these patterns are perfectly acceptable choices. A routing layer can pass control to the appropriate Presenter (or ViewModel) which in-turn can update and respond to a persistent view in the browser. With some massaging either could also be re-worked for use on the server for one reason: there is no connection between the model and the view. This allows for an ephemeral view which is rendered by a given Presenter (or ViewModel). As we will see later on, this modified pattern can be truly isomorphic.

Modern Javascript Implementations

The design patterns presented above have many modern implementations today:

• Backbone.js
• Batman.js
• Angularjs
• JavascriptMVC
• Sammy

These frameworks are generally used for creating Single-Page Applications (although more traditional AJAX is also possible). User Interaction within a single-page application comes in two distinct flavors:

• OnHashChanged or pushState events: Occur when the URL in the browser changes. For example: navigating to http://myapp.local/#/some-page
• DOM events: Occur when a user makes a specific interaction with the current DOM. For example: clicking on an anchor tag.

Lets consider some of the patterns and architectures used. If you're interested in further reading check out Peter Michaux's article on MVC Architecture for Javascript Applications.

Backbone

Backbone.js is one of the most popular client-side development frameworks available today. At it's core it is an implementation of the traditional Model-View-Controller pattern. When we examine it in more detail, however, we can see that there are some deviations from the traditional MVC pattern explored earlier.

Figure 6: Backbone Model-View-Controller

In the diagram above we have separated the control-flow represented by OnHashChanged and DOM events to illustrate the separate entry points offered by Backbone. By illustrating this nuance it is clear that Backbone has one important difference from traditional MVC: the View manipulates the Model. When we examine the Backbone Todo Sample, we can see this is clearly the accepted common wisdom:

window.AppView = Backbone.View.extend({
  // ....

  //
  // When a TodoView is instantiated it is passed an
  // instance of the Todo model.
  //
  addOne: function(todo) {
    var view = new TodoView({model: todo});
    this.$("#todo-list").append(view.render().el);
  }
  
  // ....
});

window.TodoView = Backbone.View.extend({
  // ....
  
  //
  // Here the TodoView updates the state of the Todo model.
  // This breaks from the traditional MVC in which the view
  // only listens for changes on the Model.
  //
  toggleDone: function() {
    this.model.toggle();
  }
  
  // ....
});

The decision to break from the traditional MVC pattern gives large Backbone applications all similar feel: thin controllers and models combined with heavy views. These business logic-heavy views are essentially Presenters when looked at objectively. In a large Backbone codebase you should expect a large number of views composited together via a DOM framework like jQuery or zepto.

There is nothing wrong with breaking from the traditional MVC pattern; in the context of front-end development having a reference to the Model from the View removes a lot of bookkeeping logic from the application. It does, however, make it a pattern that is not isomorphic.

Batman

Batman.js is a new Javascript framework that was discussed at JSConf 2011. Although the entities within Batman are Model, View, and Controller. The presence of a strong data binding engine and pure HTML views suggests that it is actually an implementation of the Model-View-ViewModel.

Figure 7: Batman Model-View-ViewModel

Having not worked with Batman much it is difficult to say with confidence what a large Batman codebase looks like. That said: the emphasis on the binding engine and thin views suggest that business logic will end up spread between the Controllers and Models within a given application.

As with Backbone, Batman breaks from the traditional Model-View-ViewModel pattern in that the Model communicates directly with the View and the ViewModel (i.e. Controller) does not manipulate the View directly. In addition, because of the references between the Model and the View it is not easily reusable as a server-side pattern. It is, however, more easily massaged into a server-side pattern if a Composite or Adapter pattern is implemented in the Model layer to render a static view and respond to real-time requests.

Real-time implications

Among all of the developer buzz one topic continues to be at the forefront: realtime web applications. Nodejitsu is an official sponsor of the upcoming Keeping it Realtime conference in Portland; check it out! So how do some of the patterns we've examined stack up for realtime support (such as WebSockets)?

• Model-View-Controller (Yes): Model(s) can listen for real-time events and update the view appropriately.
• Model2 (No): The Ephemeral View concept is baked into the Model2 pattern. That is: Controller(s) do not listen for events from the Model.
• Model-View-Presenter (Yes): Model(s) can listen for real-time events, which will propagate to the Presenter(s) who will update the view appropriately.
• Model-View-ViewModel (Yes): Model(s) can listen for real-time events, which will propagate to the ViewModel(s) who will update the view appropriately.

The acceptability of MVC, MVP, and MVVM patterns make Backbone.js or Batman.js workable realtime frameworks for front-end developments. On the server-side this fact no longer holds true because as we have shown: traditional MVC, MVP, and MVVM patterns will not work for static views due to the tight coupling between the View and the Model.

Enter the Resource-View-Presenter pattern.

Introducing Resource-View-Presenter

As we have shown: the MVC, MVP, and MVVM patterns will not work on both the client and the server. At the crux of Resource-View-Presenter is the realization that no pattern will work flawlessly both on the client and the server without some modification. As was suggested when examining the MVP and MVVM patterns, it is possible to make these patterns truly isomorphic due to their decoupled Model and View layers.

The main decisions which Resource-View-Presenter are:

• Decouple the Model and View
• Recognize and plan for the differences between the Client and Server.
• Expect a thin View, and a heavy Presenter and Resource.
• Prefer business logic in the Resource over the Presenter.
• Allow for both ephemeral (i.e. static server-side views) and persistent Views (i.e. the DOM).
• Prefer a Presenter over a ViewModel to preserve the purity of the Markup (i.e. HTML).
• Assume a persistent Presenter and Model.

Although these decision may seem as though they are random, each has a specific purpose:

• By decoupling the Model and the View we allow for both ephemeral and persistent Views
• A thin View is consistent with modern, logic-less templating engines such as weld and mustache.
• Having a Presenter instead of a ViewModel is consistent with designer-friendly templating such as weld.
• Assuming a persistent Presenter and Model both on the Client and Server enables encapsulation of real-time functionality in Presenters on both sides.

Lets dig deeper. On the Client-side Resource-View-Presenter resembles a traditional MVP pattern. The choice to rename the Model to Resource is heavily influenced by our assumption to prefer business logic in the Resource over the Presenter. This makes the Resource in RVP more similar to the heavy Models found in Model2, not traditional MVP the nomenclature denotes that. When implementing RVP the litmus test for what logic belongs in the Presenter is two fold: any Presentation-specific logic which is too heavy for a "thin" View or any business logic which utilizes global application state.

As with Backbone and Batman, Client-side RVP implementations should support both OnHashChange / pushState routing and DOM events.

Figure 8: Client-side Resource-View-Presenter

On the Server-side Resource-View-Presenter is almost identical with one notable exception: the View is ephemeral and does not pass calls or have a reference to the Presenter. In fact, when using RVP for JSON-based web-services the view is practically non-existent; it is just a call to JSON.stringify().

Figure 9: Server-side Resource-View-Presenter

At first glance, Server-side RVP may seem similar to Model2 MVC, but the persistent-nature of both the Presenter and Model to support real-time events makes this similarity is only skin deep. RVP does this by listening for events and changes on the model which may be backed by a real-time data source such as Redis PubSub or CouchDB changes.

The real-time support is of particular note because it enables the application developer to focus on the business logic, not the underlying network transport. This may seem trivial, but examine the pattern (or lack there-of) offered by Express and Socket.io (the most popular node.js Framework and Realtime IO libraries respectively) and it becomes clear that it is not.

Figure 10: Express and Socket.io

This is not meant to be an attack on Express or Socket.io; they both are explicit in what they offer and they both do it exceptionally well. A higher-level design pattern is simply not a concern.

Conclusion

Writing large applications is hard. Encapsulating and reusing components of those applications across the Client and the Server is even harder. After considering this analysis it is hopefully clear how utilizing the Resource-View-Presenter pattern within your own applications will help to make this easier.

Don't have the time to implement it yourself? No problem! This article is the result of a lot of careful thought and hard work born from the massive open-source undertaking Nodejitsu has been apart of from the beginning. Our philosophy remains the same: build the best tools and the best systems will follow.

And follow they will. If you haven't guessed already, we have an epic open-source release coming up that will implement the Resource-View-Presenter pattern. So if you have any feedback, let us know!. And stay close, I will be announcing the release at SWDC 2011 in Stockholm, Sweden on November 9th..

A few weeks ago we released http://docs.nodejitsu.com, just in time for Node Knockout! docs.nodejitsu.com is a collection of community written tutorials and how-to articles for node.js. There are over fifty articles so far, from the basics all the way up to more difficult topics. We like to think of it as our unofficial, non-competing entry into NKO. We also, of course, hoped that the actual competitors would find it useful.

Like our software, all of the articles are hosted on github. Contributors can fork the project on github, edit the files in markdown (or add new ones), and submit a pull request. We are actively working on adding more features to the site, such as a tighter integration to http://search.npmjs.org

Aside from Nodejitsu's West Coast Team, we've already had a number of other contributors including: Marco Rogers., Dean Landolt, Adam Malcontenti-Wilson, and Pavan Sunkara. Thank you all! We still need more articles, so if you are interested in becoming a community contributor (and possibly getting compensated for your work), please contact us at docs@nodejitsu.com.

We hope that the Node.js community finds http://docs.nodejitsu.com a handy resource and, through community contributions, it becomes the de facto source for practical Node.js documentation.

The problem here is that the extremely popular middleware style of web application development is often incompatible. If you use a body decoder middleware, for example, every request will need to be buffered in full before the body can be properly decoded. This increases memory usage and reduces performance, compared to simply relaying un-altered requests and responses to other destinations - it all has to be waited for, read into memory, possibly altered, then sent again, adding to both the time and resource cost of each request.

More importantly, though, if any subsequent middleware in your middleware chain tries to listen for 'data' or 'end' events, or otherwise treat the current HTTP connection as a stream, it just won't work, as a buffered request is no longer a stream - it is now just a data object.

Despite these concerns, 'middlewares' are an easy concept to grasp, easy to use, and are likely to remain popular. There are some gains to be had, as well - middlewares are a simple way to add new features to your application stack. If you find yourself wishing for a certain feature, the ability to use or write a middleware to add that feature can save everyone a lot of time, compared to the amount of time a pull request can take to be reviewed and merged. It also allows for conflicting features to exist more easily - finding out that two middlewares won't work well together is a much smaller problem than dealing with successive breaking API changes in a library.

Due to these facts, and the number of requests we've received from users who want to proxy to that style of application, we have added middleware support to Node HTTP Proxy.

Here's a basic example of how to use a middleware - specifically, the connect-gzip middleware. This would gzip-compress every file you serve. It can be a useful trick, at times, when applied to specific file types - replace the /.*/ with /png/, for example.

var http = require('http'),
    httpProxy = require('http-proxy'),
    gzip = require('connect-gzip');

httpProxy.createServer(
  gzip.gzip({ matchType: /.*/ }),
  9000, 'localhost'
).listen(8000);

http.createServer(function (req, res) {
  res.writeHead(200, { 'Content-Type': 'text/plain' });
  res.write('request successfully proxied to: ' + req.url + '\n' + JSON.stringify(req.headers, true, 2));
  res.end();
}).listen(9000);

The basic idea is that when you call httpProxy.createServer, you can now pass in any middlewares as arguments, in the order in which you'd like them to be used. If one of your middlewares doesn't set your proxy destination for you, pass in the port and host you want to proxy to at the end of the list, as shown above.

Here is another simple middleware usage example, this one using an example middleware by Nodejitsu's Mad-Scientist-in-Residence, Dominic Tarr, called proxy-by-url. Available on npm, this middleware provides a sugar syntax for defining both forward and reverse proxy destinations in the same block, and routing to them based on URL matching.

httpProxy.createServer(
  require('proxy-by-url')({
  '/github': { port: 80, host: 'github.com' },
  '/nodejitsu': { port: 80, host: 'nodejitsu.com' },
  '/localstuff': { port: 8080, host: 'localhost' } 
  })
).listen(8000);

Now let's take a look at how to make a middleware - even if you don't plan to make one of your own, the structure is very straightforward, and understanding it will help you better understand how middlewares work. This middleware is for very simple logging - it will write the headers of each request to a logfile.

var fs = require('fs'),
    http = require('http'),
    httpProxy = require('../lib/node-http-proxy');


module.exports = function (logging) {
  // Code here is run when the middleware is initially loaded.
  var logFile = fs.createWriteStream('./requests.log');

  return function (request, response, next) {
    // Code here is run on each request.
    if (logging) {
      logFile.write(JSON.stringify(request.headers, true, 2));
    }
    next();
  }
}

This middleware can now be used like the others:

httpProxy.createServer(
  require('./example-middleware')(true),
  8000, 'localhost'
).listen(9000)

In other Node HTTP Proxy news, we at Nodejitsu have also been fortunate enough, in the last several weeks, to receive a number of high-quality patches and bug fixes from isaacs. As a result, the resilience and stability of Node HTTP Proxy have been significantly enhanced.

So the next time you run into isaacs, say thanks! (With beer.)

You can also find a mirror of these videos @ https://github.com/hookio/tutorials

Enjoy!

Exploring the Core API

1. #12 A quick lesson in fault tolerance Part 1
2. #13 A quick lesson in fault tolerance Part 2
3. #14 Spawning child hooks Part 1
4. #15 Spawning child hooks Part 2
5. #16 Full-featured child hook spawning with Haibu and Haibu-Carapace
6. #17 Using Hook.io to Install NPM Packages Anywhere

Note: I apologize for small text size, I will try to improve this on the next batch of screencasts

hook.io is a radical departure from how web applications are currently being built. hook.io itself is very easy to use, but the concepts behind it can be hard to understand from only reading the documentation and unit tests.

For this reason, I have created eleven short video tutorials for explaining the basic usage and functionality of hook.io

You can also find a mirror of these videos @ https://github.com/hookio/tutorials

UPDATE: 6 more videos on Spawning hook.io Hooks

DOUBLE UPDATE: Due to popular demand, I have converted and uploaded all videos to youtube. You can subscribe here: http://www.youtube.com/maraksquires

Enjoy!

1. The Basics Part 1
2. The Basics Part 2
3. The Basics Part 3
4. IRC Hook Configuration Part 1
5. IRC Hook Configuration Part 2
6. 3 Minute IRC Bot
7. Hook.io Libraries
8. HTTP Webhook Server
9. Socket.io Bridge to Browser
10. Emit events from CouchDB _changes feed
11. Extending an existing application

Note: I apologize for the low quality audio and strange video overlays. I've fixed this problem in the newer batches of videos

First, install our CLI tool from NPM

npm install jitsu -g

Now lets fire up our Node Knockout wizard!

cd /path/to/mynewapp
jitsu knockout

That's it!

From here you will be prompted for your Knockout Secret Code, and then jitsu will generate and deploy a simple helloworld application for you that will register with Node Knockout's website.

Additional Deploys

Need to update your app? Add new package.json dependencies? Websockets? Custom C++ addons? No problem!

For new deploys just type:

jitsu deploy

It's super simple!

Questions? Problems?

We've got the full Nodejitsu team on duty ready and waiting to help you. The best place to get support is in #nodejitsu on irc.freenode.net. You can also email support@nodejitsu.com

Warning: These numbers should not be used for anything meaningful, and are possibly wrong. The data is pulled from http://search.npmjs.org and uses SubStack's nifty tool: npmtop

Hourly tallies can be found here: http://npmtop.nodejitsu.com

  Generated: Fri Aug 19 2011 08:41:16 GMT+0000 (UTC) 

  rank   percent   packages   author
  ----   -------   --------   ------
     1    1.85 %     62.00    James Halliday
     1    1.85 %     62.00    TJ Holowaychuk
     3    1.59 %     53.00    AJ ONeal
     4    1.05 %     35.00    Dominic Tarr
     5    0.90 %     30.00    Isaac Z. Schlueter
     6    0.87 %     29.00    Marak Squires
     6    0.87 %     29.00    Nickolay Platonov
     8    0.72 %     24.00    Charlie Robbins
     8    0.72 %     24.00    Felix Geisendorfer
    10    0.66 %     22.00    George Stagas

Tied for 1st - James Halliday and TJ Holowaychuk

James Halliday, aka SubStack, author of libraries such as: dnode, browserify, burrito, etc. Co-founder of Browserling. All around Robot.

TJ Holowaychuk, aka visionmedia, author of Express, Connect, Expresso, Jade, etc. Engineer at Learnboost, an all-around nice guy, and he has a badass ferret named Tobi.

I'll keep the profiles short for these two, they are open-source champions.

3rd - AJ Oneal

AJ is a Software Consultant living in Provo Utah. He's done a lot of work re-packaging and re-bundling client-side modules ( such as jQuery ) to work server-side and to work with the Ender.js framework. He also has some pretty neat looking libraries like Futures. When I asked him for a bios he told me he was single, looking for ladies, and his favorite animal is a Trogdor. I think AJ might be my new internet best friend.

4th - Dominic Tarr

Resident Mad Scientist at Nodejitsu. Dominic enjoys writing test frameworks for testing test frameworks, this requires a lot of support libraries. There is only one project of his that needs to be pointed out, and it's testbed. In usual fashion, there is no documentation and the site is currently offline. I think I might have to schedule an open-source intervention for Dominic to help get his project back online. Please send words of encouragement and pull requests to madscience@nodejitsu.com

5th - Isaac Z. Schlueter

Friend, author of NPM, works for Joyent, node.js core contributor, dislikes semi-colons. Including several support libraries for NPM, Isaac is also the author of libraries such as: node-supervisor, nave, and several TAP ( Test Anything Protocol ) related libraries. Isaac's projects will help you improve your node.js development workflow. If you are intending to write large amounts of node code, you should read through his project list, it will make you a better programmer.

Tied for 6th - Marak Squires and Nickolay Platonov

I am Marak Squires, huzaah. I'm a co-founder of Nodejitsu and the creator of the hook.io framework. I have a fairly eclectic collection of projects including: say.js, Faker.js, JSONLoops, colors.js, webservice.js, translate.js, and node-raps. I also assist in maintaining several Nodejitsu projects. The majority of my new projects are hook.io hooks for helping create a Standard Hook Library.

Nickolay Platonov, aka SamuraiJack, is a JavaScript expert from Russia and the author of the Joose framework. Many of his projects are related to supporting Joose and ExtJS to build full-stack javascript solutions with rich user interfaces. If you are interested in building enterprise class applications using Joose and ExtJS, I don't think you will find someone with more expertise than Nickolay.

Tied for 8th - Charlie Robbins and Felix Geisendorfer

Charlie Robbins ( aka indexzero ) is a co-founder at Nodejitsu, maker of hot sauce, and author of libraries such as: Forever, Winston, Nconf. He also leads development for Nodejitsu on such projects as: node-http-proxy, haibu, and jitsu. Charlie's projects are among the highest quality projects available, and I do not say this lightly. I would recommend not only incorporating Charlie's work into your production stack, but actually diving into his projects to better understand best practices for Git workflow, Github project management, and node.js software development.

Felix is a very tall and friendly man from Germany. He is the co-founder of debuggable, the first node.js consulting company. Felix has made numerous contributions to node's core and is best known for his libraries such as: node-formidable and node-mysql . Felix also has a lot of smaller libraries for performing very specific and useful things. You should take your time to scan through Felix's projects, there might be a gem in there that will save you several hours of time.

Felix, like Charlie, tends to only release high-quality projects. It's a safe bet that a project from either of these two will be useful, and work as intended.

10th - George Stagas

George is a developer from Greece and an early Nodejitsu beta-tester. In addition to writing a lot of utility libraries, George also has some neat projects like Maga, a lightweight framework for developing multiplayer physics-based games, and Maptail, a visual client for mapping logs of ip address to a web map based on geographic location in real-time.

Getting Daily Tallies

Since people do enjoy tracking meaningless numbers, I've setup a quick application on Nodejitsu which wraps Substacks npmtop and outputs the results every hour. It's all open-source, check it out!

http://npmtop.nodejitsu.com

Remembering what's important

These numbers are meaningless, it's the data and people behind the numbers that really matter. Through exploring this data we can get a better understanding of how developers contribute to node.js. Most developers don't have the time to maintain a lot of projects, but they do create and maintain projects that are of very high quality and actively maintained. Remember not to assume more === better.

At Nodejitsu we believe in sharing knowledge. We believe that open source software fosters innovation, relationships and offers in return something better than the sum of its parts; a continuous growth cycle. If we tell you our good ideas, you'll make them better. Now go back two steps and repeat this process.

Passionate open source projects, we understand that they should be both engineered and crafted. A patient attention to detail and refinement in our products and platform remains our methodology. These characteristics: open, crafted, and connected have evolved into the current incantation of the Nodejitsu brand.

Build a product that works.

We have a platform that has become mature and ready for public consumption. One which is accessible to everyone, yet able to exceed the requirements of the enterprise. A platform tailored to the concerns of the developer. One with a pricing model that reflects usage rather than tenancy.

Design simple.

Simple designs are highly palatable. Less is more. Signal to noise ratios are critical. Carrying over these ideas from software engineering we reached out to Tyrus Smalley, friend, industry eminent, and creative force. Going through the motions of developing a visual language was a positive experience for a lot of reasons. The most important reason being that three technical co-founders coding 20 hours a day don't spend a lot of time thinking about presentation.

Design relevant.

Our new design has lots of wide open space. Space for you to think about the content. We wanted to describe a self-healing, distributed scaling intelligence in visual terms. Draw relevant diagrams that actually help illustrate these concepts. Answer questions such as What is infrastructure opacity?, What is distributed scaling? Some of these matters culminated in this animated diagram.

Make a foundation.

At first we started using a bonsai to depict an organic form that could be shaped and crafted. We retained the concept, but simplified the form. The simplification of the branches took shape as a cloud. And the rest of the tree was depicted as a node growing outward into the cloud with more nodes.

The word mark has mature modern forms with playful ligatures that keep things from getting too serious. We added a primary color (the blue color you can see it here on this blog or on the home page). We let the dark grey take the place of the secondary color, this keeps things looking subtle.

Painting a moving picture

These forms offer us a visual foundation to build on, and we'll continue to capitalize on them. But this is a fast moving technology. Node.js is in development, our product continues to evolve. So as focused as we are on the technology side, we wont forget to keep throwing paint on it and make sure it looks as good as it works.

It was only after I had begun to heavily use Github and Node.js that I realized these mystical beasts were actually real people, who work very hard on their projects, and generally strive for quality feedback so they can improve on their work.

Unfortunately, as I now help maintain and contribute to over 100 open-source projects, I've seen a lot of inappropriate behavior from developers trying to seek support. I think much of this behavior is unintentional. There are not a lot of guidelines for developers to follow who are new to open-source. In this article, I will dive into many of the common issues open-source project maintainers encounter and I will help developers better understand how they can file for a refund on the open-source project they are using.

There are no refunds in open-source

Most people understand this, sadly, some do not. If you are going to take an aggressive or angry approach when filing a request support, you are better off finding a new project. You need to understand that the person you are reaching out to has probably spent 100s of hours working on this project, for free. They do not owe you anything. The maintainers are extending a courtesy by giving away their work for free and then making themselves available to support it.

The point is, you should try and be nice when filing for support. The maintainer of the project has literally no obligation to help you.

Wrong place, wrong time

Want to really annoy an open-source maintainer? Then ignore any communication channels they have setup for support. If they setup a support wiki, send them a personal email. If they setup a Github Issues tracker, email the wrong mailing list about your question. If they setup an IRC room for support, send them direct messages over Twitter. If they setup a mailing list, post your issue to an unrelated forum.

The point is, take the time to actually research if the maintainer has setup a specific channel for support. If they have, use this channel. If you are uncertain, just reach out to the maintainer and ask where the best place to post support issues are.

Maintainers are not psychics

Here is a list of things I do not know: Who you are, what project you are asking about, what software you are using, what operating system you are on, what versions of the software you are using, what your stack traces look like, what you had for breakfast, and what the code looks like that caused your issue.

If you open up a support issue without providing this information, don't be surprised when the only response you get is asking for more information. Also, I probably don't care what you had for breakfast.

Read the f*cking documentation

If the author has taken the time to write documentation, you should read it. I don't write documentation for myself, I write it for other people. If the information you need isn't covered in the documentation, let the author know. They probably will want to add this information so other people don't ask the same question.

If there is no documentation to read, you should ask the maintainer why. If you want to help contribute to the documentation, you might find the maintainer of the project wants to be your new best-friend. If the maintainer is unwilling to help document the project, or assist you in documenting it, you should probably find a new project to use.

Run the f*cking tests

If the author has taken the time to write unit tests for the project, you REALLY need to confirm the unit tests work before trying to get support. Unit tests help provide a common way to test and isolate issues in a project. If you are having issues and one of your tests is also failing, it will be much easier to diagnosis the problem.

If you file a support issue without confirming you ran the tests, don't be surprised when the response you get is asking you to confirm the tests are working.

Read the f*cking code

Okay, this is a bit harsh. I don't expect everyone to read every single line of the code for a project they are trying to use, that isn't very reasonable. What I do see though, is that a lot developers have a mental barrier to actually opening up the source code for the project they are trying to use. They will read the documentation, run the tests, use the example code, but when they are faced with a problem that could be solved through a one or two line change in the source code, they shut down completely.

The point is that you shouldn't be afraid to jump into the source code. Even if you don't fully understand the source code, in many cases you should be able to isolate your issue to a specific block. If you can reference this block ( or line numbers ) when opening up your support request, it will help the author better understand your problem.

If you don't like it, fix it

Does something about a project really bother you? Does the author not care as much about it as you do? Do you find yourself angrily yelling at the nice people at the bus station due to your hatred of that inefficient algorithm you found? Here is a wonderful idea: Fix it.

Developers want to improve their project. If you find an issue, bring it up. If it's a valid concern, the author will probably want to have it fixed. In many cases, the author will consider it a valid issue, but simply not have the personal time or need to address it immediately. This is where open-source is great. Just fork the project and fix it your damn self.

If you fix something, let the author know

Did you actually fix a known issue? Let the author know about it. Not everyone has time to adhere to the specific coding styles for a project, so if you can't do a full blown pull-request, there is NOTHING wrong with opening a pull-request that only has the intention of showing the author how you solved the problem.

On many occasions, I've opened up requests for support in the form of a Github pull request. This way, I am telling the author: I have found a potential problem with your library, here is how I fixed it for my circumstance, here is the code I used for reference. You get extra internet points if you open the pull request with: "I don't expect this pull request to get merged, but I wanted to you show you what I did".

Don't get upset, rejection is normal

This is a follow up from the last section. Don't assume that because you opened up a pull request, that the author will accept it. There are many reasons that a maintainer might choose to not merge in your specific patch, many of which have nothing to do with you. If your patch isn't accepted, try to assume it's for a valid technical reason and not because the author hates you.

If you feel there has been an oversight, it's okay to not give up. As long as you are being logical and open to other people's views, you will find that you might learn something new, or even teach something to the maintainer.

If you don't like a project, don't use it

You would think this is obvious, but it's not. If you don't like a project and have no inclination to fix it, just don't use it. If you feel you are being "forced" to use a project, perhaps you should re-evaluate the life decisions which brought you to this point.

Conclusion

There are no refunds for open-source. If you go in with the attitude that you are owed something, you will be thoroughly disappointed. On behalf of all open-source developers and project maintainers, I ask you try and be polite the next time you ask for support. Try to remember that there is a real human being on the other side of the screen, and they actually want to help you.

Need to get up to speed on Node? No problem. Free Training
In the run up to Knockout, we are also going to be teaching two free training sessions on Saturday, Aug 20th and Sunday, Aug 21st. Our friends at Pivotal Labs NYC have been kind enough to provide space for these events.

Not on a Node Knockout team? Also no problem.
Even if you don't have a team to compete with, sign up and show up. There will certainly be adhoc teams developing on the scene. Bottom line, you don't have to bring a party to enjoy the party.

Thanks to all of those that made this possible!
We would like to thank all of those who are supporting our efforts in New York including 10gen, Amazon, Done, Foursquare, Mashery, Rackspace and others.

Schedule
Friday, Aug 26th
6pm-7pm: Nodejitsu / General Assembly Happy Hour 7pm-8pm: Welcome Presentations
8pm-1am: Game on! Let the coding begin. Saturday, Aug 27th 9am-1am: Doors at General Assembly are open!
Sunday, Aug 28th
7pm-9pm: Optional demos, first come, first serve 9am-8pm: Last sprint for finishing up projects.

The distillation of a philosophy to coding in a particular language (or environment) takes time, trial and error, and above all a group of extremely talented and dedicated professionals. Most of the ideas central to this philosophy are not new to the practice of Software Engineering or the study of Computer Science, but merely the selection of choices that appear often in the community with motivation from choices made in node.js core. With best efforts, sources and references will be supplied to support the arguments made here.

You first may ask "has there been enough time?" Node.js is still very much in its infancy as put so eloquently by Isaac Schlueter (author of npm) earlier this year at NodeConf. That is, Node.js has barely been around for two years. It may be therefore premature to write this, but there has the immense growth seen in both node.js core and in the ecosystem surrounding it. Nodejitsu has been intimately involved in this growth and thus, in the opinion of the author, enough time has passed for a few granules of truth to trickle through.

There surely has been a large amount of trial and error: as of writing this article there are over different 3200 modules on npm (the node package manager) many of which are failed attempts at one problem or another. And last but not least there is of course an immensely talented group of professionals working both on node.js code itself and on important modules which grow the node.js ecosystem.

Looking back over the evolution of the past two years, there is truly one central tenet to the development of node.js core set out by Ryan Dahl (the creator of node.js) that has defined the node.js philosophy this article is attempting to capture. That is, node.js core should be kept as small as possible. This is very clear on both the node.js and node.js development mailing lists where frequently feature additions are not included because they simply "do not belong in core."

The focus on a very small, performant, cross-platform compatible core was the impetus for the vast number of modules available through npm today. This is because many commonly requested features:

• "Framework": templating, routing, cookies, etc.
• Asynchronous flow control
• Pluggable Middlewares

were not available in node.js core. The need for these features from developers eager to use node.js allowed talented and prolific module authors to emerge and fill the void. This continually expanding and changing ecosystem of modules is in many ways the defining factor around the node.js philosophy: experimentation with small kernels of functionality which rely on loosely coupled components.

Experimentation

We are no strangers to mad science at Nodejitsu, but at it's core why is experimentation so core to the node.js community? With no large existing codebase to solve common web application development problems combined with Javascripts minimalistic language design many modules emerged to solve the same problem(s). Revisiting established problems caused many developers to ask "are the accepted approaches to solving the problem at hand necessarily the best?" This focus on experimentation has shown clear benefits: socket.io, dnode, the recent redesign of npm are some clear examples. I think it was put best by Mikeal Rogers (organizer of NodeConf): "if you think node.js needs less modules you're $%#@!^$ crazy."

Small kernels of Functionality

One of the benefits of having a well established package manager like npm (especially now that it is not possible to have version conflicts in node.js modules), is that a module developer can really focus on developing the best tool for an individual task. In other words, why package two modules together if you can simply break them apart into two kernels of functionality which are codependent? Exactly. There is no reason to do this unless your source code distribution (e.g. package manager) is deficient in someway.

Another way to think about this is to consider a phrase that I use often to describe node.js core: "close to the metal." For example, there are not a lot of abstractions or conventions between the developer and the raw HTTP stream. Each step along the way is a small kernel of functionality (the net module, the http module, etc.). This low-level functionality clearly has impacted the approach of the community at large.

Loosely coupled components

Coupling is poison to large codebases. It allows developers to hide improperly thought out abstractions and bad APIs by strictly defining and limiting the consumer to a fixed surface area. The idea of "loosely coupling" your components is that you can easily switch one out for another. This is something that was borrowed by Rails from Merb when they merged into Rails 3.0 and something that I am very glad is engrained into the approach taken by node.js developers. Early experiments on tight coupling such as Connect have not been repeated in other approaches and (in the case of Connect) considered "harmful" by their original authors; in this case Tim Caswell.

The argument in favor of loosely coupled components becoming the de-facto standard in node.js is (again) precipitated by npm. Without a strong, easy-to-use package manager with well defined semantics, module authors would be incentivized against decoupling components and releasing them separately.

The Road Ahead

Node.js is still not yet 1.0, yet it would be surprising for the core tenets of the philosophy to change. It is a philosophy that has led to important early-stage design choices within node.js, npm, and high-profile modules that continue to make node.js the technology to watch. It is this philosophy that results in the vibrant ecosystem, continued flow of interesting projects and modules, and intelligent debate around core functionality

However, if you talk to Node.JS users you may hear something along the lines of, "Javascript is actually pretty nice for writing command line apps." As it turns out, node.js is a great platform for writing command line applications, and with npm they are as easy to install---nay, easier---than similar programs written in ruby or python and installed with rubygems or pip. In addition, node.js has all sorts of helpful libraries for writing command line apps, such as prompt, optimist, and cliff, that make writing command line interfaces easy!

Here, I hope to showcase a few of the command line apps written in node.js. Many of them are still written with web development in mind. They are all written in javascript, and they are all used from the command line.

forever

Forever, written by Nodejitsu's Charlie Robbins, is a process supervisor written in node.js. More important, however, is that forever is robust and simple to use.

There are many process supervision frameworks out there. The problem with a lot of them is that they're often written with system init scenarios in mind and have their own DSLs. Other times, they're not written in a particularly user-friendly manner, or they aren't very robust. Sometimes, all of these things are true. Because of this (and other problems), myself and others before me have written their own ad-hoc process supervisors, using some unholy combination of nohup, screen and/or bash.

Forever is a much better solution. In fact, forever might be the best solution. It's certainly one of the best I've seen.

For an example, suppose you're a grad student (as I was in a past life) and you have a simulation you need to run that, due to buggy software, might as well look like this:

#!/usr/bin/env node

console.log("=================Super Number Cruncher 5000=================");
console.log("--Running important simulations for your thesis since now!--");

var i = 0;
setInterval(function() {
    if (Math.random() < 0.05) {
        throw Error('FLAGRANT ERROR');
    } else {
        //Generate some output data
        console.log((i++)+','+(273.15+60*Math.random()));
    }
}, 500);

The output of this program looks something like this:

josh@pidgey:~/broken_science_simulation$ ./simulation 
==================Super Number Cruncher 5000==================
---Running important simulations for your thesis since now!---
0,274.32085393052546
1,273.85998188573865
2,332.435783926025
3,312.9851095430553
4,323.98653633948413
5,298.41442882139233
6,314.28931980449704
7,291.39460802134124
8,297.300358219631
9,309.1636345820501
10,281.5037131268531
11,292.7776571927592
12,332.8452686598524
13,275.4756304487586

/home/josh/broken_science_simulation/simulation:9
        throw Error('FLAGRANT ERROR');
              ^
Error: FLAGRANT ERROR
    at Error (unknown source)
    at Timer.callback (/home/josh/broken_science_simulation/simulation:9:15)

What? Flagrant error?

(Of course, given that it's academia, it's probably written in java and MATLAB instead, but you get the idea. Also, ignore the implications of buggy science software. It's better if you don't think about it too much.)

Now, supposing that we can't track down this error because it's buried inside a proprietary software suite, we can just use forever to make it restart every time it crashes:

josh@pidgey:~/broken_science_simulation$ forever -o output.log -e err.log simulation
warn: Forever detected script exited with code: 1
warn: Forever restarting script for 1 time
warn: Forever detected script exited with code: 1
warn: Forever restarting script for 2 time
warn: Forever detected script exited with code: 1

This command puts your simulation's output into output.log, and the errors into err.log. As you can see, Forever restarts your process like a boss. Here's some example text from output.log:

51,275.67675554249433
52,323.125968520157
53,299.9657142587006
54,285.97948576379565
==================Super Number Cruncher 5000==================
---Running important simulations for your thesis since now!---
0,317.5622826308012
1,311.84217803701756
2,330.1117456337437
3,331.6873355587944
==================Super Number Cruncher 5000==================
---Running important simulations for your thesis since now!---
0,298.9083861099556
1,288.8923557434231
2,300.0416149344295
3,312.97375301420686

and here's some example text from err.log:

Error: FLAGRANT ERROR
    at Error (unknown source)
    at Timer.callback (/home/josh/broken_science_simulation/simulation:9:15)

/home/josh/broken_science_simulation/simulation:9
        throw Error('FLAGRANT ERROR');
              ^
Error: FLAGRANT ERROR
    at Error (unknown source)
    at Timer.callback (/home/josh/broken_science_simulation/simulation:9:15)

/home/josh/broken_science_simulation/simulation:9
        throw Error('FLAGRANT ERROR');
              ^

Clearly, errors are still a problem. However, with forever you can keep the bad news from your advisor. Just restart the process, and rewrite it to serialize output and start where it left off!

• Author: Charlie Robbins
• Web site: https://github.com/indexzero/forever
• Install: [sudo] npm install -g forever

http-server

http-server, written by Nodejitsu's Marak Squires, is a zero-configuration http-server. This means that all you have to do to serve the files in a folder is to run http-server from the command line, in that folder, without having to set any options. It's as easy as file serving gets!

For example, here's the content of a pure client-side web page that I wrote, and normally serve using github pages:

josh@pidgey:~/virtual-window$ ls
favicon.ico  frame.png  index.html  jquery.js  README.md  wall.jpg  window.css

Of course, the smart thing to do is to test the project locally before pushing changes to github. Plus, github pages is mildly annoying to work with anyway, right? With http-server, testing it from my computer is trivial:

josh@pidgey:~/virtual-window/$ http-server
Starting up http-server, serving . on port: 8080
http-server successfully started: localhost:8080
Hit CTRL-C to stop the server

Now all I have to do is go to localhost:8080, and there it is! Just like it is on github pages, without having to do any configuration at all.

http-server is also nice for quickly sharing material from my shell at a moment's notice, and for (legally, mind you) sharing games at LAN parties (for those of us that have time for such things).

• Author: Marak Squires
• Web site: https://github.com/nodejitsu/http-server/
• Install: [sudo] npm install -g http-server

browserify

You may have heard of browserify already. Browserify began life as a node.js module for making client-side javascript more awesome by:

• Giving you a node.js-compatible, commonjs-style module system for your client-side code, helping to keep everything clean and organized.
• Allowing you to grab node.js packages from npm and use them client-side, not only enabling simple code reuse, but also allowing one to run code intended for the server in the browser in a "it's so crazy it just might work" fashion. For example, the author of browserify has used it to run pure javascript stack traces in the browser.

However, not all has been peachy in the world of browserify. Many people complained about having to run this "heavyweight" solution, requiring a node server to be running in order to serve these bundles instead of, say, bundling the javascript in a "build" step and serving the results with Apache. Other solutions along a similar vein, such as ender.js, allow you to do this, so why not Browserify?

Well, the author listened, and browserify recently gained a command line interface.

Here's a quick example: Suppose I would like to bundle the hashish library for manipulating some objects on the client side, and that I would also like to include some es5-shim action with the shimify browserify middleware, just in case Mom is insisting on using IE7. Here it is:

josh@pidgey:~$ browserify --require hashish --plugin shimify -o bundle.js

Like that, assuming traverse and shimify are both installed, browserify bundles traverse with es5-shim into a file called bundle.js. Awesome!

• Author: James Halliday
• Web site: http://github.com/substack/node-browserify/
• Install: [sudo] npm install -g browserify

uglifyjs

Suppose you built the bundle from the browserify example, and you checked its size:

josh@pidgey:~$ wc -c bundle.js
59390 bundle.js

60 kilobytes?!

Okay, that's not too bad, but certainly we can do better, right? After all, the source from that output isn't minified or anything. For example, here's a short snippet from bundle.js:

// shallow copy
Hash.copy = function (ref) {
    var hash = { __proto__ : ref.__proto__ };
    Object.keys(ref).forEach(function (key) {
        hash[key] = ref[key];
    });
    return hash;
};

Goodness, that is way too readable!

Programmatically, browserify supports many more features than it does at the command line, including using minifiers such as uglifyjs as source filters. Uglifyjs is a high-quality, speedy javascript minifier that's a favorite amongst node.js developers.

Lucky for us, uglifyjs comes with a command line application! Check it out:

josh@pidgey:~$ browserify --require hashish --plugin shimify | uglifyjs | wc -c
21243

That's much better.

• Author: Mihai Bazon
• Web site: https://github.com/mishoo/UglifyJS
• Install: [sudo] npm install -g uglify-js

ngist

Ngist is a command line github gist tool, similar to Chris Wanstrath's gist from ruby. Its use is pretty similar.

Here's a really simple example of its use, after the initial git configuration step:

josh@pidgey:~$ npm search " cli " > /tmp/cli.npm.txt
npm info it worked if it ends with ok
npm info using npm@1.0.18
npm info using node@v0.4.9
npm info ok
josh@pidgey:~$ ngist /tmp/cli.npm.txt
Gist: https://gist.github.com/1115238
josh@pidgey:~$ 

Here, I searched npm for all modules with the word "cli" in their descriptions, then used ngist to gist the output. It worked great, as you can see for yourself.

Ngist can also use the contents of the clipboard, post multiple files at once (or a combination of files and clipboard contents), allow you to set the gist description, and even use someone else's github username and token.

Finally, ngist isn't only available as a command line app; you can also use it programmatically! This seems to be a common pattern in node.js cli apps, and it's one I really like.

• Author: Jacob Chapel
• Web site: https://github.com/chapel/ngist
• Install: [sudo] npm install -g ngist

jshint

jshint is a linting tool for javascript. While a fork of the more well-known jslint, its stated goals are to be more flexible and less opinionated than Doug Crockford's tool. The project, in addition to the web-based linter, also has a handy-dandy node-based cli tool for linting your projects!

For example, I decided to run it against a project of mine to see what would happen:

josh@pidgey:~$ jshint bf.js
bf.js/bf.js: line 16, col 22, Bad line breaking before ','.
bf.js/bf.js: line 21, col 21, Bad line breaking before ','.
bf.js/bf.js: line 24, col 43, Bad line breaking before '?'.
bf.js/bf.js: line 28, col 21, Bad line breaking before ','.
bf.js/bf.js: line 31, col 43, Bad line breaking before '?'.
bf.js/bf.js: line 35, col 21, Bad line breaking before ','.
bf.js/bf.js: line 40, col 21, Bad line breaking before ','.
bf.js/bf.js: line 47, col 21, Bad line breaking before ','.
bf.js/bf.js: line 57, col 21, Bad line breaking before ','.
bf.js/bf.js: line 82, col 22, Unnecessary semicolon.
bf.js/bf.js: line 91, col 21, Bad line breaking before ','.
bf.js/bf.js: line 92, col 21, Bad line breaking before ','.
bf.js/bf.js: line 93, col 19, Bad line breaking before ','.
bf.js/bf.js: line 96, col 2, Missing semicolon.
bf.js/test/tests.js: line 8, col 2, Missing semicolon.

15 errors
josh@pidgey:~$ 

Oh my! It seems that jshint does not like how I break my lines or place my semicolons! Because, whatever, I do what I want, I'll just dump the following into ~/.jshintrc so I can continue to write code that makes people want to stab their eyes out:

{
  "asi": true,
  "laxbreak": true
}

This tells jshint to allow automatic semicolon insertion and "lax" linebreaks. Now, when I run jshint:

josh@pidgey:~$ jshint bf.js
bf.js/bf.js: line 82, col 22, Unnecessary semicolon.

1 error

There, that's not so bad! The offending code on line 82 is for loop shenanigans I picked up from doing too much code golf, and I should probably rewrite it. Thanks, jshint!

• Author: Brent Lintner (node package)
• Web site: http://github.com/jshint/node-jshint/
• Install: [sudo] npm install -g jshint

Wrapping up

Forever, http-server, browserify, uglify, ngist and jshint are just a few of the command line applications written in node.js and available on npm! They are easy to install, and are crazy useful. In another blog post, I will discuss how to build your own command line applications with node.js, using options parsers like optimist, command line formatting tools like colors and cliff, and prompt, an easy to use module for prompting the command line for input. Stay tuned!

Nodejitsu is very happy to announce the latest stable version of hook.io. This release marks a combined effort from the Nodejitsu team working across six time-zones and four countries. But what does this mean for the application developer and what is hook.io anyway? Hook.io is a full-featured I/O framework for node.js that enables a simple way to distribute your application across multiple node.js processes using the new EventEmitter2 API and leveraging the power of Crash-only software.

In a nutshell, hook.io allows you to split your node.js program into multiple worker processes, that will be automatically restarted upon any user-generated faults, allowing you to increase performance and fault-tolerance. Interested in learning more? Read on!

Wait a second, EventEmitter2?

No, you didn't read it wrong; Nodejitsu has been working on EventEmitter2 for the last couple of months and there has been positive feedback about bringing it into node.js core. EventEmitter2 is a node.js event emitter implementation with namespaces, wildcards, TTL and browser support. Even if it is not brought into node.js core, it will continued to have the full support and backing of Nodejitsu; now and forever.

Lets take a look at how EventEmitter2 works:

Now that we've had the time to build up our infrastructure, we need you to come in and start applying the UNIX philosophy to our systems. Your role will be to "own" our servers and work with the entire team to make sure our services always stay up. We are currently expanding our platform to become a utility provider agnostic system, and we'll need your expert systems programming knowledge to help facilitate integrating with cloud computing providers ( such as: Rackspace, AWS, Joyent, etc...) and setting up redundant dedicated environments across the globe.

Requirements:

• Expert Level Knowledge of Linux Server Administration in High Availability Production Systems
• Expert Knowledge in Shell Scripting, C, Assembly, OSI Model, Systems / Networking Security
• High aptitude for self-management
• Availability to report on-site in New York City or San Francisco

Preferable:

• Dislikes JavaScript
• Prefers talking to our 1000s of servers instead of people
• Has read every single BOFH story

Compensation

• 100-120k per year depending on experience
• Equity in Nodejitsu ( vested over time )
• Health Benefits

Interested candidates can apply by emailing hiring@nodejitsu.com

We do Mad Science.

First there is the Science part, a Mad Science project has to be a bold idea about how some big problem might be solved. Then there is the Mad part. This is the way that once you've thought of your mad science idea, you can't forget it. It niggles away at you, and whenever you encounter your problem again you remember it, thinking, this problem could be solved with some Mad Science.

The only option you have left is to try and build it.

One mad science project associated with nodejitsu is Marak's Hook.io. Hook.io is a generic implementation of "web events". Events are happening everywhere on the internet, all the time. But how do you listen to them? And what amazing applications could you build if you could easily connect all these different events together in an organized way?

Marak did a talk on hook.io at JsConf 2010. since then hook.io has been on the back burner as nodejitsu has been ramping up, but it has recently reemerged with a new approach, based on a marriage between dnode (substack's RPC protocol), socket.io (cross browser websocket support) and EventEmitter2 (another nodejitsu Mad Science project which is a reworking of node's native EventEmitter, but with proper namespaced events)

I also have a mad science project, Meta-Modular. Meta-Modular is based on the idea that version numbers are wrong, and are not the way your dependencies should be specified. Instead, you should declare your dependencies by specifying what tests a dependency should pass. Then, in theory, any implementation of a dependency that passes the test should work in that slot. However, implementing a system like this needs not just a way to dynamically inject dependencies into a test run, but also to analyze large amounts of multi-dimentional data, to see what combinations of dependencies work with what tests.

Imagine the time that could be saved if there was a service that could tell you what forks passed a particular test. And imagine what could then be done with that time, and that knowledge!

As you explore a mad science idea you inevitably encounter difficulties. Success usually feels like it is a month away, and that the new approach it the one, but you never know.

To illustrate this problem I like to use the example of the three parallel rivers in Yunnan province in China. In South West China the Yangtze, Mekong, and Salween rivers, three of the largest rivers in Asia, emerge from the Tibetan Plateau and flow parallel through three gorges between four mountain ranges. Suppose you are the plucky adventurer at one side of this, and you want to know whats on the other side...surely it is a lush fertile plain. So you climb the first mountain range, and what do you find? A gorge, and another mountain range. Surely the plain is on the other side of that, so you cross the river and the next mountain range. By now many members of your party have died of exposure or drowned in dangerous river crossings. You get to the top of this range...only to see another gorge, and another mountain range. This is when you start to ask yourself, just how many gorges are there?

Should you give up? Or push on into the unknown?

For the truly heroic adventurer, this is not even a question.

So far, Marak has attacked the evented web problem three times in node.js, with two other attempts using Python's Twisted and even Coldfusion. I've tried meta-modular in Java, Ruby, and now Node.js. Currently it manifests itself as testbedjs.org which is an open-source community based continuous integration system.

There are many Mad Scientists working in Node.js. Infact, Node is a mad science project. Ryan has been working on the idea of a tinyclouds framework for years. Ultimately, the idea is many lightweight asynchronous evented servers, which communicate by message passing. Before Node got popular, Ry tried out many solutions using Ruby, C, Haskell, Lua, Nginx, whatever he thought would be the best way to solve his problem. Check out some of ry's older work at http://tinyclouds.org/libebb/ and https://github.com/ry/ebb/.

It seems Ry struck gold when he switched to v8 and JavaScript, and suddenly many other mad scientists saw the potential and started working with Node.js. We see people like Isaacs, Tim Caswell, Guillermo Rauch, Matt Ranney, Paul Querna, James Halliday, and Jeremy Ashkenas all gravitating towards Node.