Lately, I've noticed quite a few people asking to see examples of flatiron apps in #nodejitsu. Now, there are actually quite a few of our projects that use flatiron extensively. Unfortunately, there are a few problems, chief among them that no two flatiron projects do quite the same things or are organized the same ways.

Now, this isn't necessarily a problem when it comes to building tools. In fact, I think it's a great demonstration of flatiron's flexibility. Each project can use exactly the libraries and organization that make sense for their applications, even for apps as different as haibu and jitsu. It can be confusing for newcomers, though, that are used to frameworks that enforce stronger conventions, or those that are trying to figure out how to get from "hello world" to something more profound.

I thought what would be helpful for flatiron newcomers is an in-depth tutorial, which iterates on a "hello world" flatiron server (like those in the project's examples) into a relatively complex webservice that uses some of flatiron's more complex functionality.

In this tutorial, I'm going to create a webservice for browserify, a popular client-side code framework for node.js. All of the code is on github, and every section of this tutorial corresponds to a code commit. In early parts of this tutorial I will show all the code, but as it gets longer I will typically explain the code with links to the relevant files/commits on github.

At Nodejitsu, we are always on the lookout to find great javascript conferences that revolve around awesome communities.

We are also co-hosting two events in April: The Node Philly Conference and the NYC MongoDB Hackaton and speaking at three more!

Here's our schedule: be sure to say hi and hangout.

At Nodejitsu, we spend a lot of time and resources on open-source software development. Before founding Nodejitsu, the three of us were all active open-source enthusiasts and developers. Tools like Github helped bring us together to better understand each others techniques and technical skill-sets. Events like NYC.JS, Music Hack Day NYC, and the first Node Knockout helped solidify the team.

Now, after three years of toiling away on Github, we have now expanded our engineering team to over fourteen strong and every Nodejitsu developer is an active contributor to the open-source community.

• How to become an open-source developer?

• How do you get recruited as an open-source developer?

#

How to become an open-source developer?

Becoming an "open-source developer" is actually quite simple. You really only need to do one thing: contribute.

Contributing can be done on several levels of participation. One of the easiest ways to contribute to a project is to use it, find an issue, and report the problem. If you've found an actual problem that can be resolved, hopefully one of the maintainers will respond by fixing the issue or providing guidance on how to resolve the issue.

There are also several other useful things you can do such as: unit-tests, benchmark suites, documentation, multi-platform support, third-party integrations, and new functionality.

How do you get recruited as an open-source developer?

If you are contributing ( as described in the previous paragraph ), you've already done the hardest part.

Good work always speaks for itself.

If you have worked on projects which are significant to companies who are hiring developers, they probably already know your name. It's much more likely a company will hire a developer who has already made an improvement to a piece of their codebase versus someone who has little knowledge of any company projects. At Nodejitsu, we primarily hire developers who have already contributed to major company supported projects like: jitsu, haibu, flatiron, and hook.io.

pattern (github, p on npm) is a way to iterate javascript collections with asynchronous functions using a technique called pattern matching.

Command-line interfaces (CLIs) are an often overlooked, but extremely important part of every developers workflow. Think about how different your day-to-day life would be if git functioned differently? Or (gasp) incorrectly or inconsistently? At Nodejitsu, we know CLI applications. The CLI for our public platform jitsu is designed to be quick to get started, highly portable and above all: consistent and robust. With these principals in mind we took extra care designing Flatiron to be adaptable to our production-quality needs.

Flatiron is an adaptable decoupled framework for both node.js and the browser. An adaptable framework enables users to build multiple types of applications. In Flatiron we are focusing on:

• Web applications (http and websockets)
• Command line (CLI) applications
• Front-end applications

This article will focus on the tools at your disposal when using Flatiron to build CLI applications as well as how many existing Flatiron CLI applications are structured.

• Getting started
• To the code!
• Configuring your Command-line Application
• Composable Command-line applications
• Flatiron Command-line apps in the wild

You may know about SPDY, an experimental protocol for a faster web http://www.chromium.org/spdy/spdy-whitepaper announced by Google in 2009. SPDY is now supported by two major browsers (Google Chrome and Mozilla Firefox) and it’s running on most of Google’s servers (also soon to be supported by nginx).

The main goal of SPDY is reducing the latency of web pages by multiplexing multiple http streams into one TCP connection and compressing http headers.

So instead of creating a TCP connection for each http request (like styles, scripts, images, or even ajax), SPDY-enabled browser will hold only one connection with the server.

Github: https://github.com/flatiron/blacksmith
Documentation/Example: http://blacksmith.jit.su

Blacksmith is a static site generator. It uses weld, jsdom and marked to turn json, markdown and 100% genuine HTML and CSS, on the filesystem, into awesome sites like the one you're looking at right now. Moreover, it includes a command line interface for quickly building, editing and testing blacksmith-built websites. Here's a short list of features:

• Easy to use
• Can be hosted anywhere, since it generates static HTML/CSS
• Write and Edit articles on the file system using github-flavored markdown
• Based on JSDOM and Weld
• Easily create custom themes using plain HTML and CSS ( no micro-templating ! )
• Default theme is a port of scribbish by Jeffrey Allan Hardy
• Ships with a robust node.js static server suitable for production

Blacksmith may remind you a bit of jekyll and octopress, and in fact it is influenced by these projects. However, whereas jekyll is "blog-aware" and uses liquid for templating, blacksmith uses the DOM for its templating needs and can easily generate not just blogs, but documentation sites and more. Plus, you don't need ruby or rake since it's all written in node.

As part of my ongoing quest to develop Skynet, I've been thinking a lot about the decentralized storage and distribution of small amounts of state in a peer-to-peer environment. With new privacy laws, the internet is fundamentally changing in a way we have not seen before. It's up to us, the developers to make sure the internet remains a fair and open place.

In software application development we usually store application state in volatile and non-volatile random access memory. We also use network devices to communicate state between our application and a central provider or server. We store state in things like databases, the file-system and cloud hosting providers. This approach is recommended for most applications, but when we start to think about the implications of building large, distributed, and decentralized applications, we begin to rethink about how these applications are going to store and maintain state.

In some way or another, almost anything can store state. Any device that can be written to, then read from later can be used to store state.

Think of it like this:

There are hundreds of thousands of endpoints on the web right now that you can post some sort of data to, and read it back later. Why not make a database out of that?

#

Basic strategies in building a decentralized and anonymous database

1. No central servers. A central server is a central point of failure.
2. No central authorities. A central authority over control of the database indicates another ( and much worse ) central point of failure.
3. Ability to encrypt and obscure data.
4. Ability to run in any combination of local / cloud / hosted. The database should be able to run locally, on hosted servers, and in the cloud.
5. Ability to easily share and replicate data. The data should be able to easily transfer and replicate between multiple systems and users.

The goals of building a decentralized and anonymous database are fairly straight forward. The data needs to have no central point of failure ( technological or political ). The data needs to be able to be secured. The data needs to be accessible and sharable.

Introducing hnet

hnet is a decentralized and anonymous database built in node.js. It works by spreading small amounts of information across several nodes using a variety of non-traditional storage engines.

http://github.com/hookio/hnet/

What is an hnet node?

An hnet node is anything that we can store a small amount of data which we can eventually read back later. The hnet client communicates with many hnet nodes in order to establish a dataset.

What is an hnet storage engine?

A storage engine can be considering a wrapper for any service on the web that can store data, and read it back later. Storage engines are small and pluggable, so it should be very easy to author new engines. Some examples of engines are: gist, pastebin, imgur, google groups, reddit, twitter, irc, etc... The only real requirement for a storage engine is that it can store state, the rest is up to your imagination.

Why hnet?

hnet was originally built as an easy way to provide distributed and semi-anonymous tables of server ips and ports for hook.io hooks. hnet helps provide hook.io a way of auto-discovery other hooks over a Wide Area Network. Note: hook.io already supports auto-discovery over the Local Area Network via mdns.

How does hnet work?

As part of my ongoing quest to develop Skynet, I've been deep diving into distributed computing.

At Nodejitsu, our node.js hosting platform deals with 1000s of live servers spanned across multiple data-centers. As scale increases, minute statistical probabilities become very real problems. The network is unreliable, disks become unwritable, streams break, unexpected input is unexpected, and entire data-centers can go down.

Think of it like this:

If you have 1,000 servers each individually rated at 99.9% uptime, on average, one of those machines is always failing.

#

So we research...

Fault tolerant system
Fallacies of Distributed Computing
Properties of distributed systems
High availability Percentage calculation

and we research more...

Shared nothing architecture
Byzantine fault tolerance
Paxos

Eventually, we hit the The Halting Problem:

Given a description of a computer program, decide whether the program finishes running or continues to run forever. In 1936, Alan Turing proved that a general algorithm to solve the halting problem for all possible program-input pairs cannot exist.

to summarize all of these links in the context of this blog post...

It's mathematically impossible your distributed application is not going to fuck up. Deal with it.

Basic strategies in building fault tolerant applications in node

1. The operating system's RAM, processes, and file descriptors are relatively cheap. Use them as needed.
2. Use JSON to pass data between nodes. ( see: nssocket or dnode ).
3. Embrace crash-only design. Nodes should be able to disconnect, restart and reconnect as fast as possible with minimal impact.
4. Never assume any one node process will never go down. Assume that any process can go down for any reason at any time.

The last point is particularly important. If your application is unable to recover from a "central" node going down, it should not be considered fault tolerant.

hook.io

hook.io is a distributed input/output framework for node.js. It allows you to seamlessly link together several processes and start sending messages between them. hook.io also provides a rich network of stand-alone hook libraries for adding additional i/o sources. In a sense, hook.io can be described as a next generation enterprise service bus.

• Email List: http://groups.google.com/group/hookio
• Video Lessons: http://youtube.com/maraksquires ( mirror )
• Wiki Pages https://github.com/hookio/hook.io/wiki/_pages
• hook.io for dummies
• Distribute Node.js Apps with hook.io:
• #nodejitsu on irc.freenode.net

Basic Paxos

Paxos is a family of protocols for solving consensus in a network of unreliable processors. Consensus is the process of agreeing on one result among a group of participants. This problem becomes difficult when the participants or their communication medium may experience failures.

As of v0.8.6, hook.io now has the ability to do a very basic consensus among all hooks to ensure recovery if any hooks die.

In previous versions of hook.io, if the "master" or "central" server hook went down, all other hooks would stop communicating.

Now, hook.io is able able to recover from the "central" server hook going down.

Previously, when the hub went missing, none of the spokes knew where to connect anymore.

Building fault-tolerant node.js application with hook.io

hook.io can be used programatically or as stand-alone binary. For simplicity, we will assume in this example that you are using hookio as a binary application, and with all it's default settings.

For more examples of using hook.io programtically check out the examples or test folders.

To install hook.io with npm

 npm install hook.io -g

Now, we are going to start up three instances of hookio in our terminal.

Terminal 1

 hookio

Terminal 2

 hookio

Terminal 3

 hookio

Using hook.io's default auto-discovery functionality the first terminal will take the role of server, and the second and third terminals will connect as clients to the first terminal.

Now, kill the first hook terminal with CTRL-C. The second hook will take over the role of the server. The third hook will now know to connect to the second hook instead of the first.

It might not seem very impressive, but it demonstrates a basic example of how to build fault-tolerant multi-process distributed applications in node.js

Applications built with hook.io now have no central point of failure.

But wait, there's more!!!

mdns

Multicast DNS (mdns) is a way of using DNS programming interfaces, packet formats and operating semantics on a small network where no DNS server is running. The mDNS protocol is used by Apple's Bonjour and Linux Avahi service discovery systems. mdns is an easy way to help networked devices find each other without any prior configuration.

As of v0.8.6, hook.io now has alpha support for zero configuration networking via mdns. This means that if you hookio -m on two machines on the same Local Area Network...they will find each other and instantly start communicating without any configuration!

Computer A

 hookio -m

Computer B

 hookio -m

Now these two computers ( connected over a LAN, with no central DNS server ) will automatically discovery each other and begin to transmit messages.

Think of the possibilities!

Conclusion

With the new mdns and paxos features, hook.io is getting better everyday. The current goal is to stabilize this new functionality in the 0.8.x branch across all platforms.

The next steps will be p2p hook trackers and hosted hook services...

npm is the node package manager.

It helps you manage your node.js modules and dependencies.

Previously, we created a npm cheat sheet describing various NPM commands.

Now, here is a package.json cheat sheet for exploring various properties of npm's package.json format.

Enjoy the package.json cheatsheet!

If you watch flatiron repositories on GitHub, you may have started noticing these:

These are Travis CI build status images. We started testing our code at Travis CI and we want to tell the whole world about it!