node-haystack Episode 1: What is it and why

What is haystack?

The original idea came from Facebook(https://www.facebook.com)’s famous paper of Finding a needle in Haystack: Facebook’s photo storage. Of course, this website doesn’t exist at all. So I uploaded the non-exist paper.

Why haystack?

Facebook doesn’t share their works, but someone provided a implementation named SeaWeedFS in [go programming language].

SeaweedFS is a simple and highly scalable distributed file system. There are two objectives: to store billions of files! to serve the files fast! Instead of supporting full POSIX file system semantics, Seaweed-FS choose to implement only a key~file mapping. Similar to the word “NoSQL”, you can call it as “NoFS”. https://groups.google.com/forum/#!forum/seaweedfs —– [SeaweedFS on Github]

I didn’t use SeaweekFS but It looks working very well. Some indexes are impressive.

What is node-haystack

node-haystack is a implementation of haystack in javascript and run under node.js. It’s low level operations(file access, threading) are written in C++ and has a node.js cover which makes it work with node.js.

Why node-haystack

Nothing, just for fun and I can.

Goal

The ultimate goal of node-haystack is:
- Efficient way to store billions files
- Server files as fast as possible
- RESTful interface