With more than 15 billion photos (and 60 billion image files with replication for different sizes), Facebook eats up a lot of storage with its photo application alone. Members are adding 220 million new photos every week. Facebook currently has more than 1.5 petabytes of storage for its photos, and that is growing at a rate of 25 terabytes a week. Last year, Facebook spent an estimated $30 million on NetApp storage appliances alone just to keep up with the growth of photos and other uploaded content. To reduce some of these costs, Facebook decided to engineer its own storage architecture called Haystack.
Now more details have emerged about how that system actually looks and works. In a nutshell, Haystack will allow Facebook to switch from expensive, commercial storage appliances to commodity off-the shelf hardware. It is going from a traditional network file system to something more akin to stripped-down network application that does only what it needs to do. Not only will Facebook get the cost savings of going commodity, but they also get a 3X improvement in storage capacity. In other words, what used to take 30 discs to store, now will take only 10.
With so many billions of images, serving the right one is like finding the proverbial needle in the haystack. With a traditional network file system, a lot of metadata goes flying around detailing when files were last modified, what directories they are listed in, and so on. All of this metadata creates a bottleneck as it is passed back and forth. So the two Facebook engineers who built Haystack (Doug Beaver, Peter Vajgel, and Jason Sobel) decided to get rid of much of this metadata. As they explain on Facebook’s engineering blog:
The new photo infrastructure merges the photo serving tier and storage tier into one physical tier. It implements a HTTP based photo server which stores photos in a generic object store called Haystack. The main requirement for the new tier was to eliminate any unnecessary metadata overhead for photo read operations, so that each read I/O operation was only reading actual photo data (instead of filesystem metadata)
All of that metadata is stored in what Facebook is calling “needle.” Each needle pulls together the metadata for hundreds of thousands of images. The needles are paired with an index to make up the Haystack object store. You can read all the technical details on the Facebook engineering blog. The company will keep its existing network file system for the 15 billion photos already uploaded (after all, those NetApp boxes are sunk costs). But going forward, all new photo uploads will be handled by Haystack. And in the future, Facebook may even open-source the architecture so other companies can benefit from it. Not bad, for something that was built by three engineers.
(Photo credit: Flickr/Vitor Antunes)