For the Nutch project we needed distributed computing technology, we needed to store datasets that were much bigger than we could store on one computer, and we needed to have processes that would run and be coordinated across multiple computers.

FORBES: Open-Source Solves Big-Data Problems: Talking to 'Mr. Hadoop,' Doug Cutting