The blog is moving to our new website. Read all the latest news at Sonian Blog.

Wednesday, August 6, 2008

Storing the world's data: Sonian invents new way to manage archive data

Storing the worlds email data is no small feat. Sonian has been working on this problem for almost 2 years. Our architecture started with the idea that cloud compute infrastructure would be the best starting point for a next-generation archive service. This was before "the cloud" was talked about as a viable platform for enterprise needs. I am very pleased to say we made the right bet, and have 24 months of experience in this area as others scramble to catch up.

Building for cloud compute gave us the opportunity to start with a clean sheet of paper for the foundation elements: storage, processing, database and presentation.

For storage we developed a compliance layer on top Amazon S3. This makes Sonian compliant with SEC storage rules, and makes S3 useful for the enterprise. Our layer also adds encryption and tamper guards.

Processing in the cloud with on-demand CPU available in real-time allowed us to create a software stack that can expand and contract computational needs as customer data flows through the pipeline process. With this we get a system that can grow to meet the indexing and searching needs without idle CPUs burning away.

Erlang is used as an enterprise service bus to coordinate all data processing needs. Erlang has been around for many years, and has had a resurgence as a great language for messaging-based applications. Map-reduce and parallel processing capabilities are all built-in.

The Sonian database is a mixture of open source and proprietary technologies coupled with the cloud storage sub-systems. All data, including the full text indexes, are stored encrypted. Our enterprise audience needs to know their data is secure and stored reliably for as long as they need the information preserved.

Our presentation layer is based on the very agile Ruby on Rails web framework. Rails is a great web framework based on model, view controller principles and allows us to quickly add new user interface features and get them to the customer in six week development cycles.

With our cloud compute foundation in place we will be adding more features that leverage what we have created so far.

No comments: