Petabyte-Scale Object-Based Storage

Description

We are investigating the construction of large-scale storage systems using object-based storage devices OSDs). An OSD is a network-attached storage device that presents an interface of arbitrarily-named data objects of variable size rather than sequentially numbered fixed-size blocks, to deal with the data storage details, such as request scheduling and data layout. Metadata is managed separately by one or more specialized metadata servers (MDSs), which is critical to scalability, reliability and security. The separation of data and metadata storage and management provides very high access bandwidth to the large-scale distributed storage systems.

Status

We are developing a prototype implementation of Ceph, a distributed file system based on our research. The metadata server (MDS) is based on Dynamic Subtree Partitioning, an architecture that adaptively distributes metadata across a cluster based on the current workload. Intelligent OSDs manage data replication, failure detection, and data migration during failure recovery or system expansion. Data is stored by each OSD using EBOFS, an object file system based on prior experience with OBFS. Data is distributed using CRUSH, a hash-like distribution function that allows any party to calculate (instead of looking up) the location of data. CRUSH is designed to cope with device failure and cluster expansion, while separating object replicas across failure domains for improved data safety.

The Ceph source code is available at SourceForge.

Publications

• Andrew Leung, Shankar Pasupathy, Garth Goodson, Ethan L. Miller, Measurement and Analysis of Large-Scale Network File System Workloads, Proceedings of the 2008 USENIX Technical Conference, June 2008.
• Andrew Leung, Ethan L. Miller, Stephanie Jones, Scalable Security for Petascale Parallel File Systems, Proceedings of SC '07, November 2007.
• Sage Weil, Andrew Leung, Scott A. Brandt, Carlos Maltzahn, RADOS: A Fast, Scalable, and Reliable Storage Service for Petabyte-scale Storage Clusters, Proceedings of the ACM Petascale Data Storage Workshop 2007 (PDSW 07), November 2007.
• Kevin Greenan, Ethan L. Miller, Thomas Schwarz, Darrell D. E. Long, Disaster Recovery Codes: Increasing Reliability with Large-Stripe Error Correction Codes, Proceedings of the 3rd International Workshop on Storage Security and Survivability (StorageSS 2007), held in conjunction with the 14th ACM Conference on Computer and Communications Security (CCS 2007), October 2007.
• Kristal Pollack, Darrell D. E. Long, Richard Golding, Ralph Becker-Szendy, Benjamin C. Reed, Quota Enforcement for High-Performance Distributed Storage Systems, Proceedings of the 24th IEEE Conference on Mass Storage Systems and Technologies (MSST 2007), September 2007, pages 72-84.
• Andrew Leung, Eric Lalonde, Jacob Telleen, James Davis, Carlos Maltzahn, Using Comprehensive Analysis for Performance Debugging in Distributed Storage Systems, Proceedings of the 24th IEEE Conference on Mass Storage Systems and Technologies (MSST 2007), September 2007, pages 281-286.
• Joel Wu, Scott A. Brandt, Providing Quality of Service Support in Object-Based File System, Proceedings of the 24th IEEE Conference on Mass Storage Systems and Technologies (MSST 2007), September 2007.
• Max Mehech, The Impact of Failures on Large Distributed Storage Systems, Technical Report UCSC-SSRC-07-10, August 2007.
• Andrew Leung, Scalable Security for High Performance, Petascale Storage, Technical Report UCSC-SSRC-07-07, June 2007.
• Qin Xin, Understanding and Coping with Failures in Large-Scale Storage Systems, Technical Report UCSC-SSRC-07-06, May 2007.
• Andrew Leung, Eric Lalonde, Jacob Telleen, James Davis, Carlos Maltzahn, Using Comprehensive Analysis for Performance Debugging in Distributed Storage Systems, Technical Report UCSC-SSRC-07-05, May 2007.
• Joel Wu, Bo Hong, Scott A. Brandt, Ensuring Performance in Activity-Based File Relocation, Proceedings of the International Performance Conference on Computers and Communication (IPCCC '07), April 2007, pages 75-84.
• Feng Wang, Storage Management in Large Distributed Object-Based Storage Systems, Ph.D. thesis, University of California, Santa Cruz, December 2006.
• Sage Weil, Scott A. Brandt, Ethan L. Miller, Carlos Maltzahn, CRUSH: Controlled, Scalable, Decentralized Placement of Replicated Data, Proceedings of SC '06, November 2006.
• Sage Weil, Scott A. Brandt, Ethan L. Miller, Darrell D. E. Long, Carlos Maltzahn, Ceph: A Scalable, High-Performance Distributed File System, Proceedings of the 7th Conference on Operating Systems Design and Implementation (OSDI '06), November 2006.
• Jehan-François Pâris, Darrell D. E. Long, Using Device Diversity to Protect Data against Batch-Correlated Disk Failures, Proceedings of the 2nd ACM Workshop on Storage Security and Survivability (StorageSS 2006), October 2006.
• Andrew Leung, Ethan L. Miller, Scalable Security for Large, High Performance Storage Systems, Proceedings of the 2nd ACM Workshop on Storage Security and Survivability (StorageSS 2006), October 2006.
• Joel Wu, Scott A. Brandt, The Design and Implementation of AQuA: an Adaptive Quality of Service Aware Object-Based Storage Device, Proceedings of the 23rd IEEE / 14th NASA Goddard Conference on Mass Storage Systems and Technologies, May 2006, pages 209-218.
• Sage Weil, Feng Wang, Qin Xin, Scott A. Brandt, Ethan L. Miller, Darrell D. E. Long, Carlos Maltzahn, Ceph: A Scalable Object-Based Storage System, Technical Report UCSC-SSRC-06-01, March 2006.
• Christopher Olson, Ethan L. Miller, Secure Capabilities for a Petabyte-Scale Object-Based Distributed File System, Proceedings of the 2005 ACM Workshop on Storage Security and Survivability (StorageSS 2005), November 2005. Won Best Full Paper award.
• Qin Xin, Thomas Schwarz, Ethan L. Miller, Disk Infant Mortality in Large Storage Systems, Proceedings of the 13th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS '05), September 2005.
• Joel Wu, Scott A. Brandt, Hierarchical Disk Sharing for Multimedia Systems and Servers, Proceedings of the 15th ACM International Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV 2005), June 2005, pages 189-194.
• Qin Xin, Ethan L. Miller, Thomas Schwarz, Darrell D. E. Long, Impact of Failure on Interconnection Networks in Large Storage Systems, Proceedings of the 22nd IEEE / 13th NASA Goddard Conference on Mass Storage Systems and Technologies, April 2005.
• Kristal Pollack, Scott A. Brandt, Efficient Access Control for Distributed Hierarchical File Systems, IEEE / NASA Goddard Conference on Mass Storage Systems and Technologies (MSST 2005), April 2005.
• Qin Xin, Thomas Schwarz, Ethan L. Miller, Availability in Global Peer-To-Peer Storage Systems, Distributed Data and Structures 6, Proceedings in Informatics, December 2004.
• Sage Weil, Kristal Pollack, Scott A. Brandt, Ethan L. Miller, Dynamic Metadata Management for Petabyte-Scale File Systems, Proceedings of the 2004 ACM/IEEE Conference on Supercomputing (SC '04), November 2004.
• Thomas Schwarz, Qin Xin, Ethan L. Miller, Darrell D. E. Long, Andy Hospodor, Spencer Ng, Disk Scrubbing in Large Archival Storage Systems, Proceedings of the 12th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS '04), October 2004, pages 409-418. Won Best Paper award.
• Qin Xin, Ethan L. Miller, Thomas Schwarz, Evaluation of Distributed Recovery in Large-Scale Storage Systems, Proceedings of the 13th IEEE International Symposium on High Performance Distributed Computing (HPDC 2004), June 2004, pages 172-181.
• Joel Wu, Scott A. Brandt, Storage Access Support for Soft Real-Time Applications, Proceedings of the 10th IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS 2004), May 2004.
• Sage Weil, Scott A. Brandt, Ethan L. Miller, Kristal Pollack, Intelligent Metadata Management for a Petabyte-Scale File System, 2nd Intelligent Storage Workshop, May 2004.
• Andy Hospodor, Ethan L. Miller, Interconnection Architectures for Petabyte-Scale High-Performance Storage Systems, Proceedings of the 21st IEEE / 12th NASA Goddard Conference on Mass Storage Systems and Technologies, April 2004, pages 273-281.
• Feng Wang, Scott A. Brandt, Ethan L. Miller, Darrell D. E. Long, OBFS: A File System for Object-Based Storage Devices, Proceedings of the 21st IEEE / 12th NASA Goddard Conference on Mass Storage Systems and Technologies, April 2004, pages 283-300.
• Feng Wang, Qin Xin, Bo Hong, Scott A. Brandt, Ethan L. Miller, Darrell D. E. Long, Tyce T. Mclarty, File System Workload Analysis For Large Scientific Computing Applications, NASA/IEEE Conference on Mass Storage Systems and Technologies (MSST 2004), April 2004, pages 139–152.
• R. J. Honicky, Ethan L. Miller, Replication Under Scalable Hashing: A Family of Algorithms for Scalable Decentralized Data Distribution, Proceedings of the 18th International Parallel & Distributed Processing Symposium (IPDPS 2004), April 2004.
• R. J. Honicky, Ethan L. Miller, A Fast Algorithm for Online Placement and Reorganization of Replicated Data, Proceedings of the 17th International Parallel & Distributed Processing Symposium (IPDPS 2003), April 2003.
• Scott A. Brandt, Ethan L. Miller, Darrell D. E. Long, Lan Xue, Efficient Metadata Management in Large Distributed File Systems, Proceedings of the 20th IEEE / 11th NASA Goddard Conference on Mass Storage Systems and Technologies (MSST 2003), April 2003, pages 290-298.
• Qin Xin, Ethan L. Miller, Thomas Schwarz, Darrell D. E. Long, Scott A. Brandt, Witold Litwin, Reliability Mechanisms for Very large Storage Systems, Proceedings of the 20th IEEE / 11th NASA Goddard Conference on Mass Storage Systems and Technologies, April 2003, pages 146-156.
• Ethan L. Miller, Darrell D. E. Long, William E. Freeman, Benjamin C. Reed, Strong Security for Network-Attached Storage, Proceedings of the 2002 Conference on File and Storage Technologies (FAST), January 2002, pages 1-13.
• Ethan L. Miller, Randy Katz, RAMA: An Easy-To-Use, High-Performance Parallel File System, Parallel Computing 23(4), July 1997, pages 419-446.