Ultra-Large Scale Storage

Description

We are investigating the construction of large-scale storage systems using object-based storage devices (OSDs). An OSD is a network-attached storage device that presents an interface of arbitrarily-named data objects of variable size rather than sequentially numbered fixed-size blocks, to deal with the data storage details, such as request scheduling and data layout. Metadata is managed separately by one or more specialized metadata servers (MDSs), which is critical to scalability, reliability and security. The separation of data and metadata storage and management provides very high access bandwidth to the large-scale distributed storage systems.

Status

We have developed a prototype implementation of Ceph, a distributed file system based on our research. The metadata server (MDS) is based on Dynamic Subtree Partitioning, an architecture that adaptively distributes metadata across a cluster based on the current workload. Intelligent OSDs manage data replication, failure detection, and data migration during failure recovery or system expansion. Data is stored by each OSD using EBOFS, an object file system based on prior experience with OBFS. Data is distributed using CRUSH, a hash-like distribution function that allows any party to calculate (instead of looking up) the location of data. CRUSH is designed to cope with device failure and cluster expansion, while separating object replicas across failure domains for improved data safety.

The Ceph source code is available at SourceForge.

If the data stored in large-scale storage systems is sensitive or confidential, security measures must be deployed to protect the data. We have designed and implemented Horus, a system that offers fine-grained encryption-based security for large-scale storage. Horus encrypts large datasets using keyed hash trees (KHT) to generate different keys for each region of the dataset, providing fine-grained security. Performance evaluation shows that our prototype’s key distribution is highly scalable and robust.

Publications

2013

• Yan Li, Nakul Dhotre, Yasuhiro Ohara, Thomas Kroeger, Ethan L. Miller, Darrell D. E. Long, Horus: Fine-Grained Encryption-Based Security for Large-Scale Storage, Proceedings of the 11th Conference on File and Storage Systems (FAST 2013), February 2013.

2012

• Aleatha Parker-Wood, Brian Madden, Michael McThrow, Darrell D. E. Long, Examining Extended and Scientific Metadata for Scalable Index Designs, Technical Report UCSC-SSRC-12-07, December 2012.
• Yulai Xie, Kiran-Kumar Muniswamy-Reddy, Dan Feng, Yan Li, Darrell D. E. Long, Zhipeng Tan, Lei Chen, A Hybrid Approach for Efficient Provenance Storage, The 21st ACM Conference on Information and Knowledge Management (CIKM), October 2012.
• Joel Frank, Ethan L. Miller, Ian Adams, Daniel Rosenthal, Evolutionary Trends in a Supercomputing Tertiary Storage Environment, Proceedings of the 20th IEEE International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS 2012), August 2012.

2011

• Alberto Miranda, Sascha Effert, Yangwook Kang, Ethan L. Miller, Andre Brinkmann, Toni Cortes, Reliable and Randomized Data Distribution Strategies for Large Scale Storage Systems, Proceedings of HiPC 2011, December 2011.
• Ranjana Rajendran, Ethan L. Miller, Darrell D. E. Long, Horus: Fine-Grained Encryption-Based Security for High Performance Petascale Storage, Proceedings of the 6th Parallel Data Storage Workshop (PDSW '11), November 2011.
• Christina Strong, Stephanie Jones, Aleatha Parker-Wood, Alexandra Holloway, Darrell D. E. Long, Los Alamos National Laboratory Interviews, Technical Report UCSC-SSRC-11-06, September 2011.
• Alex Nelson, George Dinolt, Bret Michael, Man-Tak Shing, A security and usability perspective of cloud file systems, 6th IEEE International Conference on System of Systems Engineering (SoSE), June 2011.
• Yulai Xie, Kiran-Kumar Muniswamy-Reddy, Darrell D. E. Long, Ahmed Amer, Dan Feng, Zhipeng Tan, Compressing Provenance Graphs, 3rd USENIX Workshop on the Theory and Practice of Provenance, June 2011.
• Yulai Xie, Kiran-Kumar Muniswamy-Reddy, Dan Feng, Darrell D. E. Long, Yangwook Kang, Zhongying Niu, Zhipeng Tan, Design and Evaluation of Oasis: An Active Storage Framework based on T10 OSD Standard, Proceedings of the 27th IEEE Symposium on Massive Storage Systems and Technologies (MSST 2011), May 2011.

2010

• Avani Wildani, Ethan L. Miller, Semantic Data Placement for Power Management in Archival Storage, Proceedings of the 5th International Workshop on Petascale Data Storage (PDSW10), held in conjunction with SC2010, November 2010.
• Ignacio Corderi, Thomas Schwarz, Ahmed Amer, Darrell D. E. Long, Jehan-François Pâris, Self-Adjusting Two-Failure Tolerant Disk Arrays, Proceedings of the 5th International Workshop on Petascale Data Storage (PDSW10), held in conjunction with SC2010, November 2010.
• Aleatha Parker-Wood, Christina Strong, Ethan L. Miller, Darrell D. E. Long, Security Aware Partitioning for Efficient File System Search, 26th IEEE Symposium on Massive Storage Systems and Technologies: Research Track (MSST 2010), May 2010.

2009

• Andrew Leung, Organizing, Indexing, and Searching Large-Scale File Systems, Technical Report UCSC-SSRC-09-09, December 2009.
• Andrew Leung, Ian Adams, Ethan L. Miller, Magellan: A Searchable Metadata Architecture for Large-Scale File Systems, Technical Report UCSC-SSRC-09-07, November 2009.
• Andrew Leung, Aleatha Parker-Wood, Ethan L. Miller, Copernicus: A Scalable, High-Performance Semantic File System, Technical Report UCSC-SSRC-09-06, October 2009.
• Ian Adams, Darrell D. E. Long, Ethan L. Miller, Shankar Pasupathy, Mark W. Storer, Maximizing Efficiency By Trading Storage for Computation, Proceedings of the Workshop on Hot Topics in Cloud Computing (HotCloud ’09), June 2009.
• Andrew Leung, Minglong Shao, Timothy Bisson, Shankar Pasupathy, Ethan L. Miller, Spyglass: Metadata Search for Large-Scale Storage Systems, ;login: — The USENIX Magazine 34(3), June 2009.
• Andrew Leung, Minglong Shao, Timothy Bisson, Shankar Pasupathy, Ethan L. Miller, Spyglass: Fast, Scalable Metadata Search for Large-Scale Storage Systems, Proceedings of the 7th USENIX Conference on File and Storage Technologies (FAST '09), February 2009.

2008

• Andrew Leung, Ethan L. Miller, Scalable Full-Text Search for Petascale File Systems, Proceedings of the 2008 Petascale Data Storage Workshop (PDSW 08), November 2008.
• Andrew Leung, Minglong Shao, Timothy Bisson, Shankar Pasupathy, Ethan L. Miller, High-Performance Metadata Indexing and Search in Petascale Data Storage Systems, Proceedings of the SciDAC 2008 Conference, July 2008.
• Andrew Leung, Shankar Pasupathy, Garth Goodson, Ethan L. Miller, Measurement and Analysis of Large-Scale Network File System Workloads, Proceedings of the 2008 USENIX Technical Conference, June 2008.
• Andrew Leung, Minglong Shao, Timothy Bisson, Shankar Pasupathy, Ethan L. Miller, Spyglass: Fast, Scalable Metadata Search for Large-Scale Storage Systems, Technical Report UCSC-SSRC-08-01, May 2008.

2007

• Andrew Leung, Ethan L. Miller, Stephanie Jones, Scalable Security for Petascale Parallel File Systems, Proceedings of SC '07, November 2007.
• Sage Weil, Andrew Leung, Scott A. Brandt, Carlos Maltzahn, RADOS: A Fast, Scalable, and Reliable Storage Service for Petabyte-scale Storage Clusters, Proceedings of the ACM Petascale Data Storage Workshop 2007 (PDSW 07), November 2007.
• Kevin Greenan, Ethan L. Miller, Thomas Schwarz, Darrell D. E. Long, Disaster Recovery Codes: Increasing Reliability with Large-Stripe Error Correction Codes, Proceedings of the 3rd International Workshop on Storage Security and Survivability (StorageSS 2007), held in conjunction with the 14th ACM Conference on Computer and Communications Security (CCS 2007), October 2007.
• Kristal Pollack, Darrell D. E. Long, Richard Golding, Ralph Becker-Szendy, Benjamin C. Reed, Quota Enforcement for High-Performance Distributed Storage Systems, Proceedings of the 24th IEEE Conference on Mass Storage Systems and Technologies (MSST 2007), September 2007, pages 72-84.
• Andrew Leung, Eric Lalonde, Jacob Telleen, James Davis, Carlos Maltzahn, Using Comprehensive Analysis for Performance Debugging in Distributed Storage Systems, Proceedings of the 24th IEEE Conference on Mass Storage Systems and Technologies (MSST 2007), September 2007, pages 281-286.
• Joel Wu, Scott A. Brandt, Providing Quality of Service Support in Object-Based File System, Proceedings of the 24th IEEE Conference on Mass Storage Systems and Technologies (MSST 2007), September 2007.
• Max Mehech, The Impact of Failures on Large Distributed Storage Systems, Technical Report UCSC-SSRC-07-10, August 2007.
• Andrew Leung, Scalable Security for High Performance, Petascale Storage, Technical Report UCSC-SSRC-07-07, June 2007.
• Qin Xin, Understanding and Coping with Failures in Large-Scale Storage Systems, Technical Report UCSC-SSRC-07-06, May 2007.
• Andrew Leung, Eric Lalonde, Jacob Telleen, James Davis, Carlos Maltzahn, Using Comprehensive Analysis for Performance Debugging in Distributed Storage Systems, Technical Report UCSC-SSRC-07-05, May 2007. Later version published in MSST 2007.
• Joel Wu, Bo Hong, Scott A. Brandt, Ensuring Performance in Activity-Based File Relocation, Proceedings of the International Performance Conference on Computers and Communication (IPCCC '07), April 2007, pages 75-84.

2006

• Feng Wang, Storage Management in Large Distributed Object-Based Storage Systems, Ph.D. thesis, University of California, Santa Cruz, December 2006.
• Sage Weil, Scott A. Brandt, Ethan L. Miller, Carlos Maltzahn, CRUSH: Controlled, Scalable, Decentralized Placement of Replicated Data, Proceedings of SC '06, November 2006.
• Sage Weil, Scott A. Brandt, Ethan L. Miller, Darrell D. E. Long, Carlos Maltzahn, Ceph: A Scalable, High-Performance Distributed File System, Proceedings of the 7th Conference on Operating Systems Design and Implementation (OSDI '06), November 2006.
• Jehan-François Pâris, Darrell D. E. Long, Using Device Diversity to Protect Data against Batch-Correlated Disk Failures, Proceedings of the 2nd ACM Workshop on Storage Security and Survivability (StorageSS 2006), October 2006.
• Andrew Leung, Ethan L. Miller, Scalable Security for Large, High Performance Storage Systems, Proceedings of the 2nd ACM Workshop on Storage Security and Survivability (StorageSS 2006), October 2006.
• Joel Wu, Scott A. Brandt, The Design and Implementation of AQuA: an Adaptive Quality of Service Aware Object-Based Storage Device, Proceedings of the 23rd IEEE / 14th NASA Goddard Conference on Mass Storage Systems and Technologies, May 2006, pages 209-218.
• Sage Weil, Feng Wang, Qin Xin, Scott A. Brandt, Ethan L. Miller, Darrell D. E. Long, Carlos Maltzahn, Ceph: A Scalable Object-Based Storage System, Technical Report UCSC-SSRC-06-01, March 2006.

2005

• Christopher Olson, Ethan L. Miller, Secure Capabilities for a Petabyte-Scale Object-Based Distributed File System, Proceedings of the 2005 ACM Workshop on Storage Security and Survivability (StorageSS 2005), November 2005. Won Best Full Paper award.
• Qin Xin, Thomas Schwarz, Ethan L. Miller, Disk Infant Mortality in Large Storage Systems, Proceedings of the 13th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS '05), September 2005.
• Joel Wu, Scott A. Brandt, Hierarchical Disk Sharing for Multimedia Systems and Servers, Proceedings of the 15th ACM International Workshop on Network and Operating Systems Support for Digital Audio and Video (NOSSDAV 2005), June 2005, pages 189-194.
• Qin Xin, Ethan L. Miller, Thomas Schwarz, Darrell D. E. Long, Impact of Failure on Interconnection Networks in Large Storage Systems, Proceedings of the 22nd IEEE / 13th NASA Goddard Conference on Mass Storage Systems and Technologies, April 2005.
• Kristal Pollack, Scott A. Brandt, Efficient Access Control for Distributed Hierarchical File Systems, IEEE / NASA Goddard Conference on Mass Storage Systems and Technologies (MSST 2005), April 2005.

2004

• Qin Xin, Thomas Schwarz, Ethan L. Miller, Availability in Global Peer-To-Peer Storage Systems, Distributed Data and Structures 6, Proceedings in Informatics, December 2004.
• Sage Weil, Kristal Pollack, Scott A. Brandt, Ethan L. Miller, Dynamic Metadata Management for Petabyte-Scale File Systems, Proceedings of the 2004 ACM/IEEE Conference on Supercomputing (SC '04), November 2004.
• Thomas Schwarz, Qin Xin, Ethan L. Miller, Darrell D. E. Long, Andy Hospodor, Spencer Ng, Disk Scrubbing in Large Archival Storage Systems, Proceedings of the 12th International Symposium on Modeling, Analysis, and Simulation of Computer and Telecommunication Systems (MASCOTS '04), October 2004, pages 409-418. Won Best Paper award.
• Qin Xin, Ethan L. Miller, Thomas Schwarz, Evaluation of Distributed Recovery in Large-Scale Storage Systems, Proceedings of the 13th IEEE International Symposium on High Performance Distributed Computing (HPDC 2004), June 2004, pages 172-181.
• Joel Wu, Scott A. Brandt, Storage Access Support for Soft Real-Time Applications, Proceedings of the 10th IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS 2004), May 2004.
• Sage Weil, Scott A. Brandt, Ethan L. Miller, Kristal Pollack, Intelligent Metadata Management for a Petabyte-Scale File System, 2nd Intelligent Storage Workshop, May 2004.
• Andy Hospodor, Ethan L. Miller, Interconnection Architectures for Petabyte-Scale High-Performance Storage Systems, Proceedings of the 21st IEEE / 12th NASA Goddard Conference on Mass Storage Systems and Technologies, April 2004, pages 273-281.
• Feng Wang, Scott A. Brandt, Ethan L. Miller, Darrell D. E. Long, OBFS: A File System for Object-Based Storage Devices, Proceedings of the 21st IEEE / 12th NASA Goddard Conference on Mass Storage Systems and Technologies, April 2004, pages 283-300.
• Feng Wang, Qin Xin, Bo Hong, Scott A. Brandt, Ethan L. Miller, Darrell D. E. Long, Tyce T. Mclarty, File System Workload Analysis For Large Scientific Computing Applications, NASA/IEEE Conference on Mass Storage Systems and Technologies (MSST 2004), April 2004, pages 139–152.
• R. J. Honicky, Ethan L. Miller, Replication Under Scalable Hashing: A Family of Algorithms for Scalable Decentralized Data Distribution, Proceedings of the 18th International Parallel & Distributed Processing Symposium (IPDPS 2004), April 2004.

2003

• R. J. Honicky, Ethan L. Miller, A Fast Algorithm for Online Placement and Reorganization of Replicated Data, Proceedings of the 17th International Parallel & Distributed Processing Symposium (IPDPS 2003), April 2003.
• Scott A. Brandt, Ethan L. Miller, Darrell D. E. Long, Lan Xue, Efficient Metadata Management in Large Distributed File Systems, Proceedings of the 20th IEEE / 11th NASA Goddard Conference on Mass Storage Systems and Technologies (MSST 2003), April 2003, pages 290-298.
• Qin Xin, Ethan L. Miller, Thomas Schwarz, Darrell D. E. Long, Scott A. Brandt, Witold Litwin, Reliability Mechanisms for Very large Storage Systems, Proceedings of the 20th IEEE / 11th NASA Goddard Conference on Mass Storage Systems and Technologies, April 2003, pages 146-156.

2002

• Ethan L. Miller, Darrell D. E. Long, William E. Freeman, Benjamin C. Reed, Strong Security for Network-Attached Storage, Proceedings of the 2002 Conference on File and Storage Technologies (FAST), January 2002, pages 1-13.

1997

• Ethan L. Miller, Randy Katz, RAMA: An Easy-To-Use, High-Performance Parallel File System, Parallel Computing 23(4), July 1997, pages 419-446.