Storage Systems Seminar: Trace Analysis of Large-Scale Storage Systems

Storage systems for scientific and industrial workloads involve working with petabytes of data. These systems often have complex hierarchies or pools through which data movement takes place for analysis. It is important to understand the behavior of such a system, including migration, replication, per user read/write patterns, per task usage, as well as trends over longer periods of time, such as a month or a year. Such analysis will help us identify the system usage, reduce redundant reads and writes, and design the system better to reduce movement of data, and at the same time, utilize the available storage space more efficiently. Trace analysis also allows us to identify and differentiate between recurring tasks, related tasks, and schedule them with necessary priorities, to improve throughput and reduce latency. We present our uniform trace analysis framework, which is designed to take in traces across multiple large scale systems, and compare the behavior of the archives over time. The system can take in data across multiple formats, and present a 1:1 comparison of attributes as well as usage across systems. We present an analysis of the CERN EOS filesystem traces, traces gathered from CERN's production system over a year. The analysis is across 2.49 billion unique events that happened on the EOS filesystem. We plan to integrate this trace analysis with traces from other scientific labs and archives, to compare and contrast behavior of large scale storage systems.

Monday, April 30, 2018 at 12:30 PM

E2 599

SSRC Contact:
Purandare, Devashish (Dev)

Last modified 9 Apr 2018