CORTX is S3 compatible object storage designed for great efficiency, massive capacity, and high HDD-utilization. This is a 100% open source project with engineering effort provided by Seagate. Most of the project is licensed under the Apache 2.0 License and the rest is under AGPLv3.
CORTX aims to be uniquely optimized for mass capacity storage devices, work with any processor, highly flexible (works with HDD, SDD and NVM), and massively scalable, up to a billion billion billion billion billion exabytes (2^206) and 1.3 billion billion billion billion (2^120) objects with unlimited object sizes.
CORTX use cases include data mining, machine learning, archival storage, backup, and autonomous driving. Sage2, Maestro, and EsiWACE2 are the current R&D projects based on CORTX.
- Sage2 builds a CORTX cluster based for extreme scale computing/HPC and AI moving forward from the SAGE project using the SAGE prototype. The project consists of 9 partners led by Seagate and is funded by the European Commission’s H2020 program (Grant Agreement number: 800999).
Sage2 will demonstrate a multi tier storage system (distributed NVRAM, SSD, HDD), “SAGE”, capable of handling data intensive workloads. Seagate is providing the components of CORTX, primarily Motr, lib-motr and Hare, and some of the storage enclosures (SAS & SATA HDD tiers). Experimental pieces of Motr are developed by Seagate (eg: QoS, Function Shipping etc) to meet the needs of HPC and AI use cases. KTH adopts the CORTX object storage with an I/O programming model that is suitable for large scale HPC and emerging ML/AI applications.To demonstrate the features of CORTX, pilot applications such as iPIC3Dand StreamBrain are adapted to use CORTX object storage and function shipping to accelerate post-processing workloads. Kitware visualization utilities for applications run on SAGE. UKAEA adapts the novel storage system for a unique HPC application involving parallel in time. The combination of the tiered storage with parallel in time aims to demonstrate application portability to the Exascale era. UKAEA also applies to areas such as global memory abstraction and is closely working with other CORTX partners to develop and use tools for studying performance alongside data management and analysis.
- Maestro R&D project is building a data orchestration software layer that intelligently handles data movement in the I/O stack and has CORTX Motr as one of its primary backends. Apart from Seagate – CEA, Juelich, ETH, Appentra, ECMWF and HPE are the key players and the project is led by Juelich (administration) + HPE (technical).
- The EsiWACE2 project consists of 20 EU partners from the weather and climate community led by DKRZ (Germany). EsiWACE2 looks at weather and climate applications, leveraging CORTX Motr as one of the backends.
Learn more about CORTX at: https://www.seagate.com/products/storage/object-storage-software/
