Memory Centric Storage (MCS) – Flexible Toolbox for Scalable Storage Management

With Memory Centric Storage (MCS), we are developing the European open-source toolbox that combines scalable and domain-specific storage management. Applications access different storage systems via a uniform interface and benefit from zero-copy data transport and task offloading. This reduces data movement and brings computing processes to where the data is generated – for maximum performance.

The ever-increasing computing capacities place high demands on memory and storage systems. These must not only become larger and faster, but also offer more flexible interfaces. MCS meets this challenge by breaking down the traditional separation between heap memory and file storage and enabling shared, intelligent data management.

Background: Why Traditional Memory Architectures Are Reaching Their Limits

Until now, traditional architecture has forced applications to manage two separate storage worlds – volatile heap memory and distributed persistent file storage. Both use different interfaces, which means that data is constantly being transferred back and forth between them. Alternatively, the Linux kernel can integrate file storage into heap memory, but even this solution cannot avoid data transfers.

Both approaches work with outdated interfaces (such as malloc/free and open/read/write/close), which do not take into account any knowledge about actual data access. This means that the system does not know whether a data block is immutable, whether it will only be read once, or whether it will be used exclusively by a resource. The lack of semantics means that memory processes often run inefficiently: applications wait for data instead of processing it.

Architecture: Layers for Maximum Flexibility and Control

MCS has a storage-centric design and can be understood as a generalization of classic storage and file management. It works with data blocks in a distributed network of storage devices and is divided into several layers:

Core Layer – Direct Control over Memory Segments

It manages memory segments and enables direct access to data blocks. It has no knowledge of the content, but focuses on efficiency and performance. The core layer provides scalable functions for controlling memory and transporting data. It can be expanded to include new memory and new transport networks.