GPI-Space: Memory Driven Computing and Big Data

The processing and analysis of extremely huge amounts of data not only creates new business models for industry, but also opens up completely new opportunities for research and development.

With classical methods and tools, the complex tasks of programming parallel software applications for modern high-performance solutions can only be mastered insufficiently. We have developed the programming platform GPI-Space as an effective solution. Its unique feature separates the coordination of the data from the data computation and ensures efficient execution of the applications even on future hardware. The programmers of a software application concentrate on the design of the domain-specific calculation target – without having to include the organization of the execution environment.

GPI-Space
© Fraunhofer ITWM

More Than Just a Distributed Runtime System

The platform supports the creation of the software models with graphical editors, a functional language and a comprehensive set of common workflows. The workflows used to control GPI-Space are based on Petri nets, a The workflows used to control GPI-Space are based on Petri nets, a suitable formalism specifically for describing dynamic, distributed and parallel processes.

suitable formalism specifically for describing dynamic, distributed and parallel processes

The distributed runtime environment leverage the inhouse developed Global Address Space Programming Interface (GPI) - Fraunhofer's scalable, fast and efficient PGAS layer. It provides a virtual environment in which compute nodes are able to independently access the memory of other compute nodes in a non-blocking, asynchronous way.

 

High Performance Data Analytics (HPDA)

All components of GPI-Space are based on technologies and developments from the high-performance computing domain. This ensures the highest possible performance and flexibility. The virtual memory is independent of specific applications and allows their simple and direct coupling. Data stored in virtual memory can be accessed by arbitrary applications and hardware ressources like CPUs, NVIDIA GPUs or Intel XEON Phi.

The GPI-Space operating environment is not only fault tolerant, but also fully interactive and allows you to change the size and structure of the machine used during the runtime, or to use the optimal topology for the different phases. That is another big advantage over existing solutions. The runtime system enables the coupling of legacy applications and kernel modules to form new application workflows.

The Three Main Components of GPI-Space

  • virtual memory layer
  • distributed runtime
  • workflow engine

are seamlessly integrated, but may also be used independently

Übersicht der Architektur von GPI-Space
© Fraunhofer ITWM
Übersicht der Architektur von GPI-Space

Applications

The GPI-Space system was initially developed in close cooperation with our customers in the oil and gas industry. It is used for interactive seismic imaging and Reverse Time Migration (RTM). But also the solution is in use in the area financial services and Machine Learning. However, the different fundamental components are independent of a specific application or domain and can be applied to enhance any type of large scale distributed processing workflow. Further information to our application examples:

 

Machine Learning

High Performance Deep Learning Framework

The project provides an easy access to current and future high-performance computing systems.

Seismic

Reverse Time Migration

The method of reverse time migration (RTM) stands out by high imaging quality even in case of high geological complexity.