Carme Connects the World of Machine Learning and of HPC Clusters
Machine Learning is a constantly and rapidly growing field. This agility presents data centers with the challenge of providing very different applications to individual users. In doing so, the demands of constantly new software and libraries have never before been met to such an extent on HPC systems. Related to this is the fact that these libraries usually have to be installed via Python environments rather than via the Linux system. It is also not uncommon for different Deep Learning algorithms to have mutually exclusive dependencies. In Carme, we rely on software containers to solve these problems. This facilitates maintenance for administrators and usability for users. In addition, the use of interactive development interfaces is common in Deep Learning and data analytics. With the integration of interactive cluster usage, users get the chance to use tools they are already familiar with on a complex HPC cluster. This facilitates the transition and use of a cluster.
Thus, it is not enough to provide user interfaces and libraries. There must also be smooth integration of these into existing and emerging clusters. An intuitive software environment on the clusters increases usability for all users.