Digital Talk

Abstract – Neural Architecture Search

Finding good Deep Neural Networks (DNN) for an application can turn out to be a quite tedious task, as various design and training parameter combinations have to be considered, additionally to actually training the model on the data. This is a bilevel optimization problem, where the DNN design parameters are often called hyperparameters to distinguish them from the model parameters, which are adjusted in the training of the model. Fortunately, the search for hyperparameters can be automatized. The various methods to automatize this process are known as 'Auto-ML', which differ in the actual subject of the search, for example only looking at the training hyperparameters for a fixed DNN. Neural Architecture Search (NAS) considers the DNN itself, i.e. the structure of the network, as the search subject. The challenge in NAS is the extremely high number of hyperparameters, since every single DNN layer introduces new hyperparameters, so the total number of choices grows with the depth of the DNN. Additionally, the new DNNs must be trained in order evaluate their performance, which is quite costly.

This presentation introduces the main components of the NAS and shows how the various challenges in NAS can be approached, giving examples from state of the art methods. An own implementation called NASE (Neural Architecture Search Engine) is showcased at the end.