Automated search space and search strategy selection for AutoML

Highlights：

• Search space and search strategy selection automation (vanilla LArST). Given dataset and search budget, we consider a combinatorial optimization problem: choosing both search space and search strategies for the dataset automatically. The motivation comes from the observation that in many existing works search space and search strategies are often coupled to each other, and search space is manually predefined followed with an search strategy. In contrast, we use the probing model based metafeatures and a decision tree based method to select the appropriate search space and its search strategy.

• Search space auto-selection for platform-aware multi-objective AutoML. Our experimental results show that search space design is sensitive to platforms and objectives. First of all, the operations perform differently on various platforms (see Table 5). Secondly, as the work [1] mentions, earlier stages of CNN architectures usually have higher impact on inference latency than later stages, so the cell based structure [2] can be refined for low latency. Our method can be used for choosing proper search space automatically for multi-objective AutoML on different platforms such as GPU, x86 CPU, Power series CPU, etc.

• Cell transferability via search space learning. The work [3] shows that the best basic cell of architecture on CIFAR-10 can be transferred to ImageNet. However, the search space is based on human experience and diverse (w.r.t. number of cells, number and position of pooling layers etc.). For better cross-dataset transferability, we develop a direct link from search space to basic cell in our layered architecture search tree (LArST) method via supervised learning.

• Empirical study. Our approach shows notable performance improvement (see Tables 2 and 3). Our joint automation framework can also be easily plugged by existing architecture search strategies and search space representations in an out-of-box manner.