SNAP: Shaping neural architectures progressively via information density criterion

Highlights：

• The proposed method is efficient due to the greedy strategy. And we give a detailed proof that the greedy strategy is reasonable in theory.

• We propose the information density criterion to induce the progressive pro- cess, which is both task-aware and device-aware.

• The experimental results show that the proposed method can significant- ly improve the performance of the given architecture. And it can achieve comparable or even better performance compared with the search based ar- chitecture auto-generated methods in no need of tremendous computation resources.

摘要

•We propose a progressive method SNAP to shape a given neural architecture to a more reasonable one progressively, which is inspired by the streamline of water droplet driven by the air resistance progressively.•The proposed method is efficient due to the greedy strategy. And we give a detailed proof that the greedy strategy is reasonable in theory.•We propose the information density criterion to induce the progressive pro- cess, which is both task-aware and device-aware.•The experimental results show that the proposed method can significant- ly improve the performance of the given architecture. And it can achieve comparable or even better performance compared with the search based ar- chitecture auto-generated methods in no need of tremendous computation resources.

论文评审过程：Received 12 January 2020, Revised 14 December 2020, Accepted 23 February 2021, Available online 17 March 2021, Version of Record 26 March 2021.