Survey on Deep Neural Architecture Search

doi:10.3969/j.issn.1671-1122.2023.09.006

Abstract

Abstract:

In recent years, deep neural networks have been applied to image recognition, speech recognition, target detection, machine translation and other aspects of life. Greatly accelerating the performance evolution and flexibility improvement of the network. But these networks often have complex structures, require personnel with a large amount of professional knowledge, and require a significant amount of time to adjust parameters to suit specific environments. The efficiency of adjusting parameters using conventional manual methods is too low and errors occur frequently. Therefore, research on neural network architecture search has also been put on the agenda. In order to provide readers with a comprehensive understanding of the research progress of neural network architecture search, the article introduced and evaluated existing relevant algorithms, and proposed ideas for the future development of neural network architecture search.

Key words: machine learning, automation, deep learning, convolutional neural network, artificial intelligence

CLC Number:

TP309

XUE Yu, ZHANG Yixuan. Survey on Deep Neural Architecture Search[J]. Netinfo Security, 2023, 23(9): 58-74.

Figures/Tables 9

References 68

[1]	ZOPH B, LE Q V. Neural Architecture Search with Reinforcement Learning[EB/OL]. (2016-11-05)[2023-04-20]. https://doi.org/10.48550/arXiv.1611.01578.
[2]	SIMONYAN K, ZISSERMAN A. Very Deep Convolutional Networks for Large-Scale Image Recognition[EB/OL]. (2014-09-04)[2023-04-20]. https://doi.org/10.48550/arXiv.1409.1556.
[3]	HE Kaiming, ZHANG Xiangyu, REN Shaoqing, et al. Deep Residual Learning for Image Recognition[C]// IEEE. 2016 IEEE Conference on Computer Vision and Pattern Recognition(CVPR). New York: IEEE, 2016: 770-778.
[4]	HUANG Guo, LIU Zhuang, VAN D M L, et al. Densely Connected Convolutional Networks[C]// IEEE. IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2017: 4700-4708.
[5]	HOWARD A G, ZHU Menglong, CHEN Bo, et al. Mobilenets: Efficient Convolutional Neural Networks for Mobile Vision Applications[EB/OL]. (2017-04-17)[2023-04-20]. https://doi.org/10.48550/arXiv.1704.0486.
[6]	CHU Xiangxiang, ZHANG Bo, XU Ruijun. FairNAS: Rethinking Evaluation Fairness of Weight Sharing Neural Architecture Search[C]// IEEE. IEEE/CVF International Conference on Computer Vision. New York: IEEE, 2021: 12239-12248.
[7]	TIAN Shuo, HU Kai, GUO Shasha, et al. Evolutionary Recurrent Neural Architecture Search[J]. IEEE Embedded Systems Letters, 2020, 13(3): 110-113. doi: 10.1109/LES.2020.3005753 URL
[8]	BENDER G, KINDERMANS P J, ZOPH B, et al. Understanding and Simplifying One-Shot Architecture Search[C]// PMLR. International Conference On Machine Learning. New York: PMLR, 2018: 550-559.
[9]	LU Zhichao, WHALEN I, BODDETI V, et al. NSGA-Net: Neural Architecture Search Using Multi-Objective Genetic Algorithm[C]// ACM. Genetic and Evolutionary Computation Conference. New York: ACM, 2019: 419-427.
[10]	MICHEL G, ALAOUI M A, LEBOIS A, et al. Dvolver: Efficient Pareto-Optimal Neural Network Architecture Search[EB/OL]. (2019-02-05)[2023-04-20]. https://doi.org/10.48550/arXiv.1902.01654.
[11]	YANG Zhaohui, WANG Yunhe, CHEN Xinghao, et al. CARS: Continuous Evolution for Efficient Neural Architecture Search[C]// IEEE. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2020: 1829-1838.
[12]	WANG E K, XU S P, CHEN C M, et al. Neural-Architecture-Search-Based Multiobjective Cognitive Automation System[J]. IEEE Systems Journal, 2020, 15(2): 2918-2925. doi: 10.1109/JSYST.2020.3002428 URL
[13]	WANG Zhehui, LUO Tao, LI Miqing, et al. Evolutionary Multi-Objective Model Compression for Deep Neural Networks[J]. IEEE Computational Intelligence Magazine, 2021, 16(3): 10-21.
[14]	CHU Xiangxiang, ZHANG Bo, XU Ruijun. Multi-Objective Reinforced Evolution in Mobile Neural Architecture Search[C]// Springer. Computer Vision-ECCV 2020 Workshops. Berlin: Springer, 2021: 99-113.
[15]	CAI Han, CHEN Tianyao, ZHANG Weinan, et al. Efficient Architecture Search by Network Transformation[C]// AAAI. AAAI Conference on Artificial Intelligence. Menlo: AAAI, 2018: 1-8.
[16]	SUGANUMA M, SHIRAKAWA S, NAGAO T. A Genetic Programming Approach to Designing Convolutional Neural Network Architectures[C]// CoNLL. Genetic and Evolutionary Computation Conference. New York: CoNLL, 2017: 497-504.
[17]	MENDOZA H, KLEIN A, FEURER M, et al. Towards Automatically-Tuned Neural Networks[C]// PMLR. Workshop on Automatic Machine Learning. New York: PMLR, 2016: 58-65.
[18]	BROCK A, LIM T, RITCHIE J M, et al. SMASH: One-Shot Model Architecture Search through Hypernetworks[EB/OL]. (2017-08-17)[2023-04-20]. https://doi.org/10.48550/arXiv.1708.05344.
[19]	CAI Han, YANG Jiacheng, ZHANG Weinan, et al. Path-Level Network Transformation for Efficient Architecture Search[C]// PMLR. International Conference on Machine Learning. New York: PMLR, 2018: 678-687.
[20]	ELSKEN T, METZEN J H, HUTTER F. Simple and Efficient Architecture Search for Convolutional Neural Networks[EB/OL]. (2017-11-13)[2023-04-20]. https://doi.org/10.48550/arXiv.1711.04528.
[21]	ELSKEN T, METZEN J H, HUTTER F. Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution[EB/OL]. (2018-04-24)[2023-04-20]. https://doi.org/10.48550/arXiv.1804.09081.
[22]	ZOPH B, VASUDEVAN V, SHLENS J, et al. Learning Transferable Architectures for Scalable Image Recognition[EB/OL]. (2018-04-11)[2023-04-20]. https://doi.org/10.48550/arXiv.1707.07012.
[23]	REAL E, AGGARWAL A, HUANG Yanping, et al. Regularized Evolution for Image Classifier Architecture Search[EB/OL]. (2018-02-05)[2023-04-20]. https://doi.org/10.48550/arXiv.1802.01548.
[24]	LIU Chenxi, ZOPH B, SHLENS J, et al. Progressive Neural Architecture Search[EB/OL]. (2017-12-02)[2023-04-20]. https://doi.org/10.48550/arXiv.1712.00559.
[25]	GUO Ronghao, LIN Chen, LI Chuming, et al. Powering One-Shot Topological NAS with Stabilized Share-Parameter Proxy[C]// Springer. Computer Vision-ECCV 2020. Berlin: Springer, 2020: 625-641.
[26]	HU Yiming, WANG Xingang, GU Qingyi. PWSNAS: Powering Weight Sharing NAS with General Search Space Shrinking Framework[J]. IEEE Transactions on Neural Networks and Learning Systems, 2022, 22: 1-14.
[27]	SZEGEDY C, VANHOUCKE V, IOFFE S, et al. Rethinking the Inception Architecture for Computer Vision[C]// IEEE. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). New York: IEEE, 2016: 2818-2826.
[28]	LIU Hanxiao, SIMONYAN K, YANG Yiming. Darts: Differentiable Architecture Search[EB/OL]. (2018-06-24)[2023-04-20]. https://doi.org/10.48550/arXiv.1806.09055.
[29]	PHAM H, GUAN M, ZOPH B, et al. Efficient Neural Architecture Search via Parameters Sharing[C]// PMLR. International Conference on MACHINE Learning. New York: PMLR, 2018: 4095-4104.
[30]	LIU Hanxiao, SIMONYAN K, VINYALS O, et al. Hierarchical Representations for Efficient Architecture Search[EB/OL]. (2017-11-01)[2023-04-20]. https://doi.org/10.48550/arXiv.1711.00436.
[31]	FLOREANO D, DURR P, MATTIUSSI C. Neuroevolution: From Architectures to Learning[J]. Evolutionary Intelligence, 2008, 1(1): 47-62. doi: 10.1007/s12065-007-0002-4 URL
[32]	STANLEY K O, MIIKKULAINEN R. Evolving Neural Networks through Augmenting Topologies[J]. Evolutionary Computation, 2002, 10(2): 99-127. pmid: 12180173
[33]	STANLEY K, D'AMBROSIO D, GAUCI J. A Hypercube-Based Encoding for Evolving Large-Scale Neural Networks[J]. Artificial Life, 2009, 15(2): 185-212. doi: 10.1162/artl.2009.15.2.15202 pmid: 19199382
[34]	XIE Lingxi, YUILLE A. Genetic CNN[C]// IEEE. 2017 IEEE International Conference on Computer Vision (ICCV). New York: IEEE, 2017: 2980-2988.
[35]	SUN Yanan, XUE Bing, ZHANG Mengjie, et al. Completely Automated CNN Architecture Design Based on Blocks[J]. IEEE Transactions on Neural Networks and Learning Systems, 2019, 99: 1-13.
[36]	CHU Xiangxiang, ZHOU Tianbao, ZHANG Bo, et al. Fair Darts: Eliminating Unfair Advantages in Differentiable Architecture Search[C]// Springer. European Conference on Computer Vision. Berlin:Springer, 2020: 465-480.
[37]	BAKER B, GUPTA O, NAIK N, et al. Designing Neural Network Architectures Using Reinforcement Learning[EB/OL]. (2016-11-07)[2023-04-20]. https://doi.org/10.48550/arXiv.1611.02167.
[38]	ZHONG Zhao, YAN Junjie, WU Wei, et al. Practical Block-Wise Neural Network Architecture Generation[C]// IEEE. IEEE Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2018: 2423-2432.
[39]	CHEN Xin, XIE Lingxi, WU Jun, et al. Progressive Differentiable Architecture Search: Bridging the Depth Gap Between Search and Evaluation[C]// IEEE. IEEE/CVF International Conference on Computer Vision. New York: IEEE, 2019: 1294-1303.
[40]	XU Yuhui, XIE Lingxi, ZHANG Xiaopeng, et al. PC-DARTS: Partial Channel Connections for Memory-Efficient Architecture Search[EB/OL]. (2020-07-12)[2023-04-20]. https://doi.org/10.48550/arXiv.1907.05737.
[41]	CAI Han, ZHU Ligeng, HAN Song. ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware[EB/OL]. (2018-12-02)[2023-04-20]. https://doi.org/10.48550/arXiv.1812.0033.
[42]	LUO Renqian, TIAN Fei, QIN Tao, et al. Neural Architecture Optimization[C]// ACM. Advances in Neural Information Processing Systems. New York: ACM, 2018: 7816-7827.
[43]	WU Bichen, DAI Xiaoliang, ZHANG Peizhao, et al. FBNET: Hardware-Aware Efficient Convnet Design via Differentiable Neural Architecture Search[C]// IEEE. IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2019: 10734-10742.
[44]	LIU Chenxi, CHEN L C, SCHROFF F, et al. Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation[C]// IEEE. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New York: IEEE, 2019: 1-12.
[45]	LIU C, CHEN L C, SCHROFF F, et al. Auto-Deeplab: Hierarchical Neural Architecture Search for Semantic Image Segmentation[C]// IEEE. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2019: 82-92.
[46]	YANG Zhaohui, WANG Yunhe, CHEN Xinghao, et al. HourNAS: Extremely Fast Neural Architecture Search through an Hourglass Lens[C]// IEEE. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2021: 10896-10906.
[47]	TAN Hao, CHENG Ran, HUANG Shihua, et al. RelativeNAS: Relative Neural Architecture Search via Slow-Fast Learning[J]. IEEE Transactions on Neural Networks and Learning Systems, 2021, 34(1): 475-489. doi: 10.1109/TNNLS.2021.3096658 URL
[48]	YANG Qiang, LIU Yang, CHEN Tianjian, et al. Federated Machine Learning: Concept and Applications[J]. ACM Transactions on Intelligent Systems and Technology (TIST), 2019, 10(2): 1-19.
[49]	ZHU Hangyu, ZHANG Haoyu, JIN Yaochu. From Federated Learning to Federated Neural Architecture Search: A Survey[J]. Complex and Intelligent Systems, 2021, 7: 639-657. doi: 10.1007/s40747-020-00247-z
[50]	CAI Han, GAN Chuan, WANG Tianzhe, et al. Once-for-All: Train One Network and Specialize It for Efficient Deployment[EB/OL]. (2019-08-26)[2023-04-20]. https://doi.org/10.48550/arXiv.1908.09791.
[51]	BAKER B, GUPTA O, RASKAR R, et al. Accelerating Neural Architecture Search Using Performance Prediction[EB/OL]. (2017-05-30)[2023-04-20]. https://doi.org/10.48550/arXiv.1705.10823.
[52]	KLEIN A, FALKNER S, SPRINGENBERG J T, et al. Learning Curve Prediction with Bayesian Neural Networks[C]// ICLR. International Conference on Learning Representations. New York: ICLR, 2017: 1-16.
[53]	HASSANTABAR S, DAI Xiaoliang, JHA N K. CURIOUS: Efficient Neural Architecture Search Based on a Performance Predictor and Evolutionary Search[J]. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2022, 41(11): 4975-4990. doi: 10.1109/TCAD.2022.3148202 URL
[54]	LU Zhichao, DEB K, GOODMAN E, et al. NSGANETV2: Evolutionary Multi-Objective Surrogate-Assisted Neural Architecture Search[C]// Springer. Computer Vision-ECCV 2020. Berlin: Springer, 2020: 35-51.
[55]	WEI Chen, NIU Chuang, TANG Yiping, et al. Npenas: Neural Predictor Guided Evolution for Neural Architecture Search[J]. IEEE Transactions on Neural Networks and Learning Systems, 2022, 22: 1-15.
[56]	ZHAO Yiyang, WANG Linan, TIAN Yuandong, et al. Few-Shot Neural Architecture Search[C]// PMLR. International Conference on Machine Learning. New York: PMLR, 2021: 12707-12718.
[57]	GUO Zichao, ZHANG Xiangyu, MU Haoyuan, et al. Single Path One-Shot Neural Architecture Search with Uniform Sampling[C]// Springer. Computer Vision-ECCV 2020: 16th European Conference. Berlin: Springer, 2020: 544-560.
[58]	YOU Shan, HUANG Tao, YANG Mingmin, et al. GreedyNAS: Towards Fast One-Shot NAS with Greedy Supernet[C]// IEEE. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2020: 1999-2008.
[59]	SHEN Zan, QIAN Jiang, ZHUANG Bojin, et al. BS-NAS: Broadening-and-Shrinking One-Shot NAS with Searchable Numbers of Channels[EB/OL]. (2020-03-22)[2023-04-20]. https://doi.org/10.48550/arXiv.2003.09821.
[60]	ZHENG Xiawu, JI Rongrong, WANG Qiang, et al. Rethinking Performance Estimation in Neural Architecture Search[C]// IEEE. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2020: 11353-11362.
[61]	HU Yiming, WANG Xingang, LI Lujun, et al. Improving One-Shot NAS with Shrinking-and-Expanding Supernet[J]. Pattern Recognition: The Journal of the Pattern Recognition Society, 2021, 1: 1-12.
[62]	ZHANG Miao, LI Huiqi, PAN Shuirui, et al. One-Shot Neural Architecture Search: Maximising Diversity to Overcome Catastrophic Forgetting[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 43(9): 2921-2935. doi: 10.1109/TPAMI.2020.3035351 URL
[63]	YING C, KLEIN A, CHRISTIANSEN E, et al. NAS-Bench-101: Towards Reproducible Neural Architecture Search[C]// PMLR. International Conference on Machine Learning. New York: PMLR, 2019: 7105-7114.
[64]	DONG Xuanyi, YANG Yi. NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture Search[EB/OL]. (2020-01-15)[2023-04-20]. https://doi.org/10.48550/arXiv.2001.00326.
[65]	SIEMS J, ZIMMER L, ZELA A, et al. NAS-Bench-301 and the Case for Surrogate Benchmarks for Neural Architecture Search[EB/OL]. (2022-04-14)[2023-04-20]. https://doi.org/10.48550/arXiv.2008.09777.
[66]	KLYUCHNIKOV N, TROFIMOV I, ARTEMOVA E, et al. NAS-Bench-NLP: Neural Architecture Search Benchmark for Natural Language Processing[J]. IEEE Access, 2022, 10: 45736-45747. doi: 10.1109/ACCESS.2022.3169897 URL
[67]	MEHTA Y, WHITE C, ZELA A, et al. NAS-Bench-Suite: NAS Evaluation is (Now) Surprisingly Easy[EB/OL]. (2022-02-11)[2023-04-20]. https://doi.org/10.48550/arXiv.2201.13396.
[68]	TU R, ROBERTS N, KHODAK M, et al. NAS-Bench-360: Benchmarking Neural Architecture Search on Diverse Tasks[J]. Advances in Neural Information Processing Systems, 2022, 35: 12380-12394.