Review of Image Enhancement Based on Generative Adversarial Networks

doi:10.3969/j.issn.1671-1122.2019.05.002

Abstract

Abstract:

In recent years, generative adversarial networks(GAN) has provided new techniques and means for image enhancement. It has more powerful feature learning and expression capabilities than traditional deep learning, and has achieved remarkable success in the field of image enhancement. Firstly, the basic ideas and principles of GAN model are introduced, and the improvement methods, advantages and disadvantages of GAN variants are analyzed. Secondly, the research status of GAN applied to image enhancement is analyzed from the aspects of image quality improvement, image generation, image complementation and other image processing applications. Finally, the GAN model and the problems in image enhancement are summarized and summarized, and the solution and future application of the problem are summarized.

Key words: generative adversarial networks, deep learning, generated model, image enhancement

CLC Number:

TP309

Chunguang MA, Yaoyao GUO, Peng WU, Haibo LIU. Review of Image Enhancement Based on Generative Adversarial Networks[J]. Netinfo Security, 2019, 19(5): 10-12.

Figures/Tables 7

References 67

[1]	GOODFELLOW I J, POUGET-ABADAIE J, MIRZA M, et al.Generative Adversarial Nets[C]//ACM. 27th International Conference on Neural Information Processing Systems, December 8-13, 2014, Montreal, Canada. New York: ACM, 2014: 2672-2680.
[2]	FU Jianming, LI Lin, ZHENG Rui, et al.Survey of Network Attack Detection Based on GAN[J]. Netinfo Security, 2019, 19(2): 1-9.
	傅建明,黎琳,郑锐,等.基于GAN的网络攻击检测研究综述[J]. 信息网络安全,2019,19(2):1-9.
[3]	SONG Lei, MA Chunguang, DUAN Guanghan.Research Progress on Machine Learning Security and Privacy Protection[J]. Chinese Journal of Network and Information Security, 2018, 4(8): 1-11.
	宋蕾,马春光,段广晗. 机器学习安全及隐私保护研究进展.[J]. 网络与信息安全学报,2018,4(8):1-11.
[4]	CRESWELL A, WHITE T, DUMOULIN V, et al.Generative Adversarial Networks: An Overview[J]. IEEE Signal Processing Magazine, 2017, 35(1): 53-65.
[5]	MIRZA M, OSINDERO S. Conditional Generative Adversarial Nets[EB/OL]. , 2018-11-22.
[6]	BENGIO Y, DAUPHIN Y, G M G, et al. Better Mixing via Deep Representations[C]//ACM. 30th International Conference on International Conference on Machine Learning, June 16-21, 2014, Atlanta, GA, USA. New York: ACM 2013: 552-560.
[7]	RADFORD A, METZ L, CHINTALA S. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[EB/OL]. , 2018-11-22.
[8]	DOSOVITSKIY A, TOBIAS SPRINGENBERG J, BROX T.Learning to Generate Chairs With Convolutional Neural Networks[C]//IEEE. 2015 IEEE Conference on Computer Vision and Pattern Recognition, June 7-12, 2015, Boston, MA, USA. New Jersey: IEEE, 2013: 1538-1546.
[9]	HARDT M, RECHT B, SINGER Y. Train Faster, Generalize Better: Stability of Stochastic Gradient Descent[C]//IMLS. 33rd International Conference on International Conference on Machine Learning, June 19-24, New York, USA. New York: PMLR, 2016: 1225-1234.
[10]	LOFFE S, SZEGEDY C.Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift[C]//ACM. 32nd International Conference on Machine Learning, July 6-11, 2015, Lille, France. Cambridge: MIT, 2015: 448-456.
[11]	NAIR V, HINTON G E.Rectified Linear Units Improve Restricted Boltzmann Machines[C]//ACM. 27th International Conference on Machine Learning, June 21-24, Haifa, Israel. New York: 2010: 807-814.
[12]	MAAS A L, HANNUN A Y, NG A Y. Rectifier Nonlinearities Improve Neural Network Acoustic Models[EB/OL]. , 2018-11-22.
[13]	XU Bing, WANG Naiyan, CHEN Tianqi, et al. Empirical Evaluation of Rectified Activations in Convolutional Network[EB/OL]. .
[14]	GOODFELLOW I J, WARDE-FARLEY D, MIRZA M, et al.Maxout networks[C]//IMLS. 30th International Conference on Machine Learning, June 16-21, Atlanta USA. New York: PMLR, 2013: 1311-1319.
[15]	HAN Chongzhao.Multi-source Information Fusion[M]. Beijing: Tsinghua University Press, 2006.
	韩崇昭. 多源信息融合[M]. 北京:清华大学出版社,2006.
[16]	YAN Junhua, CHEN Shaohua, AI Shufang, et al.Improved CAMShift Target Tracking Based on Kalman Predictor[J]. Journal of Chinese Inertial Technology, 2014, 22(4): 536-542.
	闫钧华,陈少华,艾淑芳,等. 基于Kalman 预测器的改进的CAMShift目标跟踪[J]. 中国惯性技术学报,2014,22(4):536-542.
[17]	ZHAO Gaochang, ZHANG Lei, WU Fengbo.Application of Improved Median Filtering Algorithm in Image Denoising[J]. Journal of Applied Optics, 2011, 32(4): 678-682.
	赵高长,张磊,武风波. 改进的中值滤波算法在图像去噪中的应用[J]. 应用光学,2011,32(4):678-682.
[18]	SHENG Zhe.UAV Attitude Control and Autonomous Returning Technology Based on Visual Navigation[D]. Shanghai: Shanghai Jiaotong University, 2015.
	盛哲. 基于视觉导航的无人机位姿控制与自主返航技术[D]. 上海:上海交通大学,2015.
[19]	WANG Xiangpeng.Research on Key Algorithms of Agricultural UAV Visual Navigation under GPS Failure[D]. Qingdao: Qingdao University of Science and Technology, 2014.
	王翔鹏. GPS失效情况下农用无人机视觉导航关键算法研究[D]. 青岛:青岛科技大学,2014.
[20]	ZHANG Han, XU Tao, LI Hongsheng, et al.StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks[C]//IEEE. 16th IEEE International Conference on Computer Vision, October 22-29, 2017, Venice, Italy. New Jersey: IEEE, 2017: 5908-5916.
[21]	CHEN Xi, DUAN Yan, HOUTHOOFT R, et al.InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets[C]//NIPS. 30st International Conference on Neural Information Processing Systems, December 5-10, 2016, Barcelona Spain. New York: Curran Associates, 2016: 2180-2188.
[22]	ARJOVSKY M, CHINTALA S, BOTTOU L. Wasserstein GAN[EB/OL]. , 2018-11-22.
[23]	ZHENG Huabin. Wasserstein GAN[EB/OL]. , 2017-4-2.
	郑华滨.令人拍案叫绝的Wasserstein GAN[EB/OL]. ,2017-4-2.
[24]	ZHANG Han, GOODFELLOW I, METAXAS D, et al. Self-Attention Generative Adversarial Networks[EB/OL]. , 2018-11-23.
[25]	HEUSEL M, RAMSAUER H, UNTERTHINER T, et al.GANs Trained by a Two Time-scale Update Rule Converge to a Local Nash Equilibrium[C]//NIPS. 31st International Conference on Neural Information Processing Systems, December 4-9, 2017, Long Beach, California, USA. New York: Curran Associates, 2017: 6629-664.
[26]	CUI Zhengyang, HU Aiqun, PENG Linning.A Method of RF Fingerprint Recognition Based on Contour Feature[J]. Netinfo Security, 2017, 17(10): 75-80.
	崔正阳,胡爱群,彭林宁. 一种基于轮廓特征的射频指纹识别方法[J]. 信息网络安全,2017,17(10):75-80.
[27]	RAHMAN S, RAHMAN M M, HUSSAIN K, et al.Image Enhancement in Spatial Domain: A Comprehensive Study[C]//IEEE. 17th International Conference on Computer and Information Technology, December 22-23, 2014, Dhaka, Bangladesh. New Jersey: IEEE, 2014: 368-373.
[28]	WEEKS A R.Fundamentals of Electronic Image Processing[M]. Bellingham: SPIE Optical Engineering Press, 1996.
[29]	RADFORD A, METZ L, CHINTALA S. Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks[EB/OL]. , 2018-11-23.
[30]	SALIMANS T, GOODFELLOW I, ZAREMBA W, et al.Improved Techniques for Training GANs[C]//NIPS. 30st International Conference on Neural Information Processing Systems, December 5-10, 2016, Barcelona Spain. New York: Curran Associates, 2016: 2234-2242.
[31]	ZHANG Long, ZHAO Yujie, YE Xulun, et al.Collaborative Generation of Confrontation Networks[J]. Acta Automatica Sinica, 2018, 44(5): 804-810.
	张龙,赵杰煜,叶绪伦,等. 协作式生成对抗网络[J]. 自动化学报,2018,44(5):804-810.
[32]	KINGMA D P, WELLING M. Auto-encoding Variational Bayes[EB/OL]. , 2018-11-24.
[33]	DETON E, CHINTALA S, SZLAM A, et al.Deep Generative Image Models Using a Laplacian Pyramid of Adversarial Networks[C]//ACM. 28th International Conference on Neural Information Processing Systems, December 7-12, 2015, Montreal, Canada. New York: ACM, 2015: 1486-1494.
[34]	DOSOVITSKIY A, BROX T.Generating Images with Perceptual Similarity Metrics Based on Deep Networks[C]//NIPS. 30st International Conference on Neural Information Processing Systems, December 5-10, 2016, Barcelona Spain. New York: Curran Associates, 2016: 658-666.
[35]	ZHU Junyan, KRÄHENBÜHL, P, SHECHTMAN E, et al. Generative Visual Manipulation on the Natural Image Manifold[C]//Springer. 14th European Conference on Computer Vision, October 11-14, 2016, Amsterdam, Netherlands. Heidelberg: Springer, 2016: 597-613.
[36]	REED S, AKATA Z, YAN Xinchen, et al.Generative Adversarial Text to Image Synthesis[C]//IMLS. 33rd International Conference on International Conference on Machine Learning, June 19-24, New York, USA. New York: PMLR, 2016: 1060-1069.
[37]	DENG Y, LOY C C, TANG Xiaoou.Aesthetic-driven Image Enhancement by Adversarial Learning[C]//ACM. 2018 ACM Multimedia Conference, October 22-26, Amsterdam, Seoul, South Korea. New York: ACM, 2018: 870-878.
[38]	PERRONNIN F.AVA: A Large-scale Database for Aesthetic Visual Analysis[C]//IEEE. 2012 IEEE Conference on Computer Vision and Pattern Recognition, June 16-21, 2012, Providence, RI, USA. New Jersey: IEEE, 2012: 2408-2415.
[39]	LUO Wei, WANG Xiaogang, TANG Xiaoou.Content-based Photo Quality Assessment[J]. IEEE Transactions on Multimedia, 2013, 15(8): 1930-1943.
[40]	ZHANG Hui, WANG Kunfeng, WANG Feiyue.Application Progress and Prospect of Deep Learning in Target Visual Detection[J]. Acta Automatica Sinica, 2017, 43(8): 1289-1305.
	张慧,王坤峰,王飞跃. 深度学习在目标视觉检测中的应用进展与展望[J]. 自动化学报,2017,43(8):1289-1305.
[41]	STAUFFER C, GRIMSON W E L. Adaptive Background Mixture Models for Real-time Tracking[C]//IEEE. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, June 23-25, 1999, Fort Collins, Colorado. New Jersey: IEEE, 1999: 246-252.
[42]	ZIVKOVIC Z.Improved Adaptive Gaussian Mixture Model for Background Subtraction[C]//IEEE. 17th International Conference on Pattern Recognition, August 23-26, 2004, Cambridge, UK. New Jersey: IEEE, 2004: 28-31.
[43]	ST-CHARLES P L, BILODEAU G A. Improving Background Subtraction Using Local Binary Similarity Patterns[C]//IEEE. 2014 IEEE Winter Conference on Applications of Computer Vision, March 24-26, 2014, Steamboat Springs, CO, USA. New Jersey: IEEE, 2014: 509-515.
[44]	FAN Jingjing, XIN Yanzhe, DAI Fenglin, et al.Distributed Multi-camera Object Tracking with Bayesian inference[C]//IEEE. 2011 IEEE International Symposium of Circuits and Systems, May 15-18, 2011, Riode Janeiro, Brazil. New Jersey: IEEE, 2011: 357-360.
[45]	YAN Junhua, WANG Jiayi, XIE Tianxia, et al.Variational Bayesian Learning for Background Subtraction Based on Local Fusion Feature[J]. IET Computer Vision, 2017, 10(8): 884-893.
[46]	WANG Yi, JODOIN P M, PORIKLI F, et al.CDnet 2014: An Expanded Change Detection Benchmark Dataset[C]//IEEE. 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops, June 23-28, 2014, Columbus, OH, USA. New Jersey: IEEE, 2014: 393-400.
[47]	MARTINS I, CARVALHO P, CORTE-REAL L, et al.BMOG: Boosted Gaussian Mixture Model with Controlled Complexity[C]//Springer. Iberian Conference on Pattern Recognition and Image Analysis, June 20-23, 2017, Faro, Portugal. Heidelberg: Springer, 2017: 50-57.
[48]	SEHAIRI K, CHOUIREB F, MEUNIER J.Comparative Study of Motion Detection Methods for Video Surveillance Systems[J]. Journal of Electronic Imaging, 2017, 26(2): 23-25.
[49]	CHEN Yinigying, WANG Jinqiao, LU Hanqiang.Learning Sharable Models for Robust Background Subtraction[C]//IEEE. 2015 IEEE International Conference on Multimedia and Expo, June 29-July 3, 2015, Turin, Italy. New Jersey: IEEE, 2015: 1-6.
[50]	JIANG Shengqin, LU Xiaobo.Wesambe: A Weight-sample-based Method for Background Subtraction[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2018, 28(9): 2105-2115.
[51]	ZHENG Wenbo, WANG Kunfeng, WANG Feiyue.Background Subtraction Algorithm Based on Bayesian Generation Confrontation Network[J]. Acta Automatica Sinica, 2018, 44(5): 878-890.
	郑文博,王坤峰,王飞跃. 基于贝叶斯生成对抗网络的背景消减算法[J]. 自动化学报,2018,44(5):878-890.
[52]	BRAHAM M, VAN DROOGENBROECK M.Deep Background Subtraction with Scene-Specific Convolutional Neural Networks[C]//IEEE. IEEE International Conference on Systems, Signals and Image Processing (IWSSIP), May 23-25, 2016, Bratislava, Slovakia. Washington, DC, USA: EEE Computer Society, 2016: 1-4.
[53]	LI Shutao, WEI Dan.Compressed Sensing Overview[J]. Acta Automatica Sinica, 2009, 35(11): 1369-1377.
	李树涛,魏丹. 压缩传感综述[J]. 自动化学报,2009,35(11):1369-1377.
[54]	PENG Yigang, SUO Jinli, DAI Qionghai, et al.From Compressed Sensing to Low Rank Matrix Recovery: Theory and Applications[J]. Acta Automatica Sinica, 2013, 39(7): 981-994.
	彭义刚,索津莉,戴琼海,等. 从压缩传感到低秩矩阵恢复:理论与应用[J]. 自动化学报,2013,39(7):981-994.
[55]	ZHAO Shuyang, LI Jianwu.Low-rank Image Generation Method Based on Generating Confrontation Network[J]. Acta Automatica Sinica, 2018, 44(5): 829-839.
	赵树阳,李建武. 基于生成对抗网络的低秩图像生成方法[J]. 自动化学报,2018,44(5):829-839.
[56]	LU Qianwen, TAO Qingchuan, ZHAO Yalin, et al.Simplified Based on Comic Draft Drawings Generated Against the Web[J]. Acta Automatica Sinica, 2018, 44(5): 840-854.
	卢倩雯,陶青川,赵娅琳,等. 基于生成对抗网络的漫画草稿图简化[J]. 自动化学报,2018,44(5):840-854.
[57]	SUN Liang, HAN Yuxuan, KANG Wenjing, et al.Multi-view Learning and Reconstruction Algorithm Based on Generating Confrontation Network[J]. Acta Automatica Sinica, 2018, 44(5): 819-828.
	孙亮,韩毓璇,康文婧,等. 基于生成对抗网络的多视图学习与重构算法[J]. 自动化学报,2018,44(5):819-828.
[58]	YAO Naiming, GUO Qingpei, QIAO Fengchun, et al.Robust Facial Expression Recognition Based on Generative Confrontation Network[J]. Acta Automatica Sinica, 2018, 44(5): 865-877.
	姚乃明,郭清沛,乔逢春,等. 基于生成式对抗网络的鲁棒人脸表情识别[J]. 自动化学报,2018,44(5):865-877.
[59]	LIU Chang, WU Xiaolin, SHU Xiao. Learning-Based Dequantization For Image Restoration Against Extremely Poor Illumination[EB/OL]. https://arxiv.org/pdf/1803.01532.pdf, 2018-11-25.
[60]	TANG Xianlun, DU Yiming, LIU Yuwei, et al.Image Recognition Method Based on Conditional Deep Convolution to Generate Anti-network[J]. Acta Automatica Sinica, 2018, 44(5): 855-864.
	唐贤伦,杜一铭,刘雨微,等. 基于条件深度卷积生成对抗网络的图像识别方法[J]. 自动化学报,2018,44(5):855-864.
[61]	WEI Jiahui, CHEN Yuanping, REN Peng, et al. Image Enhancement with Conditional Adversarial Networks[EB/OL]. , 2018-11-26.
[62]	ISOLA P, ZHU Junyan, ZHOU Tinghui, et al.Image-to-Image Translation with Conditional Adversarial Networks[C]//IEEE. 2017 IEEE Conference on Computer Vision and Pattern Recognition, July 22-25, Honolulu, Hawaii. New Jersey: IEEE, 2017: 5967-5976.
[63]	DAI Bo, FIDLER S, URTASUN R, et al.Towards Diverse and Natural Image Descriptions via a Conditional GAN[C]//IEEE. 2017 IEEE International Conference on Computer Vision, October 22-29, 2017, Venice, Italy. New Jersey: IEEE, 2017: 2989-2998.
[64]	ARJOVSKY M, BOTTOU L. Towards Principled Methods for Training Generative Adversarial Networks[EB/OL]. , 2018-11-25.
[65]	WU Zhirong, SONG Shuran, KHOSLA A, et al.3D ShapeNets: A Deep Representation for Volumetric Shapes[C]//IEEE. 2015 IEEE Conference on Computer Vision and Pattern Recognition, June 8-10, 2015, Boston, Massachusetts, US. New Jersey: IEEE, 2015: 1912-1920.
[66]	ARORA S, GE Rong, LIANG Yingyu, et al.Generalization and Equilibrium in Generative Adversarial Nets (GANs)[C]//IMLS. 34th International Conference on Machine Learning, August 6-11, 2017, Sydney, NSW, Australia. New York: PMLR, 2017: 224-232.
[67]	THEIS L, OORD A, BETHGE M.A Note on the Evaluation of Generative Models[C]//Yoshua Bengio, Yann LeCun. 2016 International Conference on Learning Representations, May 2-4, 2016, San Juan, PR, US. New York: PMLR, 2016: 1-10.

GAN模型	改进	优点	缺点
CGAN	对模型增加约束条件,指导数据生成过程	对输入输出增加一个标签,能生成指定目标,收敛更快	对数据要求高,需要有标签或标记好的数据集
DCGAN	与CNN结合,采用分步卷积、批量标准化、LRELU等操作	稳定训练过程,易收敛,生成样本种类更丰富	训练不同数据需调整参数,模型易崩溃,会出现梯度消失或爆炸
WGAN	权重剪枝	训练过程更稳定,理论上解决梯度消失问题	权重的不恰当剪枝可能导致梯度消失或爆炸
StackGAN	采用两个GAN,第1个根据文本生成粗糙的图像,第2个修正生成的图像并添加细节	通过分阶段生成,最终生成的图像清晰度提高	生成任务的两个阶段可能找不到重点,导致生成失败
infoGAN	增加Q网络,通过无监督学习学到生成数据的类型	判别器准确率更高,并可以按照类别输出	约束条件选择和优化项求解困难
SAGAN	使用来自所有特征位置的线索生成详细信息,判别器可以检查整个图像特征是否彼此一致,将频谱归一化应用于GAN生成器	可以发现图像中的依赖关系,协调好每个位置细节与远端细节关系,可以对全局图像实施复杂的几何约束,生成图像质量更高	训练过程需要给G和D不同的学习速率

目的	应用模型	参考文献
图像生成	GAN、DCGAN、协作式生成对抗网络	[29,31,35,36]
图像质量提高	EnhanceGAN、GAN、LSGAN、基于贝叶斯生成对抗网络	[37,51,55,56]
图像补全	CGAN、WGAN、DCGAN	[57-59]
图像识别	条件深度生成式对抗网络	[60]
图像风格迁移	CGAN	[61,62]
图像字幕生成	CGAN	[63]