Research on Data Ingestion Method Based on Deep Learning

doi:10.3969/j.issn.1671-1122.2019.09.008

Abstract

Abstract:

When ingesting multi-source heterogeneous data, big data centers need to analyze standard conversion problems that deal with data of different standards. In the past, most of this work relied on manual analysis. In recent years, the fast-developing deep learning technology has excellent feature extraction and classification ability. This study designed a data access matching algorithm based on deep learning technology which combined data morphological model and semantic model, and completed verification by using sample data. The results show that the proposed method could achieve a reasonable level of accuracy, and has potential for future improvement, which could replace some manual work in many practical applications.

Key words: big data, data center, data access, deep learning

CLC Number:

TP309

Yongheng XIE, Yubo FENG, Qingfeng DONG, Mei WANG. Research on Data Ingestion Method Based on Deep Learning[J]. Netinfo Security, 2019, 19(9): 36-40.

Figures/Tables 4

References 9

[1]	WANG Wenjie, HU Baiqing, LIU Chi.A Survey of Open Source Software for Big Data Governance and Security[J]. Netinfo Security, 2017, 17(5): 28-36.
	王文杰,胡柏青,刘驰.开源大数据治理与安全软件综述[J].信息网络安全,2017,17(5):28-36.
[2]	LECUN Y, BENGIO Y, HINTON G. Deep Learning[EB/OL]. , 2015-5-28.
[3]	BENGIO Y, DUCHARME R, VINCENT P, et al. A Neural Probabilistic Language Model[EB/OL]. , 2019-2-11.
[4]	SZEGEDY C, VANHOUCKE V, IOFFE S, et al. Rethinking the Inception Architecture for Computer Vision[EB/OL]. , 2019-2-11.
[5]	HOCHREITER S, SCHMIDHUBER J.Long Short-term Memory[J]. Neural Computation, 1997, 9(8): 1735-1780.
[6]	SRIVASTAVA N, HINTON G, KRIZHEVSKY A, et al.Dropout: A Simple Way to Prevent Neural Networks from Overfitting[J]. Journal of Machine Learning Research, 2014, 15(1): 1929-1958.
[7]	JONES K S.A Statistical Interpretation of Term Specificity and Its Application in Retrieval[J]. Journal of Documentation, 1972, 28(1): 11-21.
[8]	MIKOLOV T, CHEN Kai, CORRADO G, et al. Efficient Estimation of Word Representations in Vector Space[EB/OL]. , 2019-2-11.
[9]	Wikimedia Foundation. Wikimedia Chinese Corpus[EB/OL]. , 2019-2-11.

测试结果	Top-1	Top-3	Top-5
数据表匹配准确率	56%	64%	67%
字段匹配准确率	89%	95%	96%

[1]	Rong WANG, Chunguang MA, Peng WU. An Intrusion Detection Method Based on Federated Learning and Convolutional Neural Network [J]. Netinfo Security, 2020, 20(4): 47-54.
[2]	Zhaoxuan JI, Zhi YANG, Yu SUN, Yiwei SHAN. GPU High Speed Implementation of SHA1 in Big Data Environment [J]. Netinfo Security, 2020, 20(2): 75-82.
[3]	Chunguang MA, Yaoyao GUO, Peng WU, Haibo LIU. Review of Image Enhancement Based on Generative Adversarial Networks [J]. Netinfo Security, 2019, 19(5): 10-12.
[4]	Yi WEN, Xingshu CHEN, Xuemei ZENG, Yonggang LUO. DNS Protocol Restore System for Security Analysis Based on Large-scale Network [J]. Netinfo Security, 2019, 19(5): 77-83.
[5]	Yong FANG, Guangxiatian ZHU, Luping LIU, Peng JIA. Research on Browser Fuzz Sample Generation Technology Based on Deep Learning [J]. Netinfo Security, 2019, 19(3): 26-33.
[6]	ZHU Haiqi, JIANG Feng. Research and Analysis of Anomaly Detection Technology for Operation and Maintenance Data in the Era of Artificial Intelligence [J]. Netinfo Security, 2019, 19(11): 24-35.
[7]	Tianxiong WU, Xingshu CHEN, Yonggang LUO. Research and Implementation of Application Program Protection Mechanism under Big Data Platform [J]. Netinfo Security, 2019, 19(1): 68-75.
[8]	HU Ronglei, HE Yanqiong, ZENG Ping, FAN Xiaohong. Design and Implementation of Medical Privacy Protection Scheme in Big Data Environment [J]. 信息网络安全, 2018, 18(9): 48-54.
[9]	FENG Xinyang, SHEN Jianjing. A Yarn and NMF Based Big Data Clustering Algorithm [J]. 信息网络安全, 2018, 18(8): 43-49.
[10]	TAO Yuan, HUANG Tao, ZHANG Mohan, LI Shuilin. Research and Development Trend Analysis of Key Technologies for Cyberspace Security Situation Awareness [J]. 信息网络安全, 2018, 18(8): 79-85.
[11]	YOU Lin, LIANG Jiahao. Research on Secure Identity Authentication Based on Homomorphic Encryption and Biometric [J]. 信息网络安全, 2018, 18(4): 1-8.
[12]	LU Xiuqing, XIAN Hequn. A Big Data Integrity Auditing Scheme Based on User Authorization in Cloud Storage [J]. 信息网络安全, 2018, 18(4): 32-37.
[13]	DUAN Dagao, XIE Yongheng, GAI Xinxin, LIU Zhanbin. A Rumor Detection Model Based on Neural Network [J]. 信息网络安全, 2017, 17(9): 134-137.
[14]	GUO Min, ZENG Yingming, YAO Jinli, DA Xiaowen. The Analysis of Software Behavior Security Based on Big Data Samples [J]. 信息网络安全, 2017, 17(9): 153-156.
[15]	CHEN Yang, WANG Yong, SUN Wei. A YARN-based Smart Grid Big Data Abnormal Detection [J]. 信息网络安全, 2017, 17(7): 11-17.