A Yarn and NMF Based Big Data Clustering Algorithm

doi:10.3969/j.issn.1671-1122.2018.08.006

Abstract

Abstract:

In order to improve the performance of MapReduce version 1 on big data processing, a Yarn and NMF (Non-negative Matrix Factorization) based Parallel hierarchical clustering algorithm was proposed in this paper. The combination of big data classification with NMF algorithm and the task partition in our MapReduce approach were discussed subsequently. Our approach used the Yarn distributed computation programming model of Hadoop2.0 and thus the big data was stored in HDFS (Hadoop Distributed File System). The coding mechanism and flow of hierarchical data clustering on Yarn were also discussed and described in detail. In order to demonstrate the efficiency of our approach, a serial of simulation experiments on a telecommunication big data were done. The results and performance analysis demonstrated that big data can be completed in an accepted time scope with Yarn framework. Good performance and speedup had been also obtained in our test.

Key words: cloud computing, big data, Yarn platform, non-negative matrix factorization, cluster algorithm

CLC Number:

TP309

Xinyang FENG, Jianjing SHEN. A Yarn and NMF Based Big Data Clustering Algorithm[J]. Netinfo Security, 2018, 18(8): 43-49.

Figures/Tables 7

References 18

[1]	SHEN Guowei, YANG Wu, WANG Wei, et al.Large-Scale Heterogeneous Data Co-Clustering Based on Nonnegative Matrix Factorization[J].Journal of Computer Research and Development, 2016,53(2): 459-466.
[2]	LI Jiejia, GUO Pengcheng, HAN Zhonghua.Research of Affinity Propagation Clustering Dimension Reduction on High-dimensional Data[J].Control Engineering of China, 2016, 23(9): 1418-1422.
[3]	LI Huan, LIU Feng, ZHU Erzhou.Research of an Impoved K-means Algorithm for Aanalyzing Mass Data[J].Microelectronics & Computer, 2016, 33(5):52-57.
[4]	REN Zhonglu, LI Jinming.Application of Non-negative Matrix Factorization in Microarray Data Classification and Clustering Discovery[J].Computer Engineering and Science, 2014, 36(7): 1389-1397.
[5]	XU Sen, LU Zhimao, GU Guochang.Integrating K-means and Non-negative Matrix Factorization to Ensemble Document Clustering[J].Journal of Jilin University(Engineering and Technology Edition), 2011, 41(4): 1077-1082.
[6]	ZAHARIA M, CHOWDHURY M,FRANKLIN M J, et al.Spark: Cluster Computing with Working Sets[J]. Usenix Conference on Hot Topics in Cloud Computing, 2010, 15(1):10.
[7]	APACHE.Hadoop NextGen MapReduce(Yarn) [EB/OL]. ,2017-10-15.
[8]	DEAN J,GHEMAWAT S.MapReduce: Simplified Data Processing on Large Clusters[C]//USENIX.OSDI'04 Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation, December 6-8, 2004,San Francisco, CA,USA.Berkeley:USENIX, 2004: 137-150.
[9]	MAROZZO F,TALIA D,TRUNFIO P.P2P-MapReduce: Parallel Data Processing in Dynamic Cloud Environments[J].Journal of Computer & System Sciences, 2012, 78(5):1382-1402.
[10]	YU Pingping, NI Jiancheng, YAO Binxiu,et al.Highly Efficient Chinese Text Classification Algorithm of KNN Based on Spark Framework[J].Journal of Computer Applications, 2016, 36(12):3292-3297.
[11]	LI Qin, ZHU Yanchao, LIU Yi, et al.Accelerator Support in YARN Cluster[J].Journal of Computer Research and Development, 2016, 53(6):1263-1270.
[12]	CHEN Yang, WANG Yong, SUN Wei.A YARN-based Smart Grid Big Data Abnormal Detection[J].Netinfo Security,2017, 17(7):11-17.
[13]	WU Shiji, LI Chuan, TANG Changjie, et al.Efficient Adaptive Clustering Algorithm for Large Scale Information Network[J].Journal of Frontiers of Computer Science & Technology, 2014, 8(4): 406-416.
[14]	HE Mingliang, CHEN Zemao, ZUO Jin.Cluster Anomaly Detection Algorithm Based on Multi-windows Mechanism[J].Netinfo Security, 2016, 16(11):33-39.
[15]	LUO Mingwei, YAO Hongliang, LI Junzhao, et al.A Hierarchical Division Algorithm for Community Based on Node Dissimilarity[J].Computer Engineering, 2014, 40(1): 275-279.
[16]	WANG Yi, TANG Yong, LU Zexin, et al.Research on Features Selection in Malware Clustering[J].Netinfo Security,2016, 16(9):64-68.
[17]	LI Chenghua, ZHANG Xinfang, JIN Hai,et al.MapReduce: A New Programming Model for Distributed Parallel Computing[J]. Computer Engineering and Science, 2011, 33(3):129-135.
[18]	Hadoop.Open source web site 2017[EB/OL]. ,2017-10-15.