一种基于Yarn云计算平台与NMF的大数据聚类算法

doi:10.3969/j.issn.1671-1122.2018.08.006

信息网络安全 ›› 2018, Vol. 18 ›› Issue (8): 43-49.doi: 10.3969/j.issn.1671-1122.2018.08.006

一种基于Yarn云计算平台与NMF的大数据聚类算法

冯新扬¹(), 沈建京²

1.河南财经政法大学计算机与信息工程学院,河南郑州 450046
2.解放军战略支援部队信息工程大学,河南郑州 450002

收稿日期:2018-03-10 出版日期:2018-08-20 发布日期:2020-05-11
作者简介:
作者简介：冯新扬（1980—）,男,安徽,讲师,博士,主要研究方向为云计算、金融大数据;沈建京（1961—）,男,湖北,教授,博士,主要研究方向为分布式智能系统。

作为积累大数据的典型行业,电信行业积累了大量的手机用户行为数据,数据里包括用户拨出电话的基站信息、通话时间、通话时长等内容。一方面这些数据可以被用来研究用户之间形成的社交网络;另一方面,由于这些行为数据具有地理上下文,因此也可以基于网络理论来结合地理属性研究城市中不同区域之间的关系与功能。
基金资助:
国家自然科学基金[61202285];河南省科技攻关项目[122102210387];河南省教育厅科技攻关项目[13B520902]

A Yarn and NMF Based Big Data Clustering Algorithm

Xinyang FENG¹(), Jianjing SHEN²

1. School of Computer and Information Engineering, Henan University of Economics and Law, Zhengzhou Henan 450046, China
2. PLA Strategic Support Force Information Engineering University, Zhengzhou Henan 450002, China

Received:2018-03-10 Online:2018-08-20 Published:2020-05-11

摘要/Abstract

摘要：

为了改进MapReduce早期版本在大数据聚类算法方面的性能,文章提出了基于Yarn(Yet Another Resource Negotiator)云计算平台与非负矩阵分解NMF(Non-negative Matrix Factorization)的大数据聚类方法。文章讨论了高维数据相似性聚类与非负矩阵分解的结合及其面向MapReduce的数据聚类的任务划分方式。该方法的实现采用Hadoop2.0的Yarn平台,利用Hadoop的HDFS（Hadoop Distributed File System）来存储大容量的外部数据;描述了基于NMF的大数据相似性聚类方法的编码与实现过程,并以电信运营商的大数据作为案例程序进行了测试。实验结果表明,Yarn云平台比传统用于数据聚类的非负矩阵方法具有更好的运行时间与加速比,能够在可以接受的时间范围内完成电信运营商的大数据处理。

关键词: 云计算, 大数据, Yarn平台, 非负矩阵分解, 聚类算法

Abstract:

In order to improve the performance of MapReduce version 1 on big data processing, a Yarn and NMF (Non-negative Matrix Factorization) based Parallel hierarchical clustering algorithm was proposed in this paper. The combination of big data classification with NMF algorithm and the task partition in our MapReduce approach were discussed subsequently. Our approach used the Yarn distributed computation programming model of Hadoop2.0 and thus the big data was stored in HDFS (Hadoop Distributed File System). The coding mechanism and flow of hierarchical data clustering on Yarn were also discussed and described in detail. In order to demonstrate the efficiency of our approach, a serial of simulation experiments on a telecommunication big data were done. The results and performance analysis demonstrated that big data can be completed in an accepted time scope with Yarn framework. Good performance and speedup had been also obtained in our test.

Key words: cloud computing, big data, Yarn platform, non-negative matrix factorization, cluster algorithm

中图分类号:

TP309

冯新扬, 沈建京. 一种基于Yarn云计算平台与NMF的大数据聚类算法[J]. 信息网络安全, 2018, 18(8): 43-49.

Xinyang FENG, Jianjing SHEN. A Yarn and NMF Based Big Data Clustering Algorithm[J]. Netinfo Security, 2018, 18(8): 43-49.

图/表 7

图1

图2

图3

表1

表2

表3

图4

参考文献 18

[1]	SHEN Guowei, YANG Wu, WANG Wei, et al.Large-Scale Heterogeneous Data Co-Clustering Based on Nonnegative Matrix Factorization[J].Journal of Computer Research and Development, 2016,53(2): 459-466.
[2]	LI Jiejia, GUO Pengcheng, HAN Zhonghua.Research of Affinity Propagation Clustering Dimension Reduction on High-dimensional Data[J].Control Engineering of China, 2016, 23(9): 1418-1422.
[3]	LI Huan, LIU Feng, ZHU Erzhou.Research of an Impoved K-means Algorithm for Aanalyzing Mass Data[J].Microelectronics & Computer, 2016, 33(5):52-57.
[4]	REN Zhonglu, LI Jinming.Application of Non-negative Matrix Factorization in Microarray Data Classification and Clustering Discovery[J].Computer Engineering and Science, 2014, 36(7): 1389-1397.
[5]	XU Sen, LU Zhimao, GU Guochang.Integrating K-means and Non-negative Matrix Factorization to Ensemble Document Clustering[J].Journal of Jilin University(Engineering and Technology Edition), 2011, 41(4): 1077-1082.
[6]	ZAHARIA M, CHOWDHURY M,FRANKLIN M J, et al.Spark: Cluster Computing with Working Sets[J]. Usenix Conference on Hot Topics in Cloud Computing, 2010, 15(1):10.
[7]	APACHE.Hadoop NextGen MapReduce(Yarn) [EB/OL]. ,2017-10-15.
[8]	DEAN J,GHEMAWAT S.MapReduce: Simplified Data Processing on Large Clusters[C]//USENIX.OSDI'04 Proceedings of the 6th conference on Symposium on Opearting Systems Design & Implementation, December 6-8, 2004,San Francisco, CA,USA.Berkeley:USENIX, 2004: 137-150.
[9]	MAROZZO F,TALIA D,TRUNFIO P.P2P-MapReduce: Parallel Data Processing in Dynamic Cloud Environments[J].Journal of Computer & System Sciences, 2012, 78(5):1382-1402.
[10]	YU Pingping, NI Jiancheng, YAO Binxiu,et al.Highly Efficient Chinese Text Classification Algorithm of KNN Based on Spark Framework[J].Journal of Computer Applications, 2016, 36(12):3292-3297.
[11]	LI Qin, ZHU Yanchao, LIU Yi, et al.Accelerator Support in YARN Cluster[J].Journal of Computer Research and Development, 2016, 53(6):1263-1270.
[12]	CHEN Yang, WANG Yong, SUN Wei.A YARN-based Smart Grid Big Data Abnormal Detection[J].Netinfo Security,2017, 17(7):11-17.
[13]	WU Shiji, LI Chuan, TANG Changjie, et al.Efficient Adaptive Clustering Algorithm for Large Scale Information Network[J].Journal of Frontiers of Computer Science & Technology, 2014, 8(4): 406-416.
[14]	HE Mingliang, CHEN Zemao, ZUO Jin.Cluster Anomaly Detection Algorithm Based on Multi-windows Mechanism[J].Netinfo Security, 2016, 16(11):33-39.
[15]	LUO Mingwei, YAO Hongliang, LI Junzhao, et al.A Hierarchical Division Algorithm for Community Based on Node Dissimilarity[J].Computer Engineering, 2014, 40(1): 275-279.
[16]	WANG Yi, TANG Yong, LU Zexin, et al.Research on Features Selection in Malware Clustering[J].Netinfo Security,2016, 16(9):64-68.
[17]	LI Chenghua, ZHANG Xinfang, JIN Hai,et al.MapReduce: A New Programming Model for Distributed Parallel Computing[J]. Computer Engineering and Science, 2011, 33(3):129-135.
[18]	Hadoop.Open source web site 2017[EB/OL]. ,2017-10-15.

一种基于Yarn云计算平台与NMF的大数据聚类算法

A Yarn and NMF Based Big Data Clustering Algorithm

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 7

参考文献 18

相关文章 15

编辑推荐

Metrics

本文评价

[1]	刘渊, 乔巍. 云环境下基于Kubernetes集群系统的容器网络研究与优化[J]. 信息网络安全, 2020, 20(3): 36-44.
[2]	纪兆轩, 杨秩, 孙瑜, 单亦伟. 大数据环境下SHA1的GPU高速实现[J]. 信息网络安全, 2020, 20(2): 75-82.
[3]	白嘉萌, 寇英帅, 刘泽艺, 查达仁. 云计算平台基于角色的权限管理系统设计与实现[J]. 信息网络安全, 2020, 20(1): 75-82.
[4]	谢永恒, 冯宇波, 董清风, 王梅. 基于深度学习的数据接入方法研究[J]. 信息网络安全, 2019, 19(9): 36-40.
[5]	任良钦, 王伟, 王琼霄, 鲁琳俪. 一种新型云密码计算平台架构及实现[J]. 信息网络安全, 2019, 19(9): 91-95.
[6]	余奕, 吕良双, 李肖坚, 王天博. 面向移动云计算场景的动态网络拓扑描述语言[J]. 信息网络安全, 2019, 19(9): 120-124.
[7]	王紫璇, 吕良双, 李肖坚, 王天博. 基于共享存储的OpenStack虚拟机应用分发策略[J]. 信息网络安全, 2019, 19(9): 125-129.
[8]	崔艳鹏, 冯璐铭, 闫峥, 蔺华庆. 基于程序切片技术的云计算软件安全模型研究[J]. 信息网络安全, 2019, 19(7): 31-41.
[9]	葛新瑞, 崔巍, 郝蓉, 于佳. 加密云数据上支持可验证的关键词排序搜索方案[J]. 信息网络安全, 2019, 19(7): 82-89.
[10]	文奕, 陈兴蜀, 曾雪梅, 罗永刚. 面向安全分析的大规模网络下的DNS流量还原系统[J]. 信息网络安全, 2019, 19(5): 77-83.
[11]	田春岐, 李静, 王伟, 张礼庆. 一种基于机器学习的Spark容器集群性能提升方法[J]. 信息网络安全, 2019, 19(4): 11-19.
[12]	赵谱, 崔巍, 郝蓉, 于佳. 一种针对El-Gamal数字签名生成的安全外包计算方案[J]. 信息网络安全, 2019, 19(3): 81-86.
[13]	张振峰, 张志文, 王睿超. 网络安全等级保护2.0云计算安全合规能力模型[J]. 信息网络安全, 2019, 19(11): 1-7.
[14]	吴天雄, 陈兴蜀, 罗永刚. 大数据平台下应用程序保护机制的研究与实现[J]. 信息网络安全, 2019, 19(1): 68-75.
[15]	胡荣磊, 何艳琼, 曾萍, 范晓红. 一种大数据环境下医疗隐私保护方案设计与实现[J]. 信息网络安全, 2018, 18(9): 48-54.