A YARN-based Smart Grid Big Data Abnormal Detection

doi:10.3969/j.issn.1671-1122.2017.07.002

Abstract

Abstract:

The defects of processing smart grid big data in Map-Reduce early version were also discussed, and the advantages of processing smart grid big data in YARN were also described in this paper. The coding model and implementation and advantages of YARN-DPP were also analyzed. In order to demonstrate the effectiveness of YARN-DPP, the hardware configuration environments and software running environments had been completed. A serial of simulation experiments in IEEE 118 node grid system were also done. The results and performance analysis demonstrated that good throughput and speedup had been obtained in YARN-DPP. It can meet the fast demands in large scale grid system big data processing. The computing speed was faster than sequence computation and Map-Reduce computation.

Key words: smart grid, big data, YARN normalization, Map-Reduce

CLC Number:

TP393

Yang CHEN, Yong WANG, Wei SUN. A YARN-based Smart Grid Big Data Abnormal Detection[J]. Netinfo Security, 2017, 17(7): 11-17.

Figures/Tables 7

References 22

[1]	ZHENG Yu, CAPRA L, WOLFSON O, et al.Urban Computing: Concepts, Methodologies, and Applications[J]. ACM Transaction on Intelligent Systems and Technology (ACM TIST), 2014, 1(1): 1-9
[2]	李增鹏,邹岩,张磊,等. 一种基于全同态加密的智能电网数据交换隐私保护方案[J]. 信息网络安全,2016(3):1-7.
[3]	CHAN K W, DAI R C, CHEUNG C H.A Coarse Gain Parallel Solution Method for Solving Large Set of Power Systems Network Equation[C]//IEEE. 2002 International Conference on Power System Technology, October 13-17, 2002, Kunming, China. New Jersey: IEEE, 2002: 2640-2644.
[4]	曲朝阳,陈帅,杨帆,等. 基于云计算技术的电力大数据预处理属性约简方法[J]. 电力系统自动化,2014,38(8):67-71.
[5]	王伟,廖正宇,张辉,等. 基于大数据的铁路信号系统数据存储与分析系统设计与实现[J]. 信息网络安全,2017(1):29-37.
[6]	袁宝超,刘道伟,刘丽平,等. 基于Spark的大电网广域时空序列分析平台构建[J]. 电力建设,2016,37(11):48-54.
[7]	DEAN J, GHENAWAT S.MapReduce: Simplified Data Processing on Large Clusters[C]//ACM. 6th Conference on Symposium on Operating Systems Design & Implementation, December 6-8, 2004, San Francisco, CA, USA. New York: ACM, 2004: 107-113.
[8]	Marozzo F, Talia D, Trunfio P.P2P-MapReduce: Parallel Data Processing in Dynamic Cloud Environments[J]. Journal of Computer & System Sciences, 2012, 78(5): 1382-1402.
[9]	许元斌,钟小强,王丹,等. 基于MapReduce模型的电力数据并行化的异常检测[J]. 情报探索,2014,202(8):74-78.
[10]	APACHE. Hadoop NextGen MapReduce(YARN)[EB/OL]. .
[11]	诸骏伟. 电力系统分析(上册)[M]. 北京:中国电力出版社,1994.
[12]	薛巍,舒继武,王心丰,等. 电力系统潮流并行算法的研究进展[J]. 清华大学学报,2002,42(9):1192-1196.
[13]	毛建容,杨洪耕. 在线安全分析软件中的预想事故分析[J]. 四川电力技术,2002,(4):48-50.
[14]	宗永升,张祎. 电力系统在线安全分析并行分布式计算[J]. 计算机工程与设计,2010,31(19):4436-4439.
[15]	李钦,朱延超,刘轶,等. 基于YARN集群的计算加速部件扩展支持[J]. 计算机研究与发展,2016,53(6):1263-1270.
[16]	李成华,张新访,金海,等. MapReduce:新型的分布式并行计算编程模型[J]. 计算机工程与科学,2011,33(3):129-135.
[17]	APACHE. Welcome to Apache Hadoop[EB/OL]. .
[18]	刘鸿霞,李建清,张锐卿. 立体动态的大数据安全防护体系架构研究[J]. 信息网络安全,2016(9):18-25.
[19]	CHAO Hong.Implementation of Parallel-in-time Newton Method for Transient Stability Analysis on a Message Passing Multicomputer[C]//IEEE. 2002 International Conference on Power System Technology, October 13-17, 2002, Kunming, China. New Jersey: IEEE, 2002: 1239-1243.
[20]	王澄,魏庆来,赵冬斌,等. 基于数据的智能电网电能自适应优化调控[J]. 控制工程,2014,21(5):753-759.
[21]	石沙沙,孙文红,江明建,等. 基于分布式数据聚合的智能电网隐私保护协议研究[J]. 信息网络安全,2015(12):59-65.
[22]	GAO Tanjie.Data Processing with Spark[M]. Beijing: China Machine Press, 2015.