Design and Implementation on Data Storage and Analysis System of Railway Signal System Based on Big Data

doi:10.3969/j.issn.1671-1122.2017.01.005

Abstract

Abstract:

This paper designs and implements a big data’s storage and analysis system of railway signal system based on Hadoop. Firstly, this paper studies the working principle of Hadoop distributed system, HDFS (the Hadoop distributed file system) and MapRedeuce model. And then, APIs of signal data analyzers about files of txt and CSV are designed. The storage, distribution and management of files and data are achieved by HDFS. According to the structure of all data files, this paper uses HBase to design methods which can find the data file in a quick time and design the API about all the operations in HBase. Finally, this paper uses ExtJS to build the front-end display page, communicate with the Hadoop platform through the server to complete the electrical characteristic analysis and log analysis, and display the data processing results with the list, graph and scatter diagram. This paper has accomplished the data mining and visualization of the work.

Key words: big data, Hadoop, MapReduce, data storage and analysis, railway signal system

CLC Number:

TP393

Wei WANG, Zhengyu LIAO, Hui ZHANG, Dong GUO. Design and Implementation on Data Storage and Analysis System of Railway Signal System Based on Big Data[J]. Netinfo Security, 2017, 17(1): 29-37.

Figures/Tables 23

References 23

[1]	中国铁路总公司. 铁路信号集中监测系统应用与维护技术[M]. 北京:中国铁道出版社,2013.
[2]	姚丽娟. 我国铁路信号系统的现状与发展[J]. 铁路通信信号,2003,39(4):27-29.
[3]	Hadoop. Hadoop 分布式文件系统:架构和设计[EB/OL]. .
[4]	陈希林,马丁. 针对微博信息分析的HBase存储结构设计[J]. 信息网络安全,2016(9):267-271.
[5]	ALEX H.Hadooop硬实战[M]. 梁李印,宁青,杨卓荦,译. 北京:电子工业出版社,2015.
[6]	PATEL A B, BIRLA M, NAIR U. Addressing Big Data Problem using Hadoop and Map Reduce[EB/OL]., 2016-10-11.
[7]	王媛媛,吕晓丹,胡琪,等. 基于HBase的RDF数据存储方案研究与设计[J].信息网络安全,2016(3):59-63.
[8]	路嘉恒. Hadoop实战[M]. 北京:机械工业出版社,2012.
[9]	DEAN J. GHEMAWAT S. MapReduce: Simplified data processing on largeclusters[EB/OL]. , 2016-9-13.
[10]	DHRUBA B. HDFS Architecture[EB/OL]. , 2016-9-13.
[11]	PORTER G.Decoupling Storage and Computing in Hadoop with SuperDataNode[J]. ACM Sigops Operating Systems Review, 2010, 44(2): 41-46.
[12]	刘鸿霞,李建清,张锐卿. 立体动态的大数据安全防护体系架构研究[J]. 信息网络安全,2016(9):18-25.
[13]	BARKAN O, WEILL J, WOLF L, et al.Fast High Dimensional Vector Multiplication Face Recognition[C]//IEEE. 2013 IEEE International Conference on Computer Vision, December 1-8, 2013, Sydney, Australia. New Jersey: IEEE, 2013: 1960-1967.
[14]	RANGER C, RAGHURAMAN R, PENMETSA A, et al.Evaluating MapReduce for Multi-core and Multiprocessor Systems[C]//IEEE. IEEE 13th International Symposium on High Performance Computer Architecture, February 10-14, 2007, Scottsdale, AZ, USA. New Jersey: IEEE, 2007: 13-24.
[15]	李宏军, 郎为民,邓刚. 一种高效的大数据中心完整性检查方案研究[J].信息网络安全,2016(5):1-8.
[16]	姜锋. 基于Hadoop平台的海量数据处理研究及应用[D]. 北京:北京邮电大学,2014.
[17]	吕明育. Hadoop架构下数据挖掘与数据迁移系统的设计与实现[D]. 上海:上海交通大学,2012.
[18]	Apache. Hadoop Java API [EB/OL]. , 2016-9-10.
[19]	王紫东. 神朔铁路河东运输段信号专家系统方案设计[J]. 信息通信,2015(3):173-174.
[20]	BORTHAKUR D.The Hadoop Distributed File System: Architecture and Design[J]. Hadoop Project Website, 2007, 11(11): 1-10.
[21]	董西成. Hadoop技术内幕深入解析MapReduce架构设计与实现原理[M]. 北京:机械工业出版社,2013.
[22]	管磊,胡光俊,王专. 基于大数据的网络安全态势感知技术研究[J]. 信息网络安全,2016(9):45-50.
[23]	徐照岗. 基于Hadoop的海量气象数据的存储设计与实现[D]. 青岛:山东科技大学,2010.