信息网络安全

信息网络安全 ›› 2017, Vol. 17 ›› Issue (1): 29-37.doi: 10.3969/j.issn.1671-1122.2017.01.005

• • 上一篇    下一篇

基于大数据的铁路信号系统数据存储与分析系统设计与实现

王伟1,2,3(), 廖正宇1,2,3, 张辉1,2,3, 郭栋1,2,3   

  1. 1. 同济大学计算机科学与技术系,上海 200092
    2. 国家高性能计算机工程技术中心同济分中心,上海 200092
    3. 同济大学嵌入式系统与服务计算教育部重点实验室,上海 200092
  • 收稿日期:2016-11-28 出版日期:2017-01-20 发布日期:2020-05-12
  • 作者简介:

    作者简介： 王伟（1979—）,男,湖北,副教授,博士,主要研究方向为信息安全、并行分布式计算;廖正宇（1992—）,男,甘肃,硕士研究生,主要研究方向为云计算;张辉（1992—）,男,江苏,硕士研究生,主要研究方向为虚拟化、容器技术;郭栋（1991—）,男,内蒙古,硕士研究生,主要研究方向为云计算、云件。

  • 基金资助:
    国家自然科学基金[61672384];上海市优秀学术带头人计划[15XD1503600];计算机体系结构国家重点实验室开放课题[CARCH201408]

Design and Implementation on Data Storage and Analysis System of Railway Signal System Based on Big Data

Wei WANG1,2,3(), Zhengyu LIAO1,2,3, Hui ZHANG1,2,3, Dong GUO1,2,3   

  1. 1. Department of Computer Science and Engineering, Tongji University, Shanghai 200092, China
    2. National High Performance Computer Engineering Technology Center Tongji Center, Shanghai 200092, China
    3. The Key Laboratory of Embedded System and Service Computing, Ministry of Education, Tongji University, Shanghai 200092,China
  • Received:2016-11-28 Online:2017-01-20 Published:2020-05-12

RichHTML

2

PDF (PC)

103

摘要：

文章基于Hadoop技术,设计并实现了一个铁路信号数据存储与分析系统。首先,文章研究了Hadoop平台下分布式系统工作原理、HDFS分布式文件管理作用机制、MapReduce模型。然后,针对信号检测大数据设计相关文件解析接口,处理包括txt和CSV格式的数据包;利用HDFS分布式文件管理系统完成对数据的存储和管理;根据数据存储结构设计HBase表,建立快速存取查询索引,并编写HBase操作的各式API。最后,利用ExtJS搭建前端展示页面,通过服务器与Hadoop平台通信完成电气特性分析和日志分析,并将数据处理结果以列表、曲线图和散点图的可视化方式展示在前端;同时完成了对海量数据的挖掘和可视化展现工作。

关键词: 大数据, Hadoop, MapReduce, 数据存储与分析, 铁路信号系统

Abstract:

This paper designs and implements a big data’s storage and analysis system of railway signal system based on Hadoop. Firstly, this paper studies the working principle of Hadoop distributed system, HDFS (the Hadoop distributed file system) and MapRedeuce model. And then, APIs of signal data analyzers about files of txt and CSV are designed. The storage, distribution and management of files and data are achieved by HDFS. According to the structure of all data files, this paper uses HBase to design methods which can find the data file in a quick time and design the API about all the operations in HBase. Finally, this paper uses ExtJS to build the front-end display page, communicate with the Hadoop platform through the server to complete the electrical characteristic analysis and log analysis, and display the data processing results with the list, graph and scatter diagram. This paper has accomplished the data mining and visualization of the work.

Key words: big data, Hadoop, MapReduce, data storage and analysis, railway signal system

中图分类号: