Research and Implementation on Parallel Crawl Method for Source Code Based on MapReduce

doi:10.3969/j.issn.1671-1122.2018.01.009

Abstract

Abstract:

With the increment of the number of the open source code, it has been a trend to reuse existing codes in software programming. In order to quickly and accurately search open source code, this paper carries out a method of parallel crawling of source code based on MapReduce. Firstly, analyze the current open source code libraries to select the appropriate site and object, and clear the whole parallel crawl the process and target. Secondly, design the Map method and Reduce method according to source code characters, and propose a source code parallel crawl algorithm. Finally, implement the parallel crawling of open source code with cluster computing environment. The experiment shows that using the multi-machine parallel to search source code compared with the traditional method, the speed is obviously improved, and the results of the search results are more reliable.

Key words: software engineering, source code searching, parallel crawling, MapReduce, open source code

CLC Number:

TP309

Junyan MA, Guosun ZENG. Research and Implementation on Parallel Crawl Method for Source Code Based on MapReduce[J]. Netinfo Security, 2018, 18(1): 58-66.

Figures/Tables 4

References 17

[1]	文旆. 开源运动及其影响研究[D]. 长沙:国防科学技术大学,2014.
[2]	LEMOS O A L, PAULA A C D, KONISHI G, et al. Using Thesaurus-based Tag Clouds to Improve Test-Driven Code Search[C]//IEEE. 2013 VII Brazilian Symposium on Software Components, Architectures and Reuse, September 29-October 4, 2013, Brasilia, Brazil. New Jersey: IEEE, 2013: 99-108.
[3]	吕昊. 面向垂直搜索的聚焦爬虫研究及应用[D]. 杭州:浙江大学,2008.
[4]	徐哲,蔡建平. 基于开源软件的软件工程实施系统研究[J]. 微计算机信息,2009,25(9):178-180.
[5]	赵滟,顾益军. 基于主观Bayes方法和相关反馈技术的文档搜索重排序[J]. 信息网络安全,2015(4):41-44.
[6]	张丽. PageRank算法的改进[J]. 科学技术与工程,2007,7(5):673-677.
[7]	余兆钗,傅化权. 一种改进的最好优先搜索策略算法[J]. 科技视界,2014(33):89-89
[8]	KEIVANLOO I.Leveraging Clone Detection for Internet-scale Source Code Search[C]//IEEE. 2012 IEEE International Conference on Program Comprehension, June 11-13, 2012, Passau, Germany. New Jersey: IEEE, 2012: 277-280.
[9]	朱嵘良. 分布式并行环境下的网络爬虫研究[D]. 北京:中央民族大学,2015.
[10]	周德懋,李舟军. 高性能网络爬虫:研究综述[J]. 计算机科学,2009,36(8):26-29.
[11]	HEYDON A, NAJORK M.MERCATOR: A Scalable, Extensible Web Crawler[J]. World Wide Web-internet & Web Information Systems, 1999, 2(4): 219-229.
[12]	杨溥. 搜索引擎中爬虫的若干问题研究[D]. 北京:北京邮电大学,2009.
[13]	吴文忠,易平. MapReduce在分布式搜索引擎中的应用[J]. 计算机系统应用,2012,21(2):249-251.
[14]	廖子渊,陈明志,邓辉. 基于评价可信度的云计算信任管理模型研究[J]. 信息网络安全,2016(2):33-39.
[15]	崔秀峰. 基于MPI的分布式搜索引擎设计与实现[D]. 大连:大连理工大学,2013.
[16]	张治宏. 基于MPI的并行计算研究[D]. 北京:中国地质大学,2006.
[17]	刘莎,楚传仁. 基于文件等级的Ceph数据冗余存储策略的研究[J]. 信息网络安全,2016(4):50-54.