面向专用信息获取的用户定制主题网络爬虫技术研究

doi:10.3969/j.issn.1671-1122.2017.02.003

摘要/Abstract

摘要：

进入大数据时代,互联网已成为各行各业进行信息采集的重要阵地。面对爆炸式增长的网络信息资源,如何快速高效地筛选出所需的信息成为亟需解决的现实难题。在互联网海量数据和专用信息采集人员之间构建一个满足特定需求的信息筛选机制,可以大幅度提高专用信息获取工作效率。主题网络爬虫是所有互联网信息获取手段必须具备的首要环节,为了提高专用信息采集的准确性,文章进行了面向公开网络的用户定制主题网络爬虫技术研究。针对大数据时代信息筛选困难的问题,文章通过将用户的兴趣偏好融入到主题网络爬虫的抓取过程中,有效提高了信息筛选力度,并通过实验验证了文中方法能够提高查准率。

关键词: 大数据, 主题网络爬虫, Pagerank算法, 行为分析, 用户定制

Abstract:

Stepping into the era of big data, the Internet has become an important battle field for every walk of life to collect intelligence. Facing the explosive growth of network information resources, how to screen out the required information quickly and efficiently is a practical problem to solve. It is very important to construct an information screening mechanism between the mass data and intelligence personnel to meet the needs of specific tasks, which can greatly improve the efficiency. In order to improve the accuracy of the information collected, this paper conducts the research on the user customized topic Web crawler technology for information acquisition. In order to solve the difficult problem of information screening in the large data age, the user’s interest preference is integrated into the crawling process of the topic Web crawler, and the information screening is effectively improved. Experimental results show that the method can improve the precision.

Key words: big data, topic Web crawler, Pagerank algorithm, behavior analysis, user customized

中图分类号:

TP309

薛丽敏, 吴琦, 李骏. 面向专用信息获取的用户定制主题网络爬虫技术研究[J]. 信息网络安全, 2017, 17(2): 12-21.

Limin XUE, Qi WU, Jun LI. Research on User Customized Topic Web Crawler for Specialized Information Acquiration Technology[J]. Netinfo Security, 2017, 17(2): 12-21.

图/表 11

图1

表1

图2

图3

图4

图5

图6

图7

图8

图9

图10

参考文献 37

[1]	CNNIC. 第38次《中国互联网络发展状况统计报告》[EB/OL]. 2016-8-3.
[2]	CHAKRABARTI S, BERG M V D, DOM B. Focused Crawling: A New Approach to Topic Specific Web Resource Discovery[J].Computer Networks, 1999, 31(11-16): 1623-1640.
[3]	AGGARWAL C C, GARAWI A F, YU P S.Intelligent Crawling on the World Wide Web with Arbitrary Predicates[C]//ACM. 10th International Conference on World Wide Web, May 1-5, 2001, Hong Kong, China. New York: ACM, 2001:96-105.
[4]	EHRIG M, MAEDCHE A.Ontology-focused Crawling of Web Documents[C]//ACM. 2003 ACM Symposium on Applied Computing, March 9-12, 2003, Melbourne, FL, USA. New York: ACM, 2003: 1174-1178.
[5]	陈军,陈竹敏. 基于网页分块的sharksearch算法[J]. 山东大学学报:理学版,2007(9):62-66.
[6]	夏崇镨,康丽. 基于叙词表的主题网络爬虫技术研究[J]. 现代图书情报技术,2007(5):41-44.
[7]	CHO J, GARCIAM H, PAGE L.Efficient Crawling through URL Ordering[J].Computer Networks and Systems, 1998, 30(1-7): 161-172.
[8]	杨佳玉,叶昕昕,陈思聪,等. 社交网站安全问题分析与对策研究[J].信息网络安全,2014(4):82-87.
[9]	DEBRA P M B, POST R D J. Information Retrieval in the World Wide Web: Making Client-based Searching Feasible[J].Computer Networks & Isdn Systems, 1994, 27(2): 183-192.
[10]	HERSOVICI M, JACOVI M, MAAREK Y, et al.The shark-search Algorithm-an Application: Tailored Web Site Mapping[J].Computer Networks & Isdn Systems, 1998, 30(1-7): 317-326.
[11]	LIU Hongyu, MILIOS E, KORBA L.Exploiting Multiple Features with MEMMs for Focused Web Crawling[C]//ACM. 13th International Conference on Applications of Natural Language to Information Systems, June 24-27, 2008, London, UK. New York: ACM, 2008: 99-110.
[12]	WANG Can, GUAN Ziyu, CHEN Chun, et al.On-line Topical Importance Estimation: An Effective Focused Crawling Algorithm Combining Link and Content Analysis[J].Journal of Zhejiang University-Science A, 2009, 10(8): 1114-1124.
[13]	赵燕,陈晓云,莫明辉,等. 基于用户群的智能主题爬虫[J].广西师范大学学报:自然科学版,2007,2(25):230-233.
[14]	支凤麟,徐炜民. 基于主题的个性化查询扩展模型[J].计算机工程与设计,2010,31(20):4471-4475.
[15]	ABKENARI A F, SELAMAT A. Application of Clickstream Analysis in a Tailored Focused Web Crawler[EB/OL]., 2016-9-10.
[16]	闵钰麟,黄永峰. 用户定制主题聚焦爬虫的设计与实现[J].计算机工程与设计,2015,1(36):17-21.
[17]	吴经伟,刘学亮. 用户定制主题爬虫的设计与实现[J].山西科技,2016,31(2):135-140.
[18]	周永进,马春光,苗俊峰,等. 基于社区层次的机会网络移动模型[J]. 信息网络安全,2014(8):45-49.
[19]	黎柯,蔡永香,干佳林,等. 行业动态信息采集系统关键问题解决方案[J].测绘科学,2016,3(41):161-166.
[20]	王冲,纪仙慧. 基于用户反馈与链接关系的网页排序改进算法[J].计算机工程与设计,2013,5(37):1166-1170.
[21]	杜雷,辛阳. 基于规则库和网络爬虫的漏洞检测技术研究与实现[J]. 信息网络安全,2014(10):38-43.
[22]	MENCZER F, PANT G, PADMINI S.Topical Web Crawlers: Evaluating Adaptive Algorithm[J].ACM Transactions on Internet Technology, 2004, 4(4): 378-419.
[23]	王帅,周国民,王健. 主题网络爬虫相关度算法研究综述[J].计算机与现代化,2013,1(4):27-30.
[24]	陈晓,赵晶玲. 大数据处理中混合型聚类算法的研究与实现[J]. 信息网络安全,2015(4):45-49.
[25]	TAHER H.Topic-sensitive PageRank[C]//ACM. 11st International Conference on World Wide Web. May 7-11, 2002, Honolulu, Hawaii, USA. New York: ACM, 2002: 784-796.
[26]	李卫东,陆玲. 融合VSM技术的PageRank算法研究与应用[J]. 计算机与现代化,2011(7):96-98.
[27]	王钟斐,工彪. 基于锚文本相似度的PageRank改进算法[J].计算机工程,2010,36(24):258-260.
[28]	LARSON R R.Bibliometrics of the World Wide Web: an Exploratory Analysis of the Intellectual Structure of Cyberspace[J].Asis Meeting, 1996, 33(1): 71-78.
[29]	王冲,纪仙慧. 基于用户反馈与链接关系的网页排序改进算法[J].计算机工程与设计,2013,5(37):1166-1170.
[30]	LI Xin, LIU Bing, YU P.Time Sensitive Ranking with Applieation to Publication Search[C]//IEEE. IEEE 13th International Conference on Data Mining, December 15-19, 2008, Pisa, Italy. Washington: IEEE, 2008: 893-898.
[31]	李东,王虎强. 基于Timed-PageRank的聚焦爬虫优化研究[J].四川兵工学报,2015,1(1):141-144.
[32]	王旭阳,任国盛. 基于用户行为与页面分析的改进Pagerank算法[J].计算机工程,2016,2(42):164-168.
[33]	杨定中,赵刚,王泰. 网络爬虫在Web信息搜索与数据挖掘中应用[J].计算机工程与设计,2009,24(30):5658-5662.
[34]	XING Wenpu, GHORBANI A.Weighted Pagerank Algorithm[C]//IEEE. Second Annual Conference on Communication Netwoks and Services Research, May 19-21, 2004, Fredericton, NB, Canada. Washington: IEEE, 2004: 305-314.
[35]	黄华东. 基于用户模型的个性化搜索研究[D]. 上海:上海华东理工大学,2013.
[36]	黄贤英,陈红阳. 基于用户兴趣度的PageRank改进算法[J].重庆理工大学学报:自然科学版,2014,28(5):74-78.
[37]	胡雪,封化民,李明伟,等. 数据挖掘中一种增强的Apriori算法分析[J]. 信息网络安全,2015(11):77-83.