基于VIPS算法和模糊字典匹配的网页提取技术研究

doi:10.3969/j.issn.1671-1122.2014.10.009

信息网络安全 ›› 2014, Vol. 14 ›› Issue (10): 49-53.doi: 10.3969/j.issn.1671-1122.2014.10.009

基于VIPS算法和模糊字典匹配的网页提取技术研究

吴茜, 刘嘉勇, 卿粼波

四川大学电子信息学院,四川成都 610065

收稿日期:2014-07-12 出版日期:2014-10-01 发布日期:2015-08-17
作者简介:
吴茜（1988-）,女,河南,硕士研究生,主要研究方向：信息安全;刘嘉勇（1962-）,男,四川,教授,博士,主要研究方向：信息处理与信息安全;卿粼波（1982-）,男,四川,副教授,博士,主要研究方向：通信与信息处理。
基金资助:
教育部高等学校博士学科点专项科研基金[20110181120009]

Research on the Technology of Webpage Extraction Based on VIPS and Vague Dictionary

WU Qian, LIU Jia-yong, Qing Lin-bo

College of Electronics and Information Engineering, Sichuan University, Chengdu Sichuan 610065, China

Received:2014-07-12 Online:2014-10-01 Published:2015-08-17

摘要/Abstract

摘要：

在目前数据爆发的互联网时代,论坛舆论走向对于社会的影响越来越大,对舆论进行监控引导已经不可避免,在数据如此巨大的环境中,有效地监控舆论信息成为一个难题。论坛网页中标题、内容等关键信息是舆论监控中的主要以及重点信息。为了提取论坛网页中的标题、内容、作者等与舆情相关的信息,文章提出了一种基于VIPS算法和智能模糊字典匹配相结合的网页内容提取方法。VIPS算法是利用Web页面的视觉提示背景颜色,字体的颜色和大小,边框、逻辑块和逻辑块之间的间距等,结合DOM树进行页面语义分块。智能模糊字典采用AC-BM匹配算法把VIPS分块的语义块与数据库里的标签相匹配,提取出匹配正确的字段。两者的结合可以提取出帖子的标题、内容、作者、发帖时间等信息。该方法具体步骤是首先利用VIPS算法将网页页面块进行提取,再用分隔条检测设置分隔条,然后重构语义块,检测后将分割后的网页保存为xml格式文件,再将xml文件中的语义块与字典进行匹配,提取出匹配成功的内容。最后,文章通过实验证明了该方法的有效性。

关键词: 信息提取, VIPS算法, 智能字典, AC-BM算法

Abstract:

In the age of data explosion, the consensustowardsare very important to the society. It is necessaryto monitor and guide the towards of the consensus, in the environment of the big data, it’s a different problem that how to monitor the consensus effectively. In order to extra the title、content、author、time information of the BBS webpage.The paper introduces the method based on VIPS algorithm and intelligent fuzzy dictionary.VIPS uses the vision information such as background, font color, font size, border, margin and DOM tree to get semantic block. The intelligent fuzzy dictionary matches the semantic block to the tag name in database using AC-BM algorithm, and get the matched fields. Then the tow combinativemethod can extract the key messages .That method first uses VIPS algorithm to divide webpage in blocks, reconstructs semantic block, saves to a xml file, then matches the semantic block in xml file with the dictionary, extracts the matching content. This paper proves the validity of this method through the experiment.

Key words: information extraction, VIPS algorithm, intelligent dictionary, AC-BM algorithm

中图分类号:

TP309

吴茜, 刘嘉勇, 卿粼波. 基于VIPS算法和模糊字典匹配的网页提取技术研究[J]. 信息网络安全, 2014, 14(10): 49-53.

WU Qian, LIU Jia-yong, Qing Lin-bo. Research on the Technology of Webpage Extraction Based on VIPS and Vague Dictionary[J]. Netinfo Security, 2014, 14(10): 49-53.

图/表 6

参考文献 21

[1]	HE Jia, Zhou Chang-sheng, SHI Xian-feng.Implementation Method for Network Public Opinion Monitoring System[J]. Journal of Zhengzhou University(Science Edition), 2010, 42(1): 82-85.
[2]	Eirinaki M, Vazirgiannis M.Web mining for Web personalization[ J] . ACM Transactions on Internet Technology, 2003, 3(1): 12-13.
[3]	ZHANG Chao.Research on the Application of Text Orientation Analysis in the Public Opinion Monitoring System[D]. Beijing:BeiJing University of Posts and Telecommunications, 2008.
[4]	LIN Hong-fei.A Hybrid Model For Text Filtering[J]. Journal of Computer Research and Development,2001, 38(9): 1127-1131.
[5]	QIU Jiang-tao, TANG Chang-jie, LI Chuan, et al.News Content Extraction Based on Block Distribution[J]. Journal of Jilin University(Engineering and Technology Edition),2009, 39(5):207-211.
[6]	CHANG Hong-yao, ZHU Zheng-yu, CHEN Ye,et al.Content Extraction Technique for Web Pages Based on HTML-tags[J], Computer Engineering and Design,2010, 31(24):33-37.
[7]	SUN Cheng-jie, GUAN Yi.A Statistical Approach for Content Extraction from Web Page[J]. Journal of Chinese Information Processing, 2004, 18(5): 17-22.
[8]	Gao Le, Zhang Jian, Tian Xian-zhong.Improvement and Implementation of VIPS Algorithm[J]. Computer Systems & Applications, 2009, 18(4): 65-69.
[9]	ZHANG Rui-xue,SONG Ming-qiu,GONG Yan-lei.Parsing DOM Tree Reversely and Extracting Web Main Page Information[J].Computer Science, 2011, 38(4): 213-215.
[10]	FENG Yong, HE Xun, TANG Li,et al.Text Knowledge Management Oriented Adaptive Chinese Word Segmentation algorithms[J].Journal of Chongqing University(Natural Science Edition), 2010, 33(10): 110-117.
[11]	ZOU Zhi-min, GUO He-qing, GAO Yin.EnglishStirng Segmentation Method[J].Application Research of Computers, 2007, 24(7):58-60.
[12]	WU Xiao-zhou, WAN Li-ming, HAN Xiao-song,et al.An HMM Based Transcription Factor Name Mining Algorithm[J].Journal of Jilin University(Science Edition),2012,50(2):320-322.
[13]	HE Zhao-wei, CHEN Jun-liang.Information Extraction Based on Ontology Entity Relation Match[J].Computer Engineering,2007,33(21):207-209.
[14]	Cai D, Yu S,Wen JR,et al. VIPS:a vision based page segmentation algorithm[R]. Microsoft Technical Report,MSR- TR-2003–79, 2003.
[15]	CHEN Kai-qu, ZHAO Jie, PENG Zhi-wei.Fast Approximate String Matching for Chinese Text[J].Journal of Chinese Information Processing, 2004, 18(2): 58-65.
[16]	WANG Jing-fan, WU Xiao-jun, XIA Yun-qing,et al.An Approximate String Matching Algorithm for Chinese Information Retrieval Systems[J].Journal of Chinese Information Processing,2007, 21(6): 59-64.
[17]	ZHOU Si-wei,CAI Yong.Mend of AC-BM Algorithm and Application in Intrusion Detection Technique[J].Microcomputer Applications,2007,28(01):27-31.
[18]	ZHU Jun.Multiple Patterns Match Algorithm research[D].Hefei:He Fei University of Technology, 2010.
[19]	Wan, Guo-gen, Zhi-guang Qin. Improved AC-BM Algorithm for Matching Multiple Strings[J]. Journal of University of Electronic Science and Technology of China,2006,35(4): 531-533.
[20]	HE bo. Distributed algorithm for mining association rules based on FP-tree[J]. Control and Decision, 2012, 27(4): 618-622.
[21]	YI Yu, LIAO Nain-dong, HUANG Jing-quan.Pure Parameterized Extension of Pattern Trees Induction[J]. Computer Engineering and Applications, 2012, 48(31):166-170.

基于VIPS算法和模糊字典匹配的网页提取技术研究

Research on the Technology of Webpage Extraction Based on VIPS and Vague Dictionary

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 6

参考文献 21

相关文章 1

编辑推荐

Metrics

本文评价