Research on the Technology of Webpage Extraction Based on VIPS and Vague Dictionary

doi:10.3969/j.issn.1671-1122.2014.10.009

Abstract

Abstract:

In the age of data explosion, the consensustowardsare very important to the society. It is necessaryto monitor and guide the towards of the consensus, in the environment of the big data, it’s a different problem that how to monitor the consensus effectively. In order to extra the title、content、author、time information of the BBS webpage.The paper introduces the method based on VIPS algorithm and intelligent fuzzy dictionary.VIPS uses the vision information such as background, font color, font size, border, margin and DOM tree to get semantic block. The intelligent fuzzy dictionary matches the semantic block to the tag name in database using AC-BM algorithm, and get the matched fields. Then the tow combinativemethod can extract the key messages .That method first uses VIPS algorithm to divide webpage in blocks, reconstructs semantic block, saves to a xml file, then matches the semantic block in xml file with the dictionary, extracts the matching content. This paper proves the validity of this method through the experiment.

Key words: information extraction, VIPS algorithm, intelligent dictionary, AC-BM algorithm

CLC Number:

TP309

WU Qian, LIU Jia-yong, Qing Lin-bo. Research on the Technology of Webpage Extraction Based on VIPS and Vague Dictionary[J]. Netinfo Security, 2014, 14(10): 49-53.

Figures/Tables 6

References 21

[1]	HE Jia, Zhou Chang-sheng, SHI Xian-feng.Implementation Method for Network Public Opinion Monitoring System[J]. Journal of Zhengzhou University(Science Edition), 2010, 42(1): 82-85.
[2]	Eirinaki M, Vazirgiannis M.Web mining for Web personalization[ J] . ACM Transactions on Internet Technology, 2003, 3(1): 12-13.
[3]	ZHANG Chao.Research on the Application of Text Orientation Analysis in the Public Opinion Monitoring System[D]. Beijing:BeiJing University of Posts and Telecommunications, 2008.
[4]	LIN Hong-fei.A Hybrid Model For Text Filtering[J]. Journal of Computer Research and Development,2001, 38(9): 1127-1131.
[5]	QIU Jiang-tao, TANG Chang-jie, LI Chuan, et al.News Content Extraction Based on Block Distribution[J]. Journal of Jilin University(Engineering and Technology Edition),2009, 39(5):207-211.
[6]	CHANG Hong-yao, ZHU Zheng-yu, CHEN Ye,et al.Content Extraction Technique for Web Pages Based on HTML-tags[J], Computer Engineering and Design,2010, 31(24):33-37.
[7]	SUN Cheng-jie, GUAN Yi.A Statistical Approach for Content Extraction from Web Page[J]. Journal of Chinese Information Processing, 2004, 18(5): 17-22.
[8]	Gao Le, Zhang Jian, Tian Xian-zhong.Improvement and Implementation of VIPS Algorithm[J]. Computer Systems & Applications, 2009, 18(4): 65-69.
[9]	ZHANG Rui-xue,SONG Ming-qiu,GONG Yan-lei.Parsing DOM Tree Reversely and Extracting Web Main Page Information[J].Computer Science, 2011, 38(4): 213-215.
[10]	FENG Yong, HE Xun, TANG Li,et al.Text Knowledge Management Oriented Adaptive Chinese Word Segmentation algorithms[J].Journal of Chongqing University(Natural Science Edition), 2010, 33(10): 110-117.
[11]	ZOU Zhi-min, GUO He-qing, GAO Yin.EnglishStirng Segmentation Method[J].Application Research of Computers, 2007, 24(7):58-60.
[12]	WU Xiao-zhou, WAN Li-ming, HAN Xiao-song,et al.An HMM Based Transcription Factor Name Mining Algorithm[J].Journal of Jilin University(Science Edition),2012,50(2):320-322.
[13]	HE Zhao-wei, CHEN Jun-liang.Information Extraction Based on Ontology Entity Relation Match[J].Computer Engineering,2007,33(21):207-209.
[14]	Cai D, Yu S,Wen JR,et al. VIPS:a vision based page segmentation algorithm[R]. Microsoft Technical Report,MSR- TR-2003–79, 2003.
[15]	CHEN Kai-qu, ZHAO Jie, PENG Zhi-wei.Fast Approximate String Matching for Chinese Text[J].Journal of Chinese Information Processing, 2004, 18(2): 58-65.
[16]	WANG Jing-fan, WU Xiao-jun, XIA Yun-qing,et al.An Approximate String Matching Algorithm for Chinese Information Retrieval Systems[J].Journal of Chinese Information Processing,2007, 21(6): 59-64.
[17]	ZHOU Si-wei,CAI Yong.Mend of AC-BM Algorithm and Application in Intrusion Detection Technique[J].Microcomputer Applications,2007,28(01):27-31.
[18]	ZHU Jun.Multiple Patterns Match Algorithm research[D].Hefei:He Fei University of Technology, 2010.
[19]	Wan, Guo-gen, Zhi-guang Qin. Improved AC-BM Algorithm for Matching Multiple Strings[J]. Journal of University of Electronic Science and Technology of China,2006,35(4): 531-533.
[20]	HE bo. Distributed algorithm for mining association rules based on FP-tree[J]. Control and Decision, 2012, 27(4): 618-622.
[21]	YI Yu, LIAO Nain-dong, HUANG Jing-quan.Pure Parameterized Extension of Pattern Trees Induction[J]. Computer Engineering and Applications, 2012, 48(31):166-170.