基于Hadoop的异构网络协同过滤推荐算法设计

doi:10.3969/j.issn.1671-1122.2022.10.013

信息网络安全 ›› 2022, Vol. 22 ›› Issue (10): 91-97.doi: 10.3969/j.issn.1671-1122.2022.10.013

基于Hadoop的异构网络协同过滤推荐算法设计

余先荣¹(), 樊捷杰²

1.江西警察学院科技与信息安全系，南昌 330100
2.北京科技大学计算机与通信工程学院，北京 100089

收稿日期:2022-08-19 出版日期:2022-10-10 发布日期:2022-11-15
通讯作者: 余先荣 E-mail:yuxianrong@jxga.edu.cn
作者简介:余先荣（1983—），男，江西，讲师，硕士，主要研究方向为数据挖掘、复杂网络数据分析|樊捷杰（1985—），男，江西，博士研究生，主要研究方向为机器学习、人工智能和数据安全
基金资助:
江西省教育厅科学技术研究项目(GJJ151198)

Design of Collaborative Filtering Approach Recommendation Algorithm Based on Hadoop

YU Xianrong¹(), FAN Jiejie²

1. Department of Science Technology and Information Security, Jiangxi Police Institute, Nanchang 330100, China
2. School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing 100089, China

Received:2022-08-19 Online:2022-10-10 Published:2022-11-15
Contact: YU Xianrong E-mail:yuxianrong@jxga.edu.cn

摘要/Abstract

摘要：

在处理复杂计算任务时，来自不同人群的大量异构数据会在异构网络中产生异常值和噪声，这极易导致推荐算法性能低下。针对此类问题，文章提出了一种基于物品和用户的协同过滤个性化推荐算法，并在Hadoop平台下实现算法的分布式并行化。首先，基于皮尔森相关和余弦相似度的方法，在其相似度计算中引入评分贡献权重函数；其次，构造异构网络，通过对权重函数的设计，计算物品相似性，进而实现了离群点敏感性的钝化；最后，在Hadoop平台下，分别对基于物品和用户的异构网络的协同过滤推荐算法在影视数据集上进行验证。实验结果表明，该算法能有效提高推荐算法的准确率和实时性，改善网络监测质量，延长网络生存时间。

关键词: 协同过滤, 推荐, 异构网络, Hadoop

Abstract:

While dealing with complex computing tasks, the large number of heterogeneous data from different populations will cause abnormal values and noise in heterogeneous networks, which will lead to low performance of the recommendation algorithm easily. Thus, a personalized recommendation algorithm is proposed for such issue based on the items. Firstly, based on the Pearson correlation and cosine similarity method, the weight function of the item contribution is introduced in the similarity calculation.Secondly, according to the construction of the heterogeneous network, the similarity of two items is calculated by the design of the weight function, the insensitive performance of the outliers is realized. Finally, according to the movie data, we realized the collaborative filtering of the recommendation algorithm based on the Hadoop platform. Experimental results show that the method can effectively improve the accuracy and real-time performance of recommendation algorithm, improve the quality of network monitoring and prolong the network lifetime.

Key words: collaborative filtering, recommendation, heterogeneous network, Hadoop

中图分类号:

TP309

余先荣, 樊捷杰. 基于Hadoop的异构网络协同过滤推荐算法设计[J]. 信息网络安全, 2022, 22(10): 91-97.

YU Xianrong, FAN Jiejie. Design of Collaborative Filtering Approach Recommendation Algorithm Based on Hadoop[J]. Netinfo Security, 2022, 22(10): 91-97.

图/表 8

图1

图2

图3

图4

图5

图6

图7

图8

参考文献 15

[1]	RESNICK P, VARIAN H R. Recommender Systems[J]. Communications of the ACM, 1997, 40(3): 56-58.
[2]	BURKE R. Hybrid Recommender Systems: Survey and Experiments[J]. User Modeling and User-Adapted Interaction, 2002, 12(4): 331-370.
[3]	WANG Zhihu, HUANG Manying. Collaborative Filtering Recommendation Algorithm Based on User's Historical Behavior[J]. Microelectronics & Computer, 2017, 34(5): 132-136.
	王志虎, 黄曼莹. 基于用户历史行为的协同过滤推荐算法[J]. 微电子学与计算机, 2017, 34(5): 132-136.
[4]	DESHPANDE M, KARYPIS G. Item-Based Top-N Recommendation Algorithms[J]. ACM Trans on Information Systems, 2004, 22 (1): 143-177
[5]	LINDEN G, SMITH B, YORK J. Amazon. com Recommendations: Item-to-item Collaborative Filtering[J]. IEEE Internet Computing, 2003, 7(1): 76-80.
[6]	LI Kunlun, WAN Pinzhe, ZHANG Dezhi. Collaborative Filtering Recommendation Algorithm Based on Improving User Similarity Measurement and Rating Prediction[J]. Journal of Chinese Computer Systems, 2018, 39(3): 567-571.
	李昆仑, 万品哲, 张德智. 基于改进用户相似性度量和评分预测的协同过滤推荐算法[J]. 小型微型计算机系统, 2018, 39(3): 567-571.
[7]	WANG Ran, XU Yi, HU Shanzhong, et al. A Slope One Recommendation Algorithm Based on the Weighted Bipartite Graph[J]. Microelectronics & Computer, 2018, 35(3): 93-98.
	王冉, 徐怡, 胡善忠, 等. 基于加权二部图的Slope One推荐算法[J]. 微电子学与计算机, 2018, 35(3): 93-98.
[8]	YU Jinping, ZHANG Yong, LIAO Liefa, et al. Collaborative Filtering Algorithm Based on the Hybrid Frog Jump Joint Clustering[J]. Microelectronics & Computer, 2016, 33(1): 65-71.
	喻金平, 张勇, 廖列法, 等. 基于混合蛙跳联合聚类的协同过滤算法[J]. 微电子学与计算机, 2016, 33(1): 65-71.
[9]	WANG Yuanlong, SUN Weizhen, XIANG Yong. Implementation and Improvement of a Hybrid Collaborative Filtering Algorithm Based on Spark[J]. Application Research of Computers, 2019, 36(3): 855-860.
	王源龙, 孙卫真, 向勇. 基于Spark的混合协同过滤算法改进与实现[J]. 计算机应用研究, 2019, 36(3): 855-860.
[10]	PANG Hailong, ZHAO Hui, LI Wanlong, et al. Collaborative Filtering Combined with a Linear Regression Recommendation Algorithm[J]. Application Research of Computers, 2019, 36(5): 1302-1304, 1310.
	庞海龙, 赵辉, 李万龙, 等. 融合协同过滤的线性回归推荐算法[J]. 计算机应用研究, 2019, 36(5): 1302-1304,1310.
[11]	JI Xiaoyan, LI Yulong. Personalized Recommendation Algorithm Based on Improved Clustering in the Hadoop Environment[J]. Journal of Lanzhou Jiaotong University, 2017, 36(1): 70-76.
	冀晓岩, 李玉龙. Hadoop环境下基于改进聚类的个性化推荐算法[J]. 兰州交通大学学报, 2017, 36(1): 70-76.
[12]	LIU Bin, LI Fan, YAO Bin. Design and Implementation of a Personalized Recommendation Algorithm Based on Hadoop[J]. Journal of Shanxi University of Science & Technology, 2018, 36(2): 154-159.
	刘斌, 李帆, 姚斌. 基于Hadoop个性化推荐算法设计与实现[J]. 陕西科技大学学报, 2018, 36(2): 154-159.
[13]	LIU Wenchao, ZHU Yongzhi, WANG Sifeng. Parallel Collaborative Filtering Recommendation Algorithm Based on MapReduce[J]. Journal of Qufu Normal University(Natural Science), 2018, 44(1): 50-53.
	刘文超, 祝永志, 王斯锋. 基于MapReduce的并行协同过滤推荐算法[J]. 曲阜师范大学学报(自然科学版), 2018, 44(1): 50-53.
[14]	ZHOU Zhangbing, ZHAO Deng, SHU Lei, et al. Efficient Multi-Attribute Query Processing in Heterogeneous Wireless Sensor Networks[J]. Journal of Internet Technology, 2014, 15(5): 699-712.
[15]	HAJIR M, LANGAR R, GAGNON F. Coalitional Games for Joint Co-Tier and Cross-Tier Cooperative Spectrum Sharing in Dense Heterogeneous Networks[J], IEEE Access, 2016, (4): 2450-2464.

基于Hadoop的异构网络协同过滤推荐算法设计

Design of Collaborative Filtering Approach Recommendation Algorithm Based on Hadoop

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 8

参考文献 15

相关文章 15

编辑推荐

Metrics

本文评价

[1]	刘龙庚. 基于异构网络空管安全监控关联算法研究[J]. 信息网络安全, 2022, 22(4): 58-66.
[2]	罗文华, 程家兴. 基于Hadoop架构的混合型DDoS攻击分布式检测系统[J]. 信息网络安全, 2021, 21(2): 61-69.
[3]	吴天雄, 陈兴蜀, 罗永刚. 大数据平台下应用程序保护机制的研究与实现[J]. 信息网络安全, 2019, 19(1): 68-75.
[4]	陆勰, 罗守山, 张玉梅. 基于Hadoop的海量安全日志聚类算法研究[J]. 信息网络安全, 2018, 18(8): 56-63.
[5]	张春霞, 王新猛, 张晓熙. 基于Hadoop的森林公安网络舆情监测系统设计与实现[J]. 信息网络安全, 2018, 18(12): 82-86.
[6]	郁湧, 刘永刚, 顾捷. 网构软件系统中构件之间信任关系分析与度量[J]. 信息网络安全, 2018, 18(1): 31-37.
[7]	徐燕. 基于数据挖掘的网络链接预测研究[J]. 信息网络安全, 2017, 17(6): 30-34.
[8]	王伟, 廖正宇, 张辉, 郭栋. 基于大数据的铁路信号系统数据存储与分析系统设计与实现[J]. 信息网络安全, 2017, 17(1): 29-37.
[9]	姜凤燕, 姜瑾, 姜吉婷. 基于大数据环境的电子取证研究[J]. 信息网络安全, 2016, 16(9): 60-63.
[10]	陈希林, 马丁. 针对微博信息分析的HBase存储结构设计[J]. 信息网络安全, 2016, 16(9): 267-271.
[11]	孙大鹏. 云计算技术在垃圾短信过滤中的应用与实现[J]. 信息网络安全, 2015, 15(7): 13-19.
[12]	黄海, 谢冬青, 宋一赞. WiFi-WiMAX异构无线网络认证机制研究[J]. 信息网络安全, 2015, 15(6): 19-25.
[13]	柴华, 刘建毅. 一种改进的slope one推荐算法研究[J]. 信息网络安全, 2015, 15(2): 77-81.
[14]	陈耀兵;刘斌;史延涛. 基于Hadoop架构的大数据量日志存储和检索优化[J]. , 2013, 13(6): 0-0.
[15]	苏烈华;李恒训;李锁雷. 基于时空折中算法的密码分析系统设计与实现[J]. , 2013, 13(5): 0-0.