面向高速网络流实时处理的资源自适应伸缩方法

doi:10.3969/j.issn.1671-1122.2025.01.010

信息网络安全 ›› 2025, Vol. 25 ›› Issue (1): 110-123.doi: 10.3969/j.issn.1671-1122.2025.01.010

面向高速网络流实时处理的资源自适应伸缩方法

康仕才¹^,²^,³, 陈良国¹^,²^,³, 陈兴蜀¹^,²^,³()

1.四川大学网络空间安全学院，成都 610065
2.数据安全防护与智能治理教育部重点实验室，成都 610065
3.四川大学网络空间安全研究院，成都 610065

收稿日期:2024-10-22 出版日期:2025-01-10 发布日期:2025-02-14
通讯作者: 陈兴蜀 E-mail:chenxsh@scu.edu.cn
作者简介:康仕才（2000—），男，四川，硕士研究生，主要研究方向为数据安全管理|陈良国（1993—），男，贵州，博士研究生，主要研究方向为大数据和网络安全|陈兴蜀（1968—），女，贵州，教授，博士，主要研究方向为云计算安全、数据安全、威胁检测、开源情报和人工智能安全
基金资助:
中央高校基础研究基金(SCU2024D01);中央高校基础研究基金(2023SCU12126);中央高校基础研究基金(2022SCU12116);四川大学理工科发展计划(2020SCUNG129)

Resource Adaptive Scaling Method for Real-Time Processing of High-Speed Network Streaming

KANG Shicai¹^,²^,³, CHEN Liangguo¹^,²^,³, CHEN Xingshu¹^,²^,³()

1. School of Cyber Science and Engineering, Sichuan University, Chengdu 610065, China
2. Key Laboratory of Data Protection and Intelligent Management, Ministry of Education, Chengdu 610065, China
3. Cyber Science Research Institute, Sichuan University, Chengdu 610065, China

Received:2024-10-22 Online:2025-01-10 Published:2025-02-14
Contact: CHEN Xingshu E-mail:chenxsh@scu.edu.cn

摘要/Abstract

摘要：

在流处理中，静态资源配置难以应对实时变化且具有突发性的流数据负载，因此需要引入弹性机制。然而，在确定弹性伸缩时机和伸缩策略时，若未充分考虑伸缩的成本与收益之间的平衡，则会引发资源频繁调整，导致系统不稳定或效率降低。为解决这一问题，文章提出一种资源自适应伸缩算法，该算法通过分析流数据负载规模和资源使用情况，确定资源伸缩的方向和规模。同时，该算法采用一种最大平均处理吞吐量方法，以伸缩操作前后的处理吞吐量为量化指标评估资源伸缩所带来的开销与收益，优化伸缩策略，避免不必要的资源频繁调整。基于该算法，文章设计了网络流弹性处理框架，以实现框架的灵活扩展与资源的动态调整。在不同网络带宽场景下对框架进行测试，实验结果表明，该算法能够有效权衡开销与收益，精确实现资源的伸缩，应用该算法后，框架资源利用率提高40%以上，能够满足高速网络流处理的性能需求。

关键词: 网络流, 流处理, 自适应, 资源伸缩

Abstract:

In stream processing, static resource allocation is difficult to cope with real-time changes and sudden streaming data loads, so elasticity mechanisms need to be introduced. However, when determining the elastic scaling timing and scaling strategy, if the balance between the cost and benefit of scaling is not fully considered, frequent resource adjustments will be triggered, resulting in the system becoming unstable or less efficient instead. To solve this problem, this paper proposed a resource adaptive scaling algorithm, which determined the direction and scale of resource scaling by analyzing the scale of streaming data load and resource usage. At the same time, the algorithm proposed a maximum average processing throughput method, which took the processing throughput before and after the scaling operation as a quantitative index to evaluate the overhead and benefit brought by resource scaling, optimize the scaling strategy, and avoid unnecessary frequent adjustment of resources. Based on this algorithm, this paper designed a network flow elasticity processing framework, which realized the flexible expansion of the framework and the dynamic adjustment of resources. The framework is tested in different network bandwidth scenarios, and the experimental results show that the algorithm can effectively weigh the overhead and benefit, and accurately realize the resource scaling, and after applying the algorithm, the resource utilization of the framework is effectively increased more than 40%, which is able to satisfy the performance requirements of high-speed network stream processing.

Key words: network streaming, stream processing, adaptive, resource scaling

中图分类号:

TP309

康仕才, 陈良国, 陈兴蜀. 面向高速网络流实时处理的资源自适应伸缩方法[J]. 信息网络安全, 2025, 25(1): 110-123.

KANG Shicai, CHEN Liangguo, CHEN Xingshu. Resource Adaptive Scaling Method for Real-Time Processing of High-Speed Network Streaming[J]. Netinfo Security, 2025, 25(1): 110-123.

图/表 12

表1

图1

图2

图3

图4

表2

图5

图6

图7

图8

图9

图10

参考文献 23

[1]	DE M T, MENCAGLI G. Keep Calm and React with Foresight[J]. ACM SIGPLAN Notices, 2016, 51(8): 1-12.
[2]	WANG Li, FU T Z J, MA R T B, et al. Elasticutor: Rapid Elasticity for Realtime Stateful Stream Processing[C]// ACM. Proceedings of the 2019 International Conference on Management of Data. New York: ACM, 2019: 573-588.
[3]	SU Zipeng, YUAN Lei, LIU Peng, et al. Research and Implementation of Real-Time Processing Model of High-Speed Network Stream[J]. Journal of Shandong University (Natural Science), 2022, 57(9): 25-32.
	苏自鹏, 袁磊, 刘鹏, 等. 高速网络流实时处理模型研究与实现[J]. 山东大学学报(理学版), 2022, 57(9): 25-32.
[4]	ZHAN Hanglong, LIU Lantao, KANG Lianghuan, et al. An Elastic Scalable Stream Processing Framework Based on Actor Model[J]. Journal of Computer Research and Development, 2017, 54(5): 1086-1096.
	詹杭龙, 刘澜涛, 康亮环, 等. 一种基于Actor模型的弹性可伸缩的流处理框架[J]. 计算机研究与发展, 2017, 54(5): 1086-1096.
[5]	KULKARNI S, BHAGAT N, FU Maosong, et al. Twitter Heron: Stream Processing at Scale[C]// ACM. Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data. New York: ACM, 2015: 239-250.
[6]	APACHE F. Apache Flink Home Page[EB/OL]. (2023-03-29)[2024-08-12]. https://flink.apache.org.
[7]	APACHE S S. Apache Spark Streaming Home Page[EB/OL]. (2023-03-29)[2024-08-12]. https://spark.apache.org/streaming.
[8]	ZEUCH S, DEL M B, KARIMOV J, et al. Analyzing Efficient Stream Processing on Modern Hardware[J]. Proceedings of the VLDB Endowment, 2019, 12(5): 516-530.
[9]	ZHANG Liang, ZHENG Wenli, LI Chao, et al. AuTraScale: An Automated and Transfer Learning Solution for Streaming System Auto-Scaling[C]// IEEE. 2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS). New York: IEEE, 2021: 912-921.
[10]	SATZGER B, HUMMER W, LEITNER P, et al. Esc: Towards an Elastic Stream Computing Platform for the Cloud[C]// IEEE. 2011 IEEE 4th International Conference on Cloud Computing. New York: IEEE, 2011: 348-355.
[11]	RAZAVI K, LUTHRA M, KOLDEHOFE B, et al. FA2:Fast, Accurate Autoscaling for Serving Deep Learning Inference with SLA Guarantees[C]// IEEE. 2022 IEEE 28th Real-Time and Embedded Technology and Applications Symposium (RTAS). New York: IEEE, 2022: 146-159.
[12]	RUSSO G R. Model-Based Auto-Scaling of Distributed Data Stream Processing Applications[C]// ACM. Proceedings of the 21st International Middleware Conference Doctoral Symposium. New York: ACM, 2020: 5-8.
[13]	CASTRO F R, MIGLIAVACCA M, KALYVIANAKI E, et al. Integrating Scale Out and Fault Tolerance in Stream Processing Using Operator State Management[C]// ACM. Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data. New York: ACM, 2013: 725-736.
[14]	SUN Dawei, HE Hanyu, YAN Hongbin, et al. Lr-Stream: Using Latency and Resource Aware Scheduling to Improve Latency and Throughput for Streaming Applications[J]. Future Generation Computer Systems, 2021, 114: 243-258.
[15]	ZHANG Shuhao, WU Yingjun, ZHANG Feng, et al. Towards Concurrent Stateful Stream Processing on Multicore Processors[C]// IEEE. 2020 IEEE 36th International Conference on Data Engineering (ICDE). New York: IEEE, 2020: 1537-1548.
[16]	ZHANG Shuhao, HE Jiong, ZHOU A C, et al. BriskStream: Scaling Data Stream Processing on Shared-Memory Multicore Architectures[C]// ACM. Proceedings of the 2019 International Conference on Management of Data. New York: ACM, 2019: 705-722.
[17]	CHEN Liangguo. Research and Implementation of Real-Time Network Traffic Processing Framework for Security Analysis[D]. Chengdu: Sichuan University, 2020.
	陈良国. 面向安全分析的网络流量处理框架研究与实现[D]. 成都: 四川大学, 2020.
[18]	DOU Rengan, MA R T B. Latency-Oriented Elastic Memory Management at Task-Granularity for Stateful Streaming Processing[C]// IEEE. IEEE INFOCOM 2023-IEEE Conference on Computer Communications. New York: IEEE, 2023: 1-10.
[19]	LOMBARDI F, ANIELLO L, BONOMI S, et al. Elastic Symbiotic Scaling of Operators and Resources in Stream Processing Systems[J]. IEEE Transactions on Parallel and Distributed Systems, 2018, 29(3): 572-585.
[20]	DE M T, MENCAGLI G. Elastic Scaling for Distributed Latency-Sensitive Data Stream Operators[C]// IEEE. 2017 25th Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP). New York: IEEE, 2017: 61-68.
[21]	DE M T, MENCAGLI G. Proactive Elasticity and Energy Awareness in Data Stream Processing[J]. Journal of Systems and Software, 2017, 127: 302-319.
[22]	LI Lina, WEI Xiaohui, LI Xiang, et al. Burstiness-Aware Elastic Resource Allocation in Stream Data Processing[J]. Chinese Journal of Computers, 2018, 41(10): 2193-2208.
	李丽娜, 魏晓辉, 李翔, 等. 流数据处理中负载突发感知的弹性资源分配[J]. 计算机学报, 2018, 41(10): 2193-2208.
[23]	PALINO T, NARKHEDE N, SHAPIRA G. Kafka: The Definitive Guide[M]. California: O’Reilly Media, 2017.

TRC_ARC取值范围	TRC_ANBR取值范围	伸缩判定结果
[-1,-MAX_ABS_ARC)	—	扩展
[-MAX_ABS_ARC, MAX_ABS_ARC]	[0, MIN_ABS_ANBR)	扩展
	[MIN_ABS_ANBR, 1-MIN_ABS_ANBR]	不变
	(1-MIN_ABS_ANBR, 1]	收缩
(MAX_ABS_ARC, 1]	—	收缩

编号	CPU			内存	硬盘
编号	型号	主频	核数	内存	硬盘
1	IntelXeonE5-2680v4	2.40 GHz	56	256 GB	机械2.5 TB×10
2	IntelXeonE5-2680v4	2.40 GHz	56	256 GB	机械2.5 TB×10
3	IntelXeonE5-2609v2	2.50 GHz	8	128 GB	机械2.0 TB×10

面向高速网络流实时处理的资源自适应伸缩方法

Resource Adaptive Scaling Method for Real-Time Processing of High-Speed Network Streaming

RichHTML

PDF (PC)

可视化

摘要/Abstract

引用本文

使用本文

图/表 12

参考文献 23

相关文章 15

编辑推荐

Metrics

本文评价

[1]	孙隽丰, 李成海, 宋亚飞. ACCQPSO：一种改进的量子粒子群优化算法及其应用[J]. 信息网络安全, 2024, 24(4): 574-586.
[2]	张选, 万良, 罗恒, 杨阳. 基于两阶段图学习的僵尸网络自动化检测方法[J]. 信息网络安全, 2024, 24(12): 1933-1947.
[3]	沈华, 田晨, 郭森森, 慕志颖. 基于对抗性机器学习的网络入侵检测方法研究[J]. 信息网络安全, 2023, 23(8): 66-75.
[4]	李志华, 王志豪. 基于LCNN和LSTM混合结构的物联网设备识别方法[J]. 信息网络安全, 2023, 23(6): 43-54.
[5]	王浩洋, 李伟, 彭思维, 秦元庆. 一种基于集成学习的列车控制系统入侵检测方法[J]. 信息网络安全, 2022, 22(5): 46-53.
[6]	蔡满春, 王腾飞, 岳婷, 芦天亮. 基于ARF的Tor网站指纹识别技术[J]. 信息网络安全, 2021, 21(4): 39-48.
[7]	余晴, 郑崇辉, 杜晔. 面向云平台虚拟层的安全态势评估关键技术研究[J]. 信息网络安全, 2020, 20(7): 53-59.
[8]	张新跃, 胡安磊, 李炬嵘, 冯燕春. 一种自适应的异常流量检测方法[J]. 信息网络安全, 2020, 20(12): 28-32.
[9]	顾兆军, 郝锦涛, 周景贤. 基于改进双线性卷积神经网络的恶意网络流量分类算法[J]. 信息网络安全, 2020, 20(10): 67-74.
[10]	陈良臣, 刘宝旭, 高曙. 网络攻击检测中流量数据抽样技术研究[J]. 信息网络安全, 2019, 19(8): 22-28.
[11]	朱毅, 陈兴蜀, 陈敬涵, 邵国林. 基于模糊综合评价模型的DNS健康度评估[J]. 信息网络安全, 2018, 18(4): 65-71.
[12]	高洪涛, 陆伟, 杨余旺. 基于自适应调节核函数的图像显著区域提取方法[J]. 信息网络安全, 2018, 18(2): 54-60.
[13]	魏书宁, 陈幸如, 唐勇, 刘慧. AR-HELM算法在网络流量分类中的应用研究[J]. 信息网络安全, 2018, 18(1): 9-14.
[14]	闵祥参, 张雪锋. 一种新的云计算指纹中心点定位算法研究[J]. 信息网络安全, 2017, 17(7): 59-65.
[15]	宋淑男, 杨震. 密钥提取中降低密钥不一致率的量化方法研究[J]. 信息网络安全, 2017, 17(4): 46-52.