Abstract:

In stream processing, static resource allocation is difficult to cope with real-time changes and sudden streaming data loads, so elasticity mechanisms need to be introduced. However, when determining the elastic scaling timing and scaling strategy, if the balance between the cost and benefit of scaling is not fully considered, frequent resource adjustments will be triggered, resulting in the system becoming unstable or less efficient instead. To solve this problem, this paper proposed a resource adaptive scaling algorithm, which determined the direction and scale of resource scaling by analyzing the scale of streaming data load and resource usage. At the same time, the algorithm proposed a maximum average processing throughput method, which took the processing throughput before and after the scaling operation as a quantitative index to evaluate the overhead and benefit brought by resource scaling, optimize the scaling strategy, and avoid unnecessary frequent adjustment of resources. Based on this algorithm, this paper designed a network flow elasticity processing framework, which realized the flexible expansion of the framework and the dynamic adjustment of resources. The framework is tested in different network bandwidth scenarios, and the experimental results show that the algorithm can effectively weigh the overhead and benefit, and accurately realize the resource scaling, and after applying the algorithm, the resource utilization of the framework is effectively increased more than 40%, which is able to satisfy the performance requirements of high-speed network stream processing.

Key words: network streaming, stream processing, adaptive, resource scaling