Abstract:

In order to solve the problem of performance evaluation and performance optimization during the operation of Spark jobs, this paper proposed a performance evaluation and analysis method of Spark jobs based on hierarchical analysis. Firstly, to address the problem of low accuracy of traditional job type classification affected by feature selection, more realistic CPU and I/O features were selected and combined with K-Means clustering algorithm to build a job classifier to improve the classification accuracy. Secondly, the job workflow was optimized by eliminating operations such as data sorting, disk overflow writing, and file merging during job operation, and the optimized job performance index was used as the evaluation benchmark, making the job operation performance evaluation more objective and general. Afterwards, the performance metrics were quantified and stratified, hierarchical analysis was introduced to calculate their weights, and the performance evaluation model was constructed by combining job classifiers and evaluation benchmarks. Finally, experimental validation was conducted in three aspects: job type classification, workflow optimization method and performance evaluation. The experimental results show the effectiveness of the proposed job type classification and workflow optimization method, as well as the accuracy of the evaluation model.

Key words: Spark, assessment benchmark, quantification, hierarchical analysis