Abstract:

In view of the massive data intrusion detection, this paper designs and implements a network traffic anomaly detection system based on Spark framework. Data preprocessing use Python and Python data, an upgraded version of the IPython implementation. Anomaly detection uses K-means predict and classify flow records represent the type of attack. In order to avoid time overhead uses traditional distributed computing framework, this paper designs and implements an anomaly K-means detection method under the framework of Spark. The method storages temporary data into memory rather than the hard drive, and improve the computational efficiency. In order to solve the problem of K value select difficult, through the Spark iterative calculation and comparison of the different K-means value of the K algorithm in the cluster center to all points in the cluster average value of all points, to achieve the best selection of K value. Finally, the performance and function of the system are tested. The test result shows that the system achieves the predetermined design requirements, and has high computational efficiency and detection accuracy.

Key words: network traffic detection, Spark, guiding learning