In the big data environment, network security incidents emerge one after another, and network security has become a hot spot of concern. As a dark data in the new environment, the security log records the important information of the running status of the equipment. Through its analysis, it can grasp the network security situation in real time, and can be used as a security auditing tool for pre-protection and after-accusation, to achieve abnormal events. Aiming at the importance of log auditing and combining the important role of data mining in the field of log analysis, and aiming at the relative lag of processing massive data in a single machine environment, a clustering algorithm based on Hadoop for massive security log is proposed. Firstly, the K-means clustering algorithm is improved based on the maximum and minimum distance (MMD) and the mean value, which overcomes the defect of the traditional K-means algorithm in finding the randomness of the initial cluster center. Secondly, in order to adapt to the massive data. Effectively process, improve the efficiency and speed of clustering, and deploy the improved K-means clustering algorithm on Map/Reduce for iterative calculation. Experiments show that the improved clustering algorithm proposed in this paper is better than other typical methods, and the clustering effect is stable. It has better running speed and speedup ratio in cluster performance.
