Abstract:

Fuzz testing is one of the most mature and effective methods among the approaches used to mine vulnerabilities for modern software. However, traditional Fuzz testing generally have some problems, such as limited depth of exploring code space or lacking of directivity in generating samples. To alleviate these issues, a kind of framework was proposed to generate samples of browsers by making use of long short term memory (LSTM) network. The framework consists two components: sample generating and Fuzz testing. Firstly, the sample are encoded into vectors which are much easier to implement in LSTM network. This process is called file preprocessing. After finishing the learning period, the network will generate a mound of samples as test set. Then test set will be generated by mutating samples based on traditional mutation strategies. Finally, the test set will be feed into the browser for Fuzz testing. In order to verify the effectiveness of the framework, the learning results, generating sample results and Fuzz results of LSTM network have been analyzed statistically. It is proofed that the proposed framework could satisfy the needs of browser Fuzz generation and overcome the difficulties of insufficient mining depth and lack of directivity in generating samples in traditional browser Fuzz, which was suitable for mining one or several browser vulnerabilities.

Key words: browser Fuzz, deep learning, sample generation, LSTM neural network, file vectorization