Abstract:

With the extensive application of large language model(LLM) in software development, the role in enhancing development efficiency has also introduced new security risks, particularly in the field of cryptography applications that demand high security. This paper proposed an open-source prompt dataset named LLMCryptoSE, containing 460 natural language description prompts of cryptographic scenarios. It aimed to assess the security of code generated by LLM for cryptographic applications. At the same time, through an in-depth analysis of code snippets generated by LLM, this paper primarily evaluated the misuse of cryptographic API, employing the methodology that combined the static analysis tool CryptoGuard with manual review to conduct a detailed evlatuation of 1380 code snippets. The assessment of three mainstream LLM, including ChatGPT 3.5, ERNIE 3.5, and Spark 3.5, revealed that 52.90% of the code snippets contained at least one instance of cryptographic misuse, with Spark 3.5 showing a relatively better performance with a misuse rate of 48.48%. Based on these findings, the study not only reveals the current challenges in cryptographic application security faced by LLM, but also offers a series of recommendations for LLM users and developers to enhance security. These are aims at providing practical guidance for improving the application of LLM in cryptographic fields.

Key words: large language model, cryptographic application security prompts, cryptographic misuse detection