Abstract:

OpenSSL is a popular open source library for cryptography. On August 26,2021 a buffer overflow vulnerability was patched in OpenSSL, which is caused by the fact that the buffer size calculated by the SM2 decryption function could be smaller than the actual plaintext size. This paper firstly analyzed the principle of buffer overflow based on the OpenSSL source code, and then analyzed the feasibility of overflow attack according to the overflow principle. Finally this paper designed an experiment to verify the feasibility of overflow attack. It’s concluded that when SM2 decryption function calculates the size of the buffer to accommodate the plaintext, it doesn’t consider the encoding of the points on the elliptic curve, when the encoding length is smaller than the preset length, resulting in the buffer size being smaller than the actual plaintext size. Attacker can obtain appropriate points by exhaustion and further construct appropriate ciphertext for buffer overflow attack according the above feature and can use the same point to perform buffer overflow attacks on SM2 decryptors holding different key pairs.

Key words: OpenSSL, SM2 encryption and decryption interface, buffer overflow, vulnerability analysis