Abstract:

In recent years, Runtime Application Self-Protection (RASP) has emerged as an embedded defense mechanism widely used to detect and prevent common web application attacks, such as SQL injection, cross-site scripting (XSS), and Java deserialization attacks. However, existing RASP systems often rely on blacklist-based detection, which is prone to evasion and struggles against novel threats. This paper introduced a hybrid system, HP-RASP, which combined heuristic rules and deep learning models to provide adaptive security at runtime. Notably, it incorporated a BERT model into the RASP framework to analyze and detect SQL injection attacks, while employing stack monitoring and blacklist matching to defend against XSS and deserialization attacks. HP-RASP used Java instrumentation to dynamically insert monitoring logic into critical classes and methods, enabling real-time analysis of web requests. The system was evaluated on multiple open-source datasets and compared to the current mainstream RASP system, OpenRASP. Experimental results demonstrate significant improvements in detection accuracy, performance overhead, and robustness over existing approaches. For SQL injection, HP-RASP achieved an accuracy of 81.9%, 1.84 times higher than OpenRASP, with recall and F1 scores also notably surpassing OpenRASP. For XSS protection, HP-RASP achieved a 99.9% recall rate for both reflective and stored XSS attacks, and an 84.6% recall rate for deserialization attacks. HP-RASP also performed well in terms of response time and resource consumption, without significant increases in either metric.

Key words: RASP, BERT model, software security, Java web application