Abstract:

At present, traditional requirement modeling relies on manual efforts, is inefficient, and prone to semantic gaps, while the quality of unified modeling language model generation by large language model (LLM) remains unstable. To address these issues, this study proposed a quality assessment framework for LLM-generated statecharts by combining a structured requirements modeling description with a state-diagram quality dimension function. The structured requirements modeling description used five core components: system overview, pattern hierarchy, state definitions, transition logic, and safety constraints, aiming to convert natural language requirements into structured semantic inputs for LLM to generate state diagrams. State-diagram quality dimension function constructed a formalized evaluation system for statecharts across four dimensions, including executability, syntactic quality, semantic quality, and practical quality. This framework overcomes the limitations of existing research in behavioral model quality assessment, which relies on single metrics and lacks systematic verification mechanisms. Experimental evaluation is conducted using two large models, GPT-4.0 and DeepSeek-R1, on 100 cross-domain requirements documents. Results demonstrate that GPT-4.0 achieves an average semantic coverage of 82%, while DeepSeek-R1 reaches 91%, confirming the applicability of LLM in requirements-driven modeling. The analysis further reveals distinct strengths and characteristics of different models across various quality metrics. This study provides a systematic methodology for evaluating LLM-generated models and offers practical references for model selection and optimization in real-world applications.

Key words: LLM, code generation, statechart diagram, unified modeling language model