Abstract:

Low-rank adaptation (LoRA) fine-tuning freezes pretrained weights and only optimizes low-rank factor matrices, enabling efficient collaborative training of large models in distributed settings while substantially reducing communication and computation costs. As a result, LoRA has become a key technique for enabling secure data utilization for large-model training. However, the low-rank update mechanism of LoRA allows attackers to target specific subspaces, making distributed LoRA fine-tuning more vulnerable to backdoor attacks. Existing backdoor defenses methods typically rely on complex outlier detection, statistical filtering, or centralized supervision, which introduce significant additional computation and communication overhead and are therefore ill-suited to resource-constrained distributed fine-tuning of large models. To address this challenge, this paper proposed selective aggregation low-rank adaptation (SA-LoRA), a lightweight backdoor defense method that selectively aggregates only the low-rank matrix B during training while keeping matrix A local to each client. This asymmetric protocol drives benign clients to form an approximately null subspace for backdoor features, thereby suppressing backdoor propagation without introducing extra overhead. Experiments on multiple open-source large models under diverse backdoor attack methods, including word-level triggers and synonym-substitution attacks, show that SA-LoRA reduces the backdoor attack success rate on benign clients from 96.93% to 11.53%, while incurring only about a 1% drop in main-task performance and requiring no additional computation or communication resources, achieving a favorable trade-off between security and efficiency.

Key words: decentralized learning, LoRA fine-tuning, large model security, backdoor attacks