Optimization of Real-Time Synchronization Model Based on Rsync Algorithm

doi:10.3969/j.issn.1671-1122.2025.03.006

Abstract

Abstract:

To address the problem of high overheads of computation and comparison of checksums and the complexity of synchronization during the executing of the remote synchronous (Rsync) algorithm, a remote synchronous algorithm with SipHash strong check (RSip_sync) was proposed. First, the SipHash algorithm was adopted while the strong check was calculated. Then the checksum calculation function was reconstructed and its internal structure was also segmented. Second, the verification logic of checksum was updated in order to adapt to the SipHash internal segmentation algorithm whose simple operation and fewer iteration rounds reduced the cost of checksum calculation and comparison. Finally, by the Inotify real-time monitoring file technology, the changes of files in the specified backup directory were monitored in real time so that the real-time automatic synchronization of data could be achieved. The simulation results based on two datasets show that, compared with the Rsync algorithm, RSip_Sync reduces the time by 40.28% for calculating differential data, reduces the synchronization time by 4.91% in overall data synchronization, and improves the synchronization speed by 8.86%. The experimental results have verified the effectiveness of the RSip_sync algorithm which uses strong checksum that is more suitable for remote synchronization, reduces the complexity of block comparison, and enables faster data synchronization.

Key words: Rsync, SipHash, real time data synchronization, cloud files

CLC Number:

TP309

QIN Jinlei, SUN Shanshan, LI Zheng. Optimization of Real-Time Synchronization Model Based on Rsync Algorithm[J]. Netinfo Security, 2025, 25(3): 425-437.

Figures/Tables 19

哈希函数	碰撞概率	每字节所消耗的时钟周期
MD5	低 $(<{{10}^{-6}})$	5.58
SipHash	低 $(<{{10}^{-6}})$	1.13
Spooky	高 $(\approx 9.92\times {{10}^{-4}})$	0.14
FNV	高 $(\approx 1.09\times {{10}^{-4}})$	1.75
CityHash	高 $(\approx 1.03\times {{10}^{-4}})$	0.23

数据集	同步方向	同步后数据集大小/B	数据同步时间/s		同步时间减少百分比
数据集	同步方向	同步后数据集大小/B	Rsync	RSip_sync	同步时间减少百分比
cmake	3.16.1 $\to$ 3.17.1	53903360	0.225	0.208	7.56%
php	8.2.15→8.32	164526080	0.679	0.660	2.80%
gdb	8.0→8.1	207421440	0.878	0.790	10.02%
gcc	7.2.0→7.3.0	534650880	2.197	2.069	5.83%
fslhomes-user006	2012-01-01→ 2012-01-02	14865253	0.061	0.060	1.64%
fslhomes-user021	2012-01-06→ 2012-01-08	80737203	0.545	0.537	1.47%
macos	2011-06-23→ 2011-06-30	85933911	0.376	0.357	5.05%
同步时间减少百分比平均值		4.91%

数据集	版本同步方向	差异数据计算时间/s		差异数据计算时间减少百分比
数据集	版本同步方向	Rsync	RSip_sync	差异数据计算时间减少百分比
cmake	3.16.1 $\to$ 3.17.1	0.031	0.018	41.94%
php	8.2.15→8.32	0.137	0.068	50.36%
gdb	8.0→8.1	0.238	0.165	30.67%
gcc	7.2.0→7.3.0	0.663	0.467	29.56%
fslhomes-user006	2012-01-01→ 2012-01-02	0.017	0.008	52.94%
fslhomes-user021	2012-01-06→ 2012-01-08	0.026	0.014	46.15%
macos	2011-06-23→ 2011-06-30	0.056	0.039	30.36%
差异数据计算时间减少百分比平均值		40.28%

数据集	版本同步方向	校验和生成时间百分比
数据集	版本同步方向	Rsync	RSip_sync
cmake	3.16.1 $\to$ 3.17.1	12.15%	10.65%
php	8.2.15→8.32	9.54%	8.39%
gdb	8.0→8.1	13.84%	11.56%
gcc	7.2.0→7.3.0	12.98%	10.65%
fslhomes-user006	2012-01-01→ 2012-01-02	8.57%	7.36%
fslhomes-user021	2012-01-06→ 2012-01-08	11.76%	9.43%
macos	2011-06-23→ 2011-06-30	15.36%	12.35%

数据集	版本同步方向	数据同步速度/（MB·s^-1）		数据同步速度提升百分比
数据集	版本同步方向	Rsync	RSip_sync	数据同步速度提升百分比
cmake	3.16.1 $\to$ 3.17.1	233.52	256.88	10.00%
php	8.2.15→8.32	232.75	248.61	6.81%
gdb	8.0→8.1	226.81	251.95	11.08%
gcc	7.2.0→7.3.0	234.41	248.22	5.89%
fslhomes-user006	2012-01-01→ 2012-01-02	234.56	254.69	8.58%
fslhomes-user021	2012-01-06→ 2012-01-08	215.51	241.65	12.13%
macos	2011-06-23→ 2011-06-30	225.98	242.89	7.48%
数据同步速度提升百分比平均值		8.86%

References 26

[1]	HE Qian, ZHAO Bang, WANG Yong, et al. Web File Protection Based on Local Snapshot and Rsync Algorithm[J]. Computer Engineering, 2013, 39 (6): 190-193,199. doi: 10.3969/j.issn.1000-3428.2013.06.042
	何倩, 赵帮, 王勇, 等. 基于本地快照和Rsync算法的Web文件保护[J]. 计算机工程, 2013, 39(6): 190-193,199. doi: 10.3969/j.issn.1000-3428.2013.06.042
[2]	HAN Peiyi, LIU Chuanyi, WANG Jiahui, et al. Research on Data Encryption System and Technology for Cloud Storage[J]. Journal of Communications, 2020, 41 (8): 55-65.
	韩培义, 刘川意, 王佳慧, 等. 面向云存储的数据加密系统与技术研究[J]. 通信学报, 2020, 41(8): 55-65. doi: 10.11959/j.issn.1000-436x.2020140
[3]	CHEN Fei, LI Zhipeng, JIANG Changkun, et al. Cloud Object Storage Synchronization: Design, Analysis, and Implementation[J]. IEEE Transactions on Parallel and Distributed Systems, 2022, 33(12): 4295-4310.
[4]	ZHAO Haohan, HU Xiaoqin, DENG Hongmin, et al. Efficient File Backup Method Based on the Internet[J]. Computer Engineering, 2010, 36 (2): 247-248,251. doi: 10.3969/j.issn.1000-3428.2010.02.087
	赵昊汉, 胡晓勤, 邓洪敏, 等. 基于Internet的高效文件备份方法[J]. 计算机工程, 2010, 36(2): 247-248,251. doi: 10.3969/j.issn.1000-3428.2010.02.087
[5]	SUEL T, NOEL P, TRENDAFILOV D. Improved File Synchronization Techniques for Maintaining Large Replicated Collections over Slow Networks[C]// IEEE. Proceedings of the 20th International Conference on Data Engineering. New York: IEEE, 2004: 153-164.
[6]	LI Shuai, LIU Xiaojie, XU Bing. Research on a Disk Data Synchronization Method Based on Directory Hash Tree[J]. Netinfo Security, 2019, 19(2): 53-59.
	李帅, 刘晓洁, 徐兵. 一种基于目录哈希树的磁盘数据同步方法研究[J]. 信息网络安全, 2019, 19(2): 53-59.
[7]	CAI Shengxin. Design and Implementation of a Backup System Based on Data Deduplication[D]. Beijing: Beijing University of Posts and Telecommunications, 2011.
	蔡盛鑫. 一种基于重复数据删除的备份系统设计与实现[D]. 北京: 北京邮电大学, 2011.
[8]	XIA Wen, ZOU Xiangyu, JIANG Hong, et al. The Design of Fast Content-Defined Chunking for Data Deduplication Based Storage Systems[J]. IEEE Transactions on Parallel and Distributed Systems, 2020, 31(9): 2017-2031.
[9]	TRIDGELL A, MACKERRAS P. The Rsync Algorithm[EB/OL]. [2024-11-01]. https://rsync.samba.org/tech_report/.
[10]	LI Li, LAZOS L. Proofs of Physical Reliability for Cloud Storage Systems[J]. IEEE Transactions on Parallel and Distributed Systems, 2020, 31(5): 1048-1065.
[11]	IRMAK U, MIHAYLOV S, SUEL T. Improved Single-Round Protocols for Remote File Synchronization[C]// IEEE. Proceedings IEEE 24th Annual Joint Conference of the IEEE Computer and Communications Societies. New York: IEEE, 2005: 1665-1676.
[12]	WU Suzhen, LIU Longquan, JIANG Hong, et al. PandaSync: Network and Workload Aware Hybrid Cloud Sync Optimization[C]// IEEE. Proceedings of the 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS). New York: IEEE, 2019: 282-292.
[13]	ZHANG Quanu, LI Zhenhua, ZHI Yang, et al. DeltaCFS: Boosting Delta Sync for Cloud Storage Services by Learning from NFS[C]// IEEE. Proceedings of the 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS). New York: IEEE, 2017: 264-275.
[14]	HE Xiao, LI Zhenhua, ZHAI Eenna, et al. Towards Web-Based Delta Synchronization for Cloud Storage Services[C]// IEEE. Proceedings of the 16th USENIX Conference on File and Storage Technologies (FAST 18). New York: IEEE, 2018: 155-168.
[15]	WANG Longxiang, DONG Xiaoshe, ZHANG Xingjun, et al. The Impact of Expected Block Length on Duplicate Data Deletion Rate in Content Partitioning Algorithm[J]. Journal of Xi’an Jiaotong University, 2016, 50 (12): 73-78.
	王龙翔, 董小社, 张兴军, 等. 内容分块算法中预期分块长度对重复数据删除率的影响[J]. 西安交通大学学报, 2016, 50(12): 73-78.
[16]	ZHENG Yaguang, PAN Jiuhui. A Duplicate Data Detection Algorithm Based on Sliding Blocking[J]. Computer Engineering, 2016, 42(2): 38-44. doi: 10.3969/j.issn.1000-3428.2016.02.007
	郑亚光, 潘久辉. 一种基于滑动分块的重复数据检测算法[J]. 计算机工程, 2016, 42(2): 38-44. doi: 10.3969/j.issn.1000-3428.2016.02.007
[17]	XIA Wen, JIANG Hong, FENG Dan, et al. Ddelta: A Deduplication-Inspired Fast Delta Compression Approach[J]. Performance Evaluation, 2014, 79: 258-272.
[18]	HUA Yu, LIU Xue, FENG Dan. Neptune: Efficient Remote Communication Services for Cloud Backups[C]// IEEE. Proceedings of the IEEE INFOCOM 2014-IEEE Conference on Computer Communications. New York: IEEE, 2014: 844-852.
[19]	WU Xiaoyong, YANG Pin, HU Xiaoqin, et al. Differential Backup Method Based on Duplicate Data Elimination[J]. Computer Engineering, 2010, 36 (21): 251-253. doi: 10.3969/j.issn.1000-3428.2010.21.090
	吴晓勇, 杨频, 胡晓勤, 等. 基于重复数据消除的差异备份方法[J]. 计算机工程, 2010, 36(21): 251-253. doi: 10.3969/j.issn.1000-3428.2010.21.090
[20]	LEE G, KO H, PACK S. An Efficient Delta Synchronization Algorithm for Mobile Cloud Storage Applications[J]. IEEE Transactions on Services Computing, 2017, 10(3): 341-351.
[21]	CUI Yong, LAI Zeqi, WANG Xin, et al. QuickSync: Improving Synchronization Efficiency for Mobile Cloud Storage Services[J]. IEEE Transactions on Mobile Computing, 2017, 16(12): 3513-3526.
[22]	RFC1321. The MD5 Message-Digest Algorithm[S]. Fremont: Internet Engineering Task Force (IETF), 1992.
[23]	AUMASSON J P, BERNSTEIN D J. SipHash: A Fast Short-Input PRF[C]// IACR. Proceedings of the International Conference on Cryptology in India. Heidelberg: Springer, 2012: 489-508.
[24]	VAN D S D, MASCETTI L, MOSCICKI J, et al. The Redundancy of Adler32 and TCP Checksums[R]. Geneva: CERN, CERN-IT-Note-2015-006, 2015.
[25]	DAYAN N, WEISS T, DASHEVSKY S, et al. Spooky: Granulating LSM-Tree Compactions Correctly[J]. Proceedings of the VLDB Endowment, 2022, 15(11): 3071-3084.
[26]	ZUO Zhende. Research and Tool Development for Real-Time High-Speed Backup of Structured Databases Based on Rsync[D]. Guangzhou: South China University of Technology, 2017.
	左珍德. 基于Rsync的结构化数据库实时高速备份研究及工具开发[D]. 广州: 华南理工大学, 2017.

事件	事件掩码	描述
IN_CREATE	0x00000100	文件创建
IN_MODIFY	0x00000002	文件修改
IN_DELETE	0x00000200	文件删除
IN_MOVED	0x00000040	文件移动
IN_ATTRIB	0x00000004	文件属性修改

	操作系统	主存/GB
client	Centos 64位	8
server	Centos 64位	8