基于Rsync算法的实时同步模型优化

doi:10.3969/j.issn.1671-1122.2025.03.006

摘要/Abstract

摘要：

针对远程同步Rsync算法在运行过程中校验和计算对比开销大、同步过程繁琐的问题，文章提出一种基于SipHash强校验的远程同步算法RSip_sync。首先，在计算强校验时采用SipHash算法，重构校验和计算函数并对其内部进行分割；然后，更新校验和验证逻辑，以适应SipHash内部分割算法，其简单的运算过程和较少的迭代轮次使得校验和计算与对比开销减少；最后，利用Inotify实时监控文件技术，实时监控指定数据备份目录中文件的更新变化，从而实现实时自动同步数据。在两组数据集上的仿真实验结果表明，相较于Rsync算法，RSip_sync计算差异数据的耗时减少了40.28%，整体的数据同步时间减少了4.91%，同步速度提升了8.86%。实验结果验证了RSip_sync算法的有效性，其采用的强校验和更适合远程同步，降低了块比较的复杂度，可以更快地进行数据同步。

关键词: Rsync, SipHash, 实时数据同步, 云文件

Abstract:

To address the problem of high overheads of computation and comparison of checksums and the complexity of synchronization during the executing of the remote synchronous (Rsync) algorithm, a remote synchronous algorithm with SipHash strong check (RSip_sync) was proposed. First, the SipHash algorithm was adopted while the strong check was calculated. Then the checksum calculation function was reconstructed and its internal structure was also segmented. Second, the verification logic of checksum was updated in order to adapt to the SipHash internal segmentation algorithm whose simple operation and fewer iteration rounds reduced the cost of checksum calculation and comparison. Finally, by the Inotify real-time monitoring file technology, the changes of files in the specified backup directory were monitored in real time so that the real-time automatic synchronization of data could be achieved. The simulation results based on two datasets show that, compared with the Rsync algorithm, RSip_Sync reduces the time by 40.28% for calculating differential data, reduces the synchronization time by 4.91% in overall data synchronization, and improves the synchronization speed by 8.86%. The experimental results have verified the effectiveness of the RSip_sync algorithm which uses strong checksum that is more suitable for remote synchronization, reduces the complexity of block comparison, and enables faster data synchronization.

Key words: Rsync, SipHash, real time data synchronization, cloud files

中图分类号:

TP309

秦金磊, 孙珊珊, 李整. 基于Rsync算法的实时同步模型优化[J]. 信息网络安全, 2025, 25(3): 425-437.

QIN Jinlei, SUN Shanshan, LI Zheng. Optimization of Real-Time Synchronization Model Based on Rsync Algorithm[J]. Netinfo Security, 2025, 25(3): 425-437.

图/表 19

图1

图2

图3

图4

表1

哈希函数在碰撞概率和计算开销方面的比较

哈希函数	碰撞概率	每字节所消耗的时钟周期
MD5	低 $(<{{10}^{-6}})$	5.58
SipHash	低 $(<{{10}^{-6}})$	1.13
Spooky	高 $(\approx 9.92\times {{10}^{-4}})$	0.14
FNV	高 $(\approx 1.09\times {{10}^{-4}})$	1.75
CityHash	高 $(\approx 1.03\times {{10}^{-4}})$	0.23

表1

图5

表2

图6

表3

表4

表5

表6

同步时间对比

数据集	同步方向	同步后数据集大小/B	数据同步时间/s		同步时间减少百分比
数据集	同步方向	同步后数据集大小/B	Rsync	RSip_sync	同步时间减少百分比
cmake	3.16.1 $\to$ 3.17.1	53903360	0.225	0.208	7.56%
php	8.2.15→8.32	164526080	0.679	0.660	2.80%
gdb	8.0→8.1	207421440	0.878	0.790	10.02%
gcc	7.2.0→7.3.0	534650880	2.197	2.069	5.83%
fslhomes-user006	2012-01-01→ 2012-01-02	14865253	0.061	0.060	1.64%
fslhomes-user021	2012-01-06→ 2012-01-08	80737203	0.545	0.537	1.47%
macos	2011-06-23→ 2011-06-30	85933911	0.376	0.357	5.05%
同步时间减少百分比平均值		4.91%

表6

图7

表7

差异数据计算时间对比

数据集	版本同步方向	差异数据计算时间/s		差异数据计算时间减少百分比
数据集	版本同步方向	Rsync	RSip_sync	差异数据计算时间减少百分比
cmake	3.16.1 $\to$ 3.17.1	0.031	0.018	41.94%
php	8.2.15→8.32	0.137	0.068	50.36%
gdb	8.0→8.1	0.238	0.165	30.67%
gcc	7.2.0→7.3.0	0.663	0.467	29.56%
fslhomes-user006	2012-01-01→ 2012-01-02	0.017	0.008	52.94%
fslhomes-user021	2012-01-06→ 2012-01-08	0.026	0.014	46.15%
macos	2011-06-23→ 2011-06-30	0.056	0.039	30.36%
差异数据计算时间减少百分比平均值		40.28%

表7

图8

表8

校验和生成时间占数据同步时间百分比

数据集	版本同步方向	校验和生成时间百分比
数据集	版本同步方向	Rsync	RSip_sync
cmake	3.16.1 $\to$ 3.17.1	12.15%	10.65%
php	8.2.15→8.32	9.54%	8.39%
gdb	8.0→8.1	13.84%	11.56%
gcc	7.2.0→7.3.0	12.98%	10.65%
fslhomes-user006	2012-01-01→ 2012-01-02	8.57%	7.36%
fslhomes-user021	2012-01-06→ 2012-01-08	11.76%	9.43%
macos	2011-06-23→ 2011-06-30	15.36%	12.35%

表8

图9

表9

数据同步速度对比

数据集	版本同步方向	数据同步速度/（MB·s^-1）		数据同步速度提升百分比
数据集	版本同步方向	Rsync	RSip_sync	数据同步速度提升百分比
cmake	3.16.1 $\to$ 3.17.1	233.52	256.88	10.00%
php	8.2.15→8.32	232.75	248.61	6.81%
gdb	8.0→8.1	226.81	251.95	11.08%
gcc	7.2.0→7.3.0	234.41	248.22	5.89%
fslhomes-user006	2012-01-01→ 2012-01-02	234.56	254.69	8.58%
fslhomes-user021	2012-01-06→ 2012-01-08	215.51	241.65	12.13%
macos	2011-06-23→ 2011-06-30	225.98	242.89	7.48%
数据同步速度提升百分比平均值		8.86%

表9

图10

参考文献 26

[1]	HE Qian, ZHAO Bang, WANG Yong, et al. Web File Protection Based on Local Snapshot and Rsync Algorithm[J]. Computer Engineering, 2013, 39 (6): 190-193,199. doi: 10.3969/j.issn.1000-3428.2013.06.042
	何倩, 赵帮, 王勇, 等. 基于本地快照和Rsync算法的Web文件保护[J]. 计算机工程, 2013, 39(6): 190-193,199. doi: 10.3969/j.issn.1000-3428.2013.06.042
[2]	HAN Peiyi, LIU Chuanyi, WANG Jiahui, et al. Research on Data Encryption System and Technology for Cloud Storage[J]. Journal of Communications, 2020, 41 (8): 55-65.
	韩培义, 刘川意, 王佳慧, 等. 面向云存储的数据加密系统与技术研究[J]. 通信学报, 2020, 41(8): 55-65. doi: 10.11959/j.issn.1000-436x.2020140
[3]	CHEN Fei, LI Zhipeng, JIANG Changkun, et al. Cloud Object Storage Synchronization: Design, Analysis, and Implementation[J]. IEEE Transactions on Parallel and Distributed Systems, 2022, 33(12): 4295-4310.
[4]	ZHAO Haohan, HU Xiaoqin, DENG Hongmin, et al. Efficient File Backup Method Based on the Internet[J]. Computer Engineering, 2010, 36 (2): 247-248,251. doi: 10.3969/j.issn.1000-3428.2010.02.087
	赵昊汉, 胡晓勤, 邓洪敏, 等. 基于Internet的高效文件备份方法[J]. 计算机工程, 2010, 36(2): 247-248,251. doi: 10.3969/j.issn.1000-3428.2010.02.087
[5]	SUEL T, NOEL P, TRENDAFILOV D. Improved File Synchronization Techniques for Maintaining Large Replicated Collections over Slow Networks[C]// IEEE. Proceedings of the 20th International Conference on Data Engineering. New York: IEEE, 2004: 153-164.
[6]	LI Shuai, LIU Xiaojie, XU Bing. Research on a Disk Data Synchronization Method Based on Directory Hash Tree[J]. Netinfo Security, 2019, 19(2): 53-59.
	李帅, 刘晓洁, 徐兵. 一种基于目录哈希树的磁盘数据同步方法研究[J]. 信息网络安全, 2019, 19(2): 53-59.
[7]	CAI Shengxin. Design and Implementation of a Backup System Based on Data Deduplication[D]. Beijing: Beijing University of Posts and Telecommunications, 2011.
	蔡盛鑫. 一种基于重复数据删除的备份系统设计与实现[D]. 北京: 北京邮电大学, 2011.
[8]	XIA Wen, ZOU Xiangyu, JIANG Hong, et al. The Design of Fast Content-Defined Chunking for Data Deduplication Based Storage Systems[J]. IEEE Transactions on Parallel and Distributed Systems, 2020, 31(9): 2017-2031.
[9]	TRIDGELL A, MACKERRAS P. The Rsync Algorithm[EB/OL]. [2024-11-01]. https://rsync.samba.org/tech_report/.
[10]	LI Li, LAZOS L. Proofs of Physical Reliability for Cloud Storage Systems[J]. IEEE Transactions on Parallel and Distributed Systems, 2020, 31(5): 1048-1065.
[11]	IRMAK U, MIHAYLOV S, SUEL T. Improved Single-Round Protocols for Remote File Synchronization[C]// IEEE. Proceedings IEEE 24th Annual Joint Conference of the IEEE Computer and Communications Societies. New York: IEEE, 2005: 1665-1676.
[12]	WU Suzhen, LIU Longquan, JIANG Hong, et al. PandaSync: Network and Workload Aware Hybrid Cloud Sync Optimization[C]// IEEE. Proceedings of the 2019 IEEE 39th International Conference on Distributed Computing Systems (ICDCS). New York: IEEE, 2019: 282-292.
[13]	ZHANG Quanu, LI Zhenhua, ZHI Yang, et al. DeltaCFS: Boosting Delta Sync for Cloud Storage Services by Learning from NFS[C]// IEEE. Proceedings of the 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS). New York: IEEE, 2017: 264-275.
[14]	HE Xiao, LI Zhenhua, ZHAI Eenna, et al. Towards Web-Based Delta Synchronization for Cloud Storage Services[C]// IEEE. Proceedings of the 16th USENIX Conference on File and Storage Technologies (FAST 18). New York: IEEE, 2018: 155-168.
[15]	WANG Longxiang, DONG Xiaoshe, ZHANG Xingjun, et al. The Impact of Expected Block Length on Duplicate Data Deletion Rate in Content Partitioning Algorithm[J]. Journal of Xi’an Jiaotong University, 2016, 50 (12): 73-78.
	王龙翔, 董小社, 张兴军, 等. 内容分块算法中预期分块长度对重复数据删除率的影响[J]. 西安交通大学学报, 2016, 50(12): 73-78.
[16]	ZHENG Yaguang, PAN Jiuhui. A Duplicate Data Detection Algorithm Based on Sliding Blocking[J]. Computer Engineering, 2016, 42(2): 38-44. doi: 10.3969/j.issn.1000-3428.2016.02.007
	郑亚光, 潘久辉. 一种基于滑动分块的重复数据检测算法[J]. 计算机工程, 2016, 42(2): 38-44. doi: 10.3969/j.issn.1000-3428.2016.02.007
[17]	XIA Wen, JIANG Hong, FENG Dan, et al. Ddelta: A Deduplication-Inspired Fast Delta Compression Approach[J]. Performance Evaluation, 2014, 79: 258-272.
[18]	HUA Yu, LIU Xue, FENG Dan. Neptune: Efficient Remote Communication Services for Cloud Backups[C]// IEEE. Proceedings of the IEEE INFOCOM 2014-IEEE Conference on Computer Communications. New York: IEEE, 2014: 844-852.
[19]	WU Xiaoyong, YANG Pin, HU Xiaoqin, et al. Differential Backup Method Based on Duplicate Data Elimination[J]. Computer Engineering, 2010, 36 (21): 251-253. doi: 10.3969/j.issn.1000-3428.2010.21.090
	吴晓勇, 杨频, 胡晓勤, 等. 基于重复数据消除的差异备份方法[J]. 计算机工程, 2010, 36(21): 251-253. doi: 10.3969/j.issn.1000-3428.2010.21.090
[20]	LEE G, KO H, PACK S. An Efficient Delta Synchronization Algorithm for Mobile Cloud Storage Applications[J]. IEEE Transactions on Services Computing, 2017, 10(3): 341-351.
[21]	CUI Yong, LAI Zeqi, WANG Xin, et al. QuickSync: Improving Synchronization Efficiency for Mobile Cloud Storage Services[J]. IEEE Transactions on Mobile Computing, 2017, 16(12): 3513-3526.
[22]	RFC1321. The MD5 Message-Digest Algorithm[S]. Fremont: Internet Engineering Task Force (IETF), 1992.
[23]	AUMASSON J P, BERNSTEIN D J. SipHash: A Fast Short-Input PRF[C]// IACR. Proceedings of the International Conference on Cryptology in India. Heidelberg: Springer, 2012: 489-508.
[24]	VAN D S D, MASCETTI L, MOSCICKI J, et al. The Redundancy of Adler32 and TCP Checksums[R]. Geneva: CERN, CERN-IT-Note-2015-006, 2015.
[25]	DAYAN N, WEISS T, DASHEVSKY S, et al. Spooky: Granulating LSM-Tree Compactions Correctly[J]. Proceedings of the VLDB Endowment, 2022, 15(11): 3071-3084.
[26]	ZUO Zhende. Research and Tool Development for Real-Time High-Speed Backup of Structured Databases Based on Rsync[D]. Guangzhou: South China University of Technology, 2017.
	左珍德. 基于Rsync的结构化数据库实时高速备份研究及工具开发[D]. 广州: 华南理工大学, 2017.

编辑推荐 0

Metrics

阅读次数

全文

HTML			PDF

最新录用	在线预览	正式出版	最新录用	在线预览	正式出版
0	0	7	0	0	18

来源	本网站	其他网站

次数	25	0
比例	100%	0%

摘要

最新录用	在线预览	正式出版

0	0	69

	来源	本网站

	次数	69
	比例	100%

事件	事件掩码	描述
IN_CREATE	0x00000100	文件创建
IN_MODIFY	0x00000002	文件修改
IN_DELETE	0x00000200	文件删除
IN_MOVED	0x00000040	文件移动
IN_ATTRIB	0x00000004	文件属性修改

	操作系统	主存/GB
client	Centos 64位	8
server	Centos 64位	8