面向隐蔽通信检测的DoH加密流量行为分析与识别机制研究

doi:10.3969/j.issn.1671-1122.2026.09.012

摘要/Abstract

摘要：

针对基于超文本传输安全协议的域名系统传输机制（DoH）被用于构建隐蔽信道的问题，以及现有检测方法在跨层关联不足和易受类别分布变化影响等方面的缺陷，文章提出多层行为特征融合检测方法MBF-DoH。该方法融合了域名系统查询、传输层安全协议握手与流量时序特征，借助双向长短期记忆网络和DoH语义关系图完成时空联合识别，并通过类别敏感损失、关系保持和原型约束适应新型攻击。实验结果表明，在公开数据集和自采集数据集上，该方法的F1值分别为98.59%和86.60%，均衡准确率分别为97.80%和96.92%。增量更新后，针对新攻击的F1值达到95.27%，针对已学习的攻击的F1值仅下降了0.46个百分点，具备较好的检测与持续适应能力。

关键词: DoH隐蔽通信, 跨层行为一致性, 时空联合识别, 类别不平衡, 增量学习

Abstract:

To address the misuse of domain name system over HTTPS (DoH) for establishing covert channels, as well as the limitations of existing detection methods in cross-layer correlation and robustness to changes in class distribution, a multi-level behavioral feature fusion detection method, MBF-DoH, was proposed. The method integrated DNS query, TLS handshake, and traffic temporal features, and combined a bidirectional long short-term memory network with a DoH semantic relation graph to perform joint spatiotemporal detection. It further incorporated class-sensitive loss, relation preservation, and prototype constraints to adapt to emerging attacks. On the public dataset and the self-collected dataset, F1-score is 98.59% and 86.60%, and balanced accuracy is 97.80% and 96.92%, respectively. It can be seen that after the additional update, the F1-score of the new attack reached 95.27%, and the F1-score of the previously learned attack only decreased by 0.46 percentage points, which has stable detection power and the ability to continue to respond.

Key words: DoH covert communication, cross-layer behavioral consistency, spatiotemporal joint detection, class imbalance, incremental learning

中图分类号:

TP309

苏楠, 王惠惠, 郭昊, 王晓东, 余英杰. 面向隐蔽通信检测的DoH加密流量行为分析与识别机制研究[J]. 信息网络安全, 2026, 26(9): 1468-1480.

Su Nan, Wang Huihui, Guo Hao, Wang Xiaodong, Yu Yingjie. Research on behavioral analysis and recognition mechanism of DoH encrypted traffic for covert communication detection[J]. Netinfo Security, 2026, 26(9): 1468-1480.

图/表 13

图1

图2

图3

表1

表2

表3

表4

图4

表5

表6

表7

表8

表9

参考文献 38

[1]	王影, 王钢, 高雲鹏, 等. 基于深度学习的加密流量分类研究综述[J]. 计算机工程与应用, 2025, 61(21): 61-80. doi: 10.3778/j.issn.1002-8331.2501-0218
[2]	孙剑文, 张斌, 常禾雨. 网络异常检测中的流量表示研究[J]. 通信学报, 2025, 46(1): 192-209. doi: 10.11959/j.issn.1000-436x.2025003
[3]	Cajaraville-Aboy D, Moure-Garrido M, Beis-Penedo C, et al. CO-DEFEND: continuous decentralized federated learning for secure DoH-based threat detection[J]. Computer Networks, 2026, 276: 111961. doi: 10.1016/j.comnet.2025.111961 URL
[4]	陈良臣, 傅德印, 刘宝旭, 等. 面向网络加密流量的增量式入侵检测关键技术研究综述[J]. 计算机工程, 2025, 51(12): 18-30. doi: 10.19678/j.issn.1000-3428.0252468
[5]	Ginige Y, Silva B, Dahanayaka T, et al. TrafficLLM: LLMs for improved open-set encrypted traffic analysis[J]. Computer Networks, 2026, 274: 111847. doi: 10.1016/j.comnet.2025.111847 URL
[6]	Kuang Boyu, Chen Yuchi, Gao Yansong, et al. META: multi-classified encrypted traffic anomaly detection with fine-grained flow and interaction analysis[J]. Computer Communications, 2025, 243: 108333. doi: 10.1016/j.comcom.2025.108333 URL
[7]	Sha M, Binbusayyis A. Implementing a hybrid deep learning technique for detecting malicious DNS over HTTPS (DoH) traffic[J]. The Journal of Supercomputing, 2025, 81(12): 1241. doi: 10.1007/s11227-025-07715-8
[8]	王钢, 高雲鹏, 杨松儒, 等. 基于深度学习的加密恶意流量检测方法研究综述[J]. 信息网络安全, 2025, 25(8): 1276-1301.
[9]	Lin Wanshuang, Xia Chunhe, Wang Tianbo, et al. Enhancing encrypted traffic analysis via source APIs: a robust approach for malicious traffic detection[J]. Computers & Security, 2025, 156: 104529. doi: 10.1016/j.cose.2025.104529 URL
[10]	Mitsuhashi R, Jin Yong, Iida K, et al. DGA-based malware communication detection from DoH traffic using hierarchical machine learning analysis[J]. IEICE Transactions on Information and Systems, 2025, E108.D(6): 526-534. doi: 10.1587/transinf.2024NTP0004 URL
[11]	Alserhani F. Analysis of encrypted network traffic for enhancing cyber-security in dynamic environments[J]. Applied Artificial Intelligence, 2024, 38: 2381882. doi: 10.1080/08839514.2024.2381882 URL
[12]	Cui Susu, Han Xueying, Dong Cong, et al. MVDet: encrypted malware traffic detection via multi-view analysis[J]. Journal of Computer Security, 2024, 32(6): 533-555. doi: 10.3233/JCS-230024 URL
[13]	Wu Shaoqian, Wang Wei, Ding Zhanmeng. Detecting malicious DoH traffic: leveraging small sample analysis and adversarial networks for detection[J]. Journal of Information Security and Applications, 2024, 84: 103827. doi: 10.1016/j.jisa.2024.103827 URL
[14]	Jung I S, Song Y R, Jilcha L A, et al. Enhanced encrypted traffic analysis leveraging graph neural networks and optimized feature dimensionality reduction[J]. Symmetry, 2024, 16(6): 733. doi: 10.3390/sym16060733 URL
[15]	王涛, 翟江涛, 王子豪, 等. 低维快速DNS-over-HTTPS隧道流量检测方法[J]. 网络空间安全科学学报, 2024, 2(6): 123-130. doi: 10.20172/j.issn.2097-3136.240609
[16]	吴桦, 全婧文, 江初晴, 等. 基于单向流特征的主干网DoH隧道流量检测方法[J]. 网络空间安全科学学报, 2024, 2(6): 110-122. doi: 10.20172/j.issn.2097-3136.240608
[17]	李道全, 任大用, 李腾, 等. 基于优化支持向量机的DoH隧道流量检测[J]. 计算机应用与软件, 2025, 42(12): 121-130, 150.
[18]	郑智强, 王锐棋, 范子静, 等. DNS隧道检测技术研究综述[J]. 计算机应用, 2025, 45(7): 2079-2091. doi: 10.11772/j.issn.1001-9081.2024070972
[19]	顾国民, 陈文浩, 黄伟达. 一种基于多模型融合的隐蔽隧道和加密恶意流量检测方法[J]. 信息网络安全, 2024, 24(5): 694-708.
[20]	Yan Xiaodan. Deep learning-based efficient analysis for encrypted traffic[J]. Applied Sciences, 2023, 13(21): 11776. doi: 10.3390/app132111776 URL
[21]	Aggarwal A, Kumar M. An ensemble framework for detection of DNS-over-HTTPS (DoH) traffic[J]. Multimedia Tools and Applications, 2024, 83(11): 32945-32972. doi: 10.1007/s11042-023-16956-9
[22]	Moure-Garrido M, Campo C, Garcia-Rubio C. Real time detection of malicious DoH traffic using statistical analysis[J]. Computer Networks, 2023, 234: 109910. doi: 10.1016/j.comnet.2023.109910 URL
[23]	Hong Yueping, Li Qi, Yang Yanqing, et al. Graph based encrypted malicious traffic detection with hybrid analysis of multi-view features[J]. Information Sciences, 2023, 644: 119229. doi: 10.1016/j.ins.2023.119229 URL
[24]	Gonzalez C L F, Lin P C. Malicious network traffic detection for DNS over HTTPS using machine learning algorithms[J]. APSIPA Transactions on Signal and Information Processing, 2023, 12(2): 1-24.
[25]	侯剑, 鲁辉, 刘方爱, 等. 加密恶意流量检测及对抗综述[J]. 软件学报, 2024, 35(1): 333-355.
[26]	Alzighaibi A R. Detection of DoH traffic tunnels using deep learning for encrypted traffic classification[J]. Computers, 2023, 12(3): 47. doi: 10.3390/computers12030047 URL
[27]	Sirinam P, Imani M, Juarez M, et al. Deep fingerprinting: undermining website fingerprinting defenses with deep learning[C]// The 2018 ACM SIGSAC Conference on Computer and Communications Security. New York: ACM, 2018: 1928-1943.
[28]	Mirsky Y, Doitshman T, Elovici Y, et al. Kitsune: an ensemble of autoencoders for online network intrusion detection[EB/OL]. [2026-01-05]. https://www.ndss-symposium.org/wp-content/uploads/2018/02/ndss2018_03A-3_Mirsky_paper.pdf.
[29]	孙剑文, 张斌, 司念文, 等. 基于知识蒸馏的轻量化恶意流量检测方法[J]. 信息网络安全, 2025, 25(6): 859-871.
[30]	Niktabe S, Lashkari A H, Roudsari A H. Unveiling DoH tunnel: toward generating a balanced DoH encrypted traffic dataset and profiling malicious behavior using inherently interpretable machine learning[J]. Peer-to-Peer Networking and Applications, 2024, 17(1): 507-531. doi: 10.1007/s12083-023-01597-4
[31]	Shen Meng, Zhang Jinpeng, Zhu Liehuang, et al. Accurate decentralized application identification via encrypted traffic analysis using graph neural networks[J]. IEEE Transactions on Information Forensics and Security, 2021, 16: 2367-2380. doi: 10.1109/TIFS.10206 URL
[32]	Premkumar A, Schneider M, Spivey C, et al. Graph representation learning for context-aware network intrusion detection[EB/OL]. [2026-01-05]. https://doi.org/10.1117/12.2663162.
[33]	唐瑛, 王宝会. 基于图神经网络的SSL/TLS加密恶意流量检测算法研究[J]. 计算机科学, 2024, 51(9): 365-370. doi: 10.11896/jsjkx.230800079
[34]	Ma Xiaobo, Qu Jian, Li Jianfeng, et al. Pinpointing hidden IoT devices via spatial-temporal traffic fingerprinting[C]// IEEE INFOCOM 2020-IEEE Conference on Computer Communications. New York: IEEE, 2020: 894-903.
[35]	Msadek N, Soua R, Engel T. IoT device fingerprinting: machine learning based encrypted traffic analysis[C]// 2019 IEEE wireless communications and networking conference (WCNC). New York: IEEE, 2019: 1-8.
[36]	Wang Wei, Zhu Ming, Wang Jinlin, et al. End-to-end encrypted traffic classification with one-dimensional convolution neural networks[C]// 2017 IEEE International Conference on Intelligence and Security Informatics (ISI). New York: IEEE, 2017: 43-48.
[37]	Lyu Minzhao, Gharakheili H H, Sivaraman V. A survey on DNS encryption: current development, malware misuse, and inference techniques[J]. ACM Computing Surveys, 2023, 55(8): 1-28.
[38]	Mungwarakarama I, Wang Yichuan, Hei Xinhong, et al. FSDC: flow samples and dimensions compression for efficient detection of DNS-over-HTTPS tunnels[J]. Electronics, 2024, 13(13): 2604. doi: 10.3390/electronics13132604 URL

配置类别	配置项	具体配置
硬件环境	处理器	AMD 锐龙R9 9950X3D（主频5.7 GHz，16核心32线程）
	内存	G.SKILL芝奇幻锋戟64 GB（32GB?2）DDR5 6000 MHz RGB
	GPU	华硕ROG夜神 GeForce RTX 5090-O32G-GAMING
	存储	SAMSUNG三星 9100 PRO 2 TB PCI-E 5.0 M.2固态硬盘
软件环境	操作系统	Ubuntu 20.04 LTS
	深度学习框架	PyTorch 1.12.1
	图神经网络库	PyTorch Geometric 2.0.4
	流量处理工具	Scapy 2.4.5、dpkt 1.9.8

数据集名称	流量类型	样本数量 /条	DoH 会话数 /个	隐蔽通信会话数 /个	正常DoH 会话数 /个	采集时长 /h	数据规模 /GB	数据划分
CIRA-CIC-DoHBrw- 2020	DoH流量+非DoH流量	1167136	269643	249836	19807	—	15.3	按客户端、攻击工具和采集批次分组后，以7:1:2划分训练集、验证集和测试集
自采集数据集（训练集）	混合流量	458920	102347	8456	93891	48	6.8	按客户端和采集时段分组，其中80%用于训练、20%用于验证
自采集数据集（测试集）	混合流量	114730	25562	2114	23448	24	1.7	按时间独立采集，仅用于测试

方法	类型	特征表示	核心技术
RF-Statistical^[26]	传统机器学习	统计特征（流级别）	随机森林分类器
CNN-Image^[35]	深度学习	灰度图像（流量可视化）	一维卷积神经网络
LSTM-Temporal^[20]	深度学习	时序特征（数据包序列）	长短期记忆网络
GNN-Topology^[30]	深度学习	图结构特征（流交互图）	图神经网络
Ensemble-ML^[19]	混合方法	统计+时序特征	多分类器集成（RF+GBDT+XGBoost）
MBF-DoH （本文方法）	深度学习	多层次特征融合	双向LSTM+图注意力网络

方法	数据集	Accuracy	Precision	Recall	F1值	FPR	BA	AUC-ROC
RF-Statistical	CIRA-CIC-DoHBrw-2020	89.19%	99.52%	88.76%	93.83%	5.4%	91.67%	0.9268
RF-Statistical	自采集测试集	92.25%	51.89%	85.91%	64.70%	7.18%	89.37%	0.9012
CNN-Image	CIRA-CIC-DoHBrw-2020	92.16%	99.64%	91.87%	95.60%	4.16%	93.86%	0.9521
CNN-Image	自采集测试集	93.69%	57.74%	88.34%	69.83%	5.83%	91.25%	0.9234
LSTM-Temporal	CIRA-CIC-DoHBrw-2020	92.84%	99.67%	92.58%	95.99%	3.87%	94.36%	0.9603
LSTM-Temporal	自采集测试集	94.46%	61.25%	89.76%	72.81%	5.12%	92.32%	0.9378
GNN-Topology	CIRA-CIC-DoHBrw-2020	94.37%	99.73%	94.18%	96.87%	3.24%	95.47%	0.9698
GNN-Topology	自采集测试集	95.32%	65.50%	91.83%	76.46%	4.36%	93.73%	0.9512
Ensemble-ML	CIRA-CIC-DoHBrw-2020	95.76%	99.80%	95.62%	97.66%	2.45%	96.59%	0.9812
Ensemble-ML	自采集测试集	96.04%	69.39%	93.27%	79.57%	3.71%	94.78%	0.9647
MBF-DoH（本文方法）	CIRA-CIC-DoHBrw-2020	97.42%	99.86%	97.35%	98.59%	1.76%	97.80%	0.9926
MBF-DoH（本文方法）	自采集测试集	97.54%	78.75%	96.18%	86.60%	2.34%	96.92%	0.9851

攻击类型	更新阶段	Precision	Recall	F1值	F1值变化/百分点
dnstt隧道	更新前	68.45%	58.92%	63.32%	—
dnstt隧道	更新后	95.71%	94.25%	94.98%	+31.66
DNSExfiltrator隧道	更新前	66.18%	62.34%	64.21%	—
DNSExfiltrator隧道	更新后	96.28%	95.46%	95.87%	+31.66
自定义Python隧道	更新前	64.83%	65.76%	65.29%	—
自定义Python隧道	更新后	95.63%	94.31%	94.97%	+29.68
新增攻击平均	更新前	66.49%	62.34%	64.27%	—
新增攻击平均	更新后	95.87%	94.67%	95.27%	+31.00
原有攻击模式	更新前	98.12%	97.35%	97.73%	—
原有攻击模式	更新后	97.65%	96.89%	97.27%	-0.46