Research on behavioral analysis and recognition mechanism of DoH encrypted traffic for covert communication detection

doi:10.3969/j.issn.1671-1122.2026.09.012

Abstract

Abstract:

To address the misuse of domain name system over HTTPS (DoH) for establishing covert channels, as well as the limitations of existing detection methods in cross-layer correlation and robustness to changes in class distribution, a multi-level behavioral feature fusion detection method, MBF-DoH, was proposed. The method integrated DNS query, TLS handshake, and traffic temporal features, and combined a bidirectional long short-term memory network with a DoH semantic relation graph to perform joint spatiotemporal detection. It further incorporated class-sensitive loss, relation preservation, and prototype constraints to adapt to emerging attacks. On the public dataset and the self-collected dataset, F1-score is 98.59% and 86.60%, and balanced accuracy is 97.80% and 96.92%, respectively. It can be seen that after the additional update, the F1-score of the new attack reached 95.27%, and the F1-score of the previously learned attack only decreased by 0.46 percentage points, which has stable detection power and the ability to continue to respond.

Key words: DoH covert communication, cross-layer behavioral consistency, spatiotemporal joint detection, class imbalance, incremental learning

CLC Number:

TP309

Su Nan, Wang Huihui, Guo Hao, Wang Xiaodong, Yu Yingjie. Research on behavioral analysis and recognition mechanism of DoH encrypted traffic for covert communication detection[J]. Netinfo Security, 2026, 26(9): 1468-1480.

Figures/Tables 13

References 38

[1]	王影, 王钢, 高雲鹏, 等. 基于深度学习的加密流量分类研究综述[J]. 计算机工程与应用, 2025, 61(21): 61-80. doi: 10.3778/j.issn.1002-8331.2501-0218
[2]	孙剑文, 张斌, 常禾雨. 网络异常检测中的流量表示研究[J]. 通信学报, 2025, 46(1): 192-209. doi: 10.11959/j.issn.1000-436x.2025003
[3]	Cajaraville-Aboy D, Moure-Garrido M, Beis-Penedo C, et al. CO-DEFEND: continuous decentralized federated learning for secure DoH-based threat detection[J]. Computer Networks, 2026, 276: 111961. doi: 10.1016/j.comnet.2025.111961 URL
[4]	陈良臣, 傅德印, 刘宝旭, 等. 面向网络加密流量的增量式入侵检测关键技术研究综述[J]. 计算机工程, 2025, 51(12): 18-30. doi: 10.19678/j.issn.1000-3428.0252468
[5]	Ginige Y, Silva B, Dahanayaka T, et al. TrafficLLM: LLMs for improved open-set encrypted traffic analysis[J]. Computer Networks, 2026, 274: 111847. doi: 10.1016/j.comnet.2025.111847 URL
[6]	Kuang Boyu, Chen Yuchi, Gao Yansong, et al. META: multi-classified encrypted traffic anomaly detection with fine-grained flow and interaction analysis[J]. Computer Communications, 2025, 243: 108333. doi: 10.1016/j.comcom.2025.108333 URL
[7]	Sha M, Binbusayyis A. Implementing a hybrid deep learning technique for detecting malicious DNS over HTTPS (DoH) traffic[J]. The Journal of Supercomputing, 2025, 81(12): 1241. doi: 10.1007/s11227-025-07715-8
[8]	王钢, 高雲鹏, 杨松儒, 等. 基于深度学习的加密恶意流量检测方法研究综述[J]. 信息网络安全, 2025, 25(8): 1276-1301.
[9]	Lin Wanshuang, Xia Chunhe, Wang Tianbo, et al. Enhancing encrypted traffic analysis via source APIs: a robust approach for malicious traffic detection[J]. Computers & Security, 2025, 156: 104529. doi: 10.1016/j.cose.2025.104529 URL
[10]	Mitsuhashi R, Jin Yong, Iida K, et al. DGA-based malware communication detection from DoH traffic using hierarchical machine learning analysis[J]. IEICE Transactions on Information and Systems, 2025, E108.D(6): 526-534. doi: 10.1587/transinf.2024NTP0004 URL
[11]	Alserhani F. Analysis of encrypted network traffic for enhancing cyber-security in dynamic environments[J]. Applied Artificial Intelligence, 2024, 38: 2381882. doi: 10.1080/08839514.2024.2381882 URL
[12]	Cui Susu, Han Xueying, Dong Cong, et al. MVDet: encrypted malware traffic detection via multi-view analysis[J]. Journal of Computer Security, 2024, 32(6): 533-555. doi: 10.3233/JCS-230024 URL
[13]	Wu Shaoqian, Wang Wei, Ding Zhanmeng. Detecting malicious DoH traffic: leveraging small sample analysis and adversarial networks for detection[J]. Journal of Information Security and Applications, 2024, 84: 103827. doi: 10.1016/j.jisa.2024.103827 URL
[14]	Jung I S, Song Y R, Jilcha L A, et al. Enhanced encrypted traffic analysis leveraging graph neural networks and optimized feature dimensionality reduction[J]. Symmetry, 2024, 16(6): 733. doi: 10.3390/sym16060733 URL
[15]	王涛, 翟江涛, 王子豪, 等. 低维快速DNS-over-HTTPS隧道流量检测方法[J]. 网络空间安全科学学报, 2024, 2(6): 123-130. doi: 10.20172/j.issn.2097-3136.240609
[16]	吴桦, 全婧文, 江初晴, 等. 基于单向流特征的主干网DoH隧道流量检测方法[J]. 网络空间安全科学学报, 2024, 2(6): 110-122. doi: 10.20172/j.issn.2097-3136.240608
[17]	李道全, 任大用, 李腾, 等. 基于优化支持向量机的DoH隧道流量检测[J]. 计算机应用与软件, 2025, 42(12): 121-130, 150.
[18]	郑智强, 王锐棋, 范子静, 等. DNS隧道检测技术研究综述[J]. 计算机应用, 2025, 45(7): 2079-2091. doi: 10.11772/j.issn.1001-9081.2024070972
[19]	顾国民, 陈文浩, 黄伟达. 一种基于多模型融合的隐蔽隧道和加密恶意流量检测方法[J]. 信息网络安全, 2024, 24(5): 694-708.
[20]	Yan Xiaodan. Deep learning-based efficient analysis for encrypted traffic[J]. Applied Sciences, 2023, 13(21): 11776. doi: 10.3390/app132111776 URL
[21]	Aggarwal A, Kumar M. An ensemble framework for detection of DNS-over-HTTPS (DoH) traffic[J]. Multimedia Tools and Applications, 2024, 83(11): 32945-32972. doi: 10.1007/s11042-023-16956-9
[22]	Moure-Garrido M, Campo C, Garcia-Rubio C. Real time detection of malicious DoH traffic using statistical analysis[J]. Computer Networks, 2023, 234: 109910. doi: 10.1016/j.comnet.2023.109910 URL
[23]	Hong Yueping, Li Qi, Yang Yanqing, et al. Graph based encrypted malicious traffic detection with hybrid analysis of multi-view features[J]. Information Sciences, 2023, 644: 119229. doi: 10.1016/j.ins.2023.119229 URL
[24]	Gonzalez C L F, Lin P C. Malicious network traffic detection for DNS over HTTPS using machine learning algorithms[J]. APSIPA Transactions on Signal and Information Processing, 2023, 12(2): 1-24.
[25]	侯剑, 鲁辉, 刘方爱, 等. 加密恶意流量检测及对抗综述[J]. 软件学报, 2024, 35(1): 333-355.
[26]	Alzighaibi A R. Detection of DoH traffic tunnels using deep learning for encrypted traffic classification[J]. Computers, 2023, 12(3): 47. doi: 10.3390/computers12030047 URL
[27]	Sirinam P, Imani M, Juarez M, et al. Deep fingerprinting: undermining website fingerprinting defenses with deep learning[C]// The 2018 ACM SIGSAC Conference on Computer and Communications Security. New York: ACM, 2018: 1928-1943.
[28]	Mirsky Y, Doitshman T, Elovici Y, et al. Kitsune: an ensemble of autoencoders for online network intrusion detection[EB/OL]. [2026-01-05]. https://www.ndss-symposium.org/wp-content/uploads/2018/02/ndss2018_03A-3_Mirsky_paper.pdf.
[29]	孙剑文, 张斌, 司念文, 等. 基于知识蒸馏的轻量化恶意流量检测方法[J]. 信息网络安全, 2025, 25(6): 859-871.
[30]	Niktabe S, Lashkari A H, Roudsari A H. Unveiling DoH tunnel: toward generating a balanced DoH encrypted traffic dataset and profiling malicious behavior using inherently interpretable machine learning[J]. Peer-to-Peer Networking and Applications, 2024, 17(1): 507-531. doi: 10.1007/s12083-023-01597-4
[31]	Shen Meng, Zhang Jinpeng, Zhu Liehuang, et al. Accurate decentralized application identification via encrypted traffic analysis using graph neural networks[J]. IEEE Transactions on Information Forensics and Security, 2021, 16: 2367-2380. doi: 10.1109/TIFS.10206 URL
[32]	Premkumar A, Schneider M, Spivey C, et al. Graph representation learning for context-aware network intrusion detection[EB/OL]. [2026-01-05]. https://doi.org/10.1117/12.2663162.
[33]	唐瑛, 王宝会. 基于图神经网络的SSL/TLS加密恶意流量检测算法研究[J]. 计算机科学, 2024, 51(9): 365-370. doi: 10.11896/jsjkx.230800079
[34]	Ma Xiaobo, Qu Jian, Li Jianfeng, et al. Pinpointing hidden IoT devices via spatial-temporal traffic fingerprinting[C]// IEEE INFOCOM 2020-IEEE Conference on Computer Communications. New York: IEEE, 2020: 894-903.
[35]	Msadek N, Soua R, Engel T. IoT device fingerprinting: machine learning based encrypted traffic analysis[C]// 2019 IEEE wireless communications and networking conference (WCNC). New York: IEEE, 2019: 1-8.
[36]	Wang Wei, Zhu Ming, Wang Jinlin, et al. End-to-end encrypted traffic classification with one-dimensional convolution neural networks[C]// 2017 IEEE International Conference on Intelligence and Security Informatics (ISI). New York: IEEE, 2017: 43-48.
[37]	Lyu Minzhao, Gharakheili H H, Sivaraman V. A survey on DNS encryption: current development, malware misuse, and inference techniques[J]. ACM Computing Surveys, 2023, 55(8): 1-28.
[38]	Mungwarakarama I, Wang Yichuan, Hei Xinhong, et al. FSDC: flow samples and dimensions compression for efficient detection of DNS-over-HTTPS tunnels[J]. Electronics, 2024, 13(13): 2604. doi: 10.3390/electronics13132604 URL

配置类别	配置项	具体配置
硬件环境	处理器	AMD 锐龙R9 9950X3D（主频5.7 GHz，16核心32线程）
	内存	G.SKILL芝奇幻锋戟64 GB（32GB?2）DDR5 6000 MHz RGB
	GPU	华硕ROG夜神 GeForce RTX 5090-O32G-GAMING
	存储	SAMSUNG三星 9100 PRO 2 TB PCI-E 5.0 M.2固态硬盘
软件环境	操作系统	Ubuntu 20.04 LTS
	深度学习框架	PyTorch 1.12.1
	图神经网络库	PyTorch Geometric 2.0.4
	流量处理工具	Scapy 2.4.5、dpkt 1.9.8

数据集名称	流量类型	样本数量 /条	DoH 会话数 /个	隐蔽通信会话数 /个	正常DoH 会话数 /个	采集时长 /h	数据规模 /GB	数据划分
CIRA-CIC-DoHBrw- 2020	DoH流量+非DoH流量	1167136	269643	249836	19807	—	15.3	按客户端、攻击工具和采集批次分组后，以7:1:2划分训练集、验证集和测试集
自采集数据集（训练集）	混合流量	458920	102347	8456	93891	48	6.8	按客户端和采集时段分组，其中80%用于训练、20%用于验证
自采集数据集（测试集）	混合流量	114730	25562	2114	23448	24	1.7	按时间独立采集，仅用于测试

方法	类型	特征表示	核心技术
RF-Statistical^[26]	传统机器学习	统计特征（流级别）	随机森林分类器
CNN-Image^[35]	深度学习	灰度图像（流量可视化）	一维卷积神经网络
LSTM-Temporal^[20]	深度学习	时序特征（数据包序列）	长短期记忆网络
GNN-Topology^[30]	深度学习	图结构特征（流交互图）	图神经网络
Ensemble-ML^[19]	混合方法	统计+时序特征	多分类器集成（RF+GBDT+XGBoost）
MBF-DoH （本文方法）	深度学习	多层次特征融合	双向LSTM+图注意力网络

方法	数据集	Accuracy	Precision	Recall	F1值	FPR	BA	AUC-ROC
RF-Statistical	CIRA-CIC-DoHBrw-2020	89.19%	99.52%	88.76%	93.83%	5.4%	91.67%	0.9268
RF-Statistical	自采集测试集	92.25%	51.89%	85.91%	64.70%	7.18%	89.37%	0.9012
CNN-Image	CIRA-CIC-DoHBrw-2020	92.16%	99.64%	91.87%	95.60%	4.16%	93.86%	0.9521
CNN-Image	自采集测试集	93.69%	57.74%	88.34%	69.83%	5.83%	91.25%	0.9234
LSTM-Temporal	CIRA-CIC-DoHBrw-2020	92.84%	99.67%	92.58%	95.99%	3.87%	94.36%	0.9603
LSTM-Temporal	自采集测试集	94.46%	61.25%	89.76%	72.81%	5.12%	92.32%	0.9378
GNN-Topology	CIRA-CIC-DoHBrw-2020	94.37%	99.73%	94.18%	96.87%	3.24%	95.47%	0.9698
GNN-Topology	自采集测试集	95.32%	65.50%	91.83%	76.46%	4.36%	93.73%	0.9512
Ensemble-ML	CIRA-CIC-DoHBrw-2020	95.76%	99.80%	95.62%	97.66%	2.45%	96.59%	0.9812
Ensemble-ML	自采集测试集	96.04%	69.39%	93.27%	79.57%	3.71%	94.78%	0.9647
MBF-DoH（本文方法）	CIRA-CIC-DoHBrw-2020	97.42%	99.86%	97.35%	98.59%	1.76%	97.80%	0.9926
MBF-DoH（本文方法）	自采集测试集	97.54%	78.75%	96.18%	86.60%	2.34%	96.92%	0.9851

攻击类型	更新阶段	Precision	Recall	F1值	F1值变化/百分点
dnstt隧道	更新前	68.45%	58.92%	63.32%	—
dnstt隧道	更新后	95.71%	94.25%	94.98%	+31.66
DNSExfiltrator隧道	更新前	66.18%	62.34%	64.21%	—
DNSExfiltrator隧道	更新后	96.28%	95.46%	95.87%	+31.66
自定义Python隧道	更新前	64.83%	65.76%	65.29%	—
自定义Python隧道	更新后	95.63%	94.31%	94.97%	+29.68
新增攻击平均	更新前	66.49%	62.34%	64.27%	—
新增攻击平均	更新后	95.87%	94.67%	95.27%	+31.00
原有攻击模式	更新前	98.12%	97.35%	97.73%	—
原有攻击模式	更新后	97.65%	96.89%	97.27%	-0.46