A semi-byte second-order Markov chain and convolution-enhanced Transformer-based method for encrypted traffic classification

doi:10.3969/j.issn.1671-1122.2026.07.004

Abstract

Abstract:

To address the limitations in feature representation and insufficient modeling of deep temporal dependencies in encrypted traffic classification, as well as the inadequate integration of convolutional neural networks and Transformer architectures, this paper proposed NSOM-CF, an encrypted traffic classification method that combines a semi-byte second-order Markov chain with a convolution-enhanced Transformer (ConvFormer). By modeling the original traffic byte sequences at the semi-byte level, NSOM-CF constructed a second-order Markov transition probability matrix that reflected byte transition patterns, effectively alleviating matrix sparsity while preserving the dynamic variations and temporal dependencies of traffic. By further mapping the transition matrix into multi-channel feature maps, NSOM-CF leveraged ConvFormer to capture both local spatial features and global contextual dependencies, generating vector representations of the feature maps to achieve accurate encrypted traffic classification. Experimental results show that the proposed method achieves accuracies of 0.9967 and 0.9968 on the ISCX-VPN and USTC-TFC datasets, respectively, demonstrating the effectiveness of NSOM-CF in encrypted traffic classification tasks.

Key words: encrypted traffic classification, Markov chain, sparse reduction, ConvFormer

CLC Number:

TP309

Feng Jingyu, Luo Chen, Yang Shuaiyi, Fan Yu. A semi-byte second-order Markov chain and convolution-enhanced Transformer-based method for encrypted traffic classification[J]. Netinfo Security, 2026, 26(7): 1044-1057.

Figures/Tables 10

References 29

[1]	王钢, 高雲鹏, 杨松儒, 等. 基于深度学习的加密恶意流量检测方法研究综述[J]. 信息网络安全, 2025, 25(8):1276-1301.
[2]	Erman J, Mahanti A, Arlitt M, et al. Identifying and discriminating between web and peer-to-peer traffic in the network core[C]// The 16th International Conference on World Wide Web. New York: ACM, 2007: 883-892.
[3]	Yan Haonan, Li Hui, Xiao Mingchi, et al. PGSM-DPI: precisely guided signature matching of deep packet inspection for traffic analysis[C]// 2019 IEEE Global Communications Conference (GLOBECOM). New York: IEEE, 2019: 1-6.
[4]	张志强, 暴亚东. 融合RF和CNN的异常流量检测算法[J]. 信息网络安全, 2024, 24(11):1655-1664.
[5]	冯景瑜, 张静, 时翌飞. 物联网中具备终端匿名的加密流量双层过滤方法[J]. 西安邮电大学学报, 2023, 28(2):72-81.
[6]	冯景瑜, 王锦康, 张宝军, 等. 基于信任过滤的轻量级加密流量异常检测方案[J]. 西安邮电大学学报, 2023, 28(5):56-66.
[7]	Korczynski M, Duda A. Markov chain fingerprinting to classify encrypted traffic[C]// 2014 IEEE Conference on Computer Communications. New York: IEEE, 2014: 781-789.
[8]	Cai Wei, Gou Gaopeng, Jiang Minghao, et al. MEMG: mobile encrypted traffic classification with markov chains and graph neural network[C]// 2021 IEEE 23rd International Conference on High Performance Computing and Communications. New York: IEEE, 2021: 478-486.
[9]	Lu Zhiying, Xie Hongtao, Liu Chuanbin, et al. Bridging the gap between vision transformers and convolutional neural networks on small datasets[C]// Advances in Neural Information Processing Systems 35. New York: NeurIPS, 2022: 14663-14677.
[10]	Taylor V F, Spolaor R, Conti M, et al. AppScanner: automatic fingerprinting of smartphone apps from encrypted network traffic[C]// 2016 IEEE European Symposium on Security and Privacy. New York: IEEE, 2016: 439-454.
[11]	Shen Meng, Zhang Jinpeng, Zhu Liehuang, et al. Accurate decentralized application identification via encrypted traffic analysis using graph neural networks[J]. IEEE Transactions on Information Forensics and Security, 2021, 16: 2367-2380. doi: 10.1109/TIFS.10206 URL
[12]	Shapira T, Shavitt Y. FlowPic: a generic representation for encrypted traffic classification and applications identification[J]. IEEE Transactions on Network and Service Management, 2021, 18(2): 1218-1232. doi: 10.1109/TNSM.2021.3071441 URL
[13]	Liu Chang, He Longtao, Xiong Gang, et al. FS-Net: a flow sequence network for encrypted traffic classification[C]// IEEE INFOCOM 2019 - IEEE Conference on Computer Communications. New York: IEEE, 2019: 1171-1179.
[14]	Lin Kunda, Xu Xiaolong, Gao Honghao. TSCRNN: a novel classification scheme of encrypted traffic based on flow spatiotemporal features for efficient management of IIoT[J]. Computer Networks, 2021, 190: 107974. doi: 10.1016/j.comnet.2021.107974 URL
[15]	Lotfollahi M, Jafari S M, Shirali H Z R, et al. DeepPacket: a novel approach for encrypted traffic classification using deep learning[J]. Soft Computing, 2020, 24(3): 1999-2012. doi: 10.1007/s00500-019-04030-2
[16]	Liu Ya, Wang Xiao, Qu Bo, et al. ATVITSC: a novel encrypted traffic classification method based on deep learning[J]. IEEE Transactions on Information Forensics and Security, 2024, 19: 9374-9389. doi: 10.1109/TIFS.2024.3433446 URL
[17]	He Hongye, Yang Zhiguo, Chen Xiangning. Payload encoding representation from transformer for encrypted traffic classification[J]. ZTE Communications, 2021, 19(4): 90-97.
[18]	Lin Xinjie, Xiong Gang, Gou Gaopeng, et al. ET-BERT: a contextualized datagram representation with pre-training transformers for encrypted traffic classification[C]// The ACM Web Conference 2022. New York: ACM, 2022: 633-642.
[19]	Liu Ze, Lin Yutong, Cao Yue, et al. Swin transformer: hierarchical vision transformer using shifted windows[C]// 2021 IEEE/CVF International Conference on Computer Vision (ICCV). New York: IEEE, 2021: 9992-10002.
[20]	Li Yehao, Yao Ting, Pan Yingwei, et al. Contextual transformer networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023, 45(2): 1489-1500. doi: 10.1109/TPAMI.2022.3164083 URL
[21]	Wang Wenhai, Xie Enze, Li Xiang, et al. Pyramid vision transformer: a versatile backbone for dense prediction without convolutions[C]// 2021 IEEE/CVF International Conference on Computer Vision (ICCV). New York: IEEE, 2021: 548-558.
[22]	Yang Lu, Guo Songtao, Liu Defang, et al. ConViTML: a convolutional vision transformer-based meta-learning framework for real-time edge network traffic classification[J]. IEEE Transactions on Network and Service Management, 2024, 21(3): 3344-3357. doi: 10.1109/TNSM.2024.3383218 URL
[23]	KemenY. Denumerable markov chains[M]. Heidelberg: Springer-Verlag, 1967.
[24]	Cao Xin, Luo Qin, Liu Jinzhou. Bit-level malicious traffic detection based on markov images and deep learning[C]// 2022 5th International Conference on Pattern Recognition and Artificial Intelligence (PRAI). New York: IEEE, 2022: 1148-1153.
[25]	Tang Zhangguo, Wang Junfeng, Yuan Baoguo, et al. Markov-GAN: markov image enhancement method for malicious encrypted traffic classification[J]. IET Information Security, 2022, 16(6): 442-458. doi: 10.1049/ise2.v16.6 URL
[26]	Sandler M, Howard A, Zhu Menglong, et al. MobileNetV2: inverted residuals and linear bottlenecks[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. New York: IEEE, 2018: 4510-4520.
[27]	Draper-gil G, Lashkari A H, Mamun M S I, et al. Characterization of encrypted and VPN traffic using time-related features[EB/OL]. [2025-11-07]. https://www.scitepress.org/Link.aspx?doi=10.5220/0005740704070414.
[28]	Wang Wei, Zhu Ming, Zeng Xuewen, et al. Malware traffic classification using convolutional neural network for representation learning[C]// 2017 International Conference on Information Networking (ICOIN). New York: IEEE, 2017: 712-717.
[29]	Wang Wei, Zhu Ming, Wang Jinlin, et al. End-to-end encrypted traffic classification with one-dimensional convolution neural networks[C]// 2017 IEEE International Conference on Intelligence and Security Informatics (ISI). New York: IEEE, 2017: 43-48.

数据集	类型	标签
ISCX-VPN	VPN	VPN_VoIP,VPN_FileTransfer,VPN_Chat, VPN_Email,VPN_Streaming
ISCX-VPN	NonVPN	NonVPN_VoIP, NonVPN_FileTransfer, NonVPN_Chat, NonVPN_Email, NonVPN_Streaming
USTC-TFC	Malware	Cridex, Geodo, Htbot, Miuref, Neris, Nsis-ay, Shifu, Tinba, Virut, Zeus
USTC-TFC	Benign	BitTorrent, Facetime, FTP,Gmail, MySQL, Outlook, Skype, SMB,Weibo,WorldOfWarcraft

数据集	方法	Acc	PR	RC	F1
USTC-TFC	Markov-16	0.9812	0.9715	0.9737	0.9754
	Markov-256	0.9701	0.9516	0.9445	0.9441
	TGI	0.9749	0.9462	0.9449	0.9455
	NSOM-Payload	0.9783	0.9493	0.9484	0.9476
	NSOM	0.9885	0.9773	0.9768	0.9790
ISCX-VPN	Markov-16	0.9747	0.9648	0.9609	0.9628
	Markov-256	0.8684	0.8453	0.8176	0.8312
	TGI	0.7172	0.7016	0.6784	0.6898
	NSOM-Payload	0.7446	0.7269	0.7048	0.7157
	NSOM	0.9861	0.9759	0.9782	0.9770

数据集	方法	Acc	PR	RC	F1
USTC-TFC	AppScanner	0.8954	0.8984	0.8968	0.8892
	GraphDApp	0.8789	0.8226	0.8260	0.8234
	FS-Net	0.8846	0.8846	0.8920	0.8840
	DeepPacket	0.9640	0.9650	0.9631	0.9641
	PERT	0.9909	0.9911	0.9910	0.9911
	ET-BERT	0.9929	0.9930	0.9930	0.9930
	NSOM-CF	0.9968	0.9892	0.9889	0.9890
ISCX-VPN	AppScanner	0.7182	0.7339	0.7225	0.7197
	GraphDApp	0.5977	0.6045	0.6220	0.6036
	FS-Net	0.7205	0.7502	0.7238	0.7137
	DeepPacket	0.9329	0.9377	0.9306	0.9321
	PERT	0.9352	0.9400	0.9349	0.9368
	ET-BERT	0.9890	0.9891	0.9890	0.9890
	NSOM-CF	0.9967	0.9851	0.9847	0.9849

数据集	方法	Down	Acc	PR	RC	F1	参数量/ 计算量
USTC-TFC	Baseline	Strdie	0.9740	0.9406	0.9379	0.9392	3.42M/495.59M
	Markov	Strdie	0.9922	0.9824	0.9817	0.9820	3.43M/123.31M
	Markov+CSCA	Strdie	0.9958	0.9876	0.9861	0.9873	3.45M/125.88M
	Markov+CSCA	Pooling	0.9953	0.9868	0.9845	0.9864	3.13M/109.10M
	Markov+CSCA	IRDonw	0.9968	0.9892	0.9889	0.9890	3.51M/149.38M
ISCX-VPN	Baseline	Strdie	0.9051	0.8872	0.8736	0.8768	3.42M/495.59M
	Markov	Strdie	0.9932	0.9823	0.9814	0.9816	3.43M/123.31M
	Markov+CSCA	Strdie	0.9956	0.9835	0.9829	0.9832	3.45M/125.88M
	Markov+CSCA	Pooling	0.9921	0.9782	0.9775	0.9778	3.13M/109.10M
	Markov+CSCA	IRDonw	0.9967	0.9851	0.9847	0.9849	3.51M/149.38M