JBT Journal
Advanced

Raqamli Transformatsiya va Sun’iy Intellekt

PROTOCOL EFFECTS IN BENCHMARK INTRUSION DETECTION EVALUATION

Buriboyev Tolibjon1

1 Alfraganus University

Zenodo DOI: 10.5281/zenodo.22886390

Keywords

measurement protocol intrusion detection evaluation benchmark datasets experimental bias base-rate fallacy reproducibility data leakage temporal splitting

Abstract

Published intrusion detection results routinely report accuracies above 0.99, yet operational deployments do not behave as those figures predict. Existing critiques have established that individual evaluation choices - defective labels, random splitting, shortcut features, rebalanced test sets - inflate reported performance, but each has been examined in isolation, so the size of each effect relative to the others, and whether they compound or overlap, remains unquantified. This study treats the evaluation protocol itself as the object of measurement, crossing four binary protocol factors in a full factorial design over nine detector families on CIC-IDS2017 with an equal hyperparameter search budget in every cell. The permissive protocol yields a macro-averaged F1 of 0.9817 and the strict protocol 0.8214, a gap of 16.3 % relative, of which partitioning accounts for the largest share and labelling the smallest. The effects are not additive: removing the destination-port shortcut costs 0.0407 points after temporal splitting but only 0.0119 before it, because episode overlap and the shortcut encode partly the same information. Method ranking is unstable, three of nine detectors changing position between protocols, so protocol choice can determine which method appears superior. Transfer loss falls as the protocol tightens, supplying a leakage diagnostic that requires no corrected corpus. A reference protocol and reporting checklist are proposed.

Views: 267 Published: 2026-08-28 Pages: 250-256 Volume: 4 Issue: 4

References

  1. Muhammad Eid Balbaa, Mansur Pulatovich Eshov, and Nilufar Ismailova. 2023. The Impacts of Russian Ukrainian War on the Global Economy in the frame of digital banking networks and cyber attacks. In Proceedings of the 6th International Conference on Future Networks & Distributed Systems (ICFNDS '22). Association for Computing Machinery, New York, NY, USA, 137–146. https://doi.org/10.1145/3584202.3584223
  2. G. Apruzzese, P. Laskov, E. Montes de Oca, W. Mallouli, L. Búrdalo Rapa, A. V. Grammatopoulos, and F. Di Franco, “The role of machine learning in cybersecurity,” Digital Threats: Research and Practice, vol. 4, no. 1, art. 8, Mar. 2023, doi: 10.1145/3545574.
  3. R. Sommer and V. Paxson, "Outside the closed world: On using machine learning for network intrusion detection," in Proc. 2010 IEEE Symp. Security and Privacy, Oakland, CA, USA, 2010, pp. 305–316, doi: 10.1109/SP.2010.25.
  4. G. Engelen, V. Rimmer, and W. Joosen, “Troubleshooting an intrusion detection dataset: The CICIDS2017 case study,” in Proc. 2021 IEEE Security and Privacy Workshops (SPW), 2021, pp. 7–12, doi: 10.1109/SPW53761.2021.00009.
  5. L. Liu, G. Engelen, T. Lynar, D. Essam, and W. Joosen, “Error prevalence in NIDS datasets: A case study on CIC-IDS-2017 and CSE-CIC-IDS-2018,” in Proc. 2022 IEEE Conf. Communications and Network Security (CNS), 2022, pp. 254–262, doi: 10.1109/CNS56114.2022.9947235.
  6. Farhod Xoliyarov, Sherzod Gulomov, and Suhrobjon Bozorov. 2024. The Impact of Artificial Neural Network Architecture on Network Attack Detection. In Proceedings of the 7th International Conference on Future Networks and Distributed Systems (ICFNDS '23). Association for Computing Machinery, New York, NY, USA, 532–539. https://doi.org/10.1145/3644713.3644792
  7. R. Flood, G. Engelen, D. Aspinall, and L. Desmet, “Bad design smells in benchmark NIDS datasets,” in Proc. 2024 IEEE 9th European Symp. Security and Privacy (EuroS&P), 2024, pp. 658–675.
  8. Kuvonchbek Rakhimberdiev, Akram Ishnazarov, Piratdin Allayarov, Fayzulla Ollamberganov, Rahman Kamalov, and Madina Matyakubova. 2023. PROSPECTS FOR THE USE OF NEURAL NETWORK MODELS IN THE PREVENTION OF POSSIBLE NETWORK ATTACKS ON MODERN BANKING INFORMATION SYSTEMS BASED ON BLOCKCHAIN TECHNOLOGY IN THE CONTEXT OF THE DIGITAL ECONOMY. In Proceedings of the 6th International Conference on Future Networks & Distributed Systems (ICFNDS '22). Association for Computing Machinery, New York, NY, USA, 592–599. https://doi.org/10.1145/3584202.3584291
  9. Juraev, G.U., Kuvonchbek, R., Toshpulov, B. (2023). Application Fuzzy Neural Network Methods to Detect Cryptoattacks on Financial Information Systems Based on Blockchain Technology. In: Koucheryavy, Y., Aziz, A. (eds) Internet of Things, Smart Spaces, and Next Generation Networks and Systems. NEW2AN 2022. Lecture Notes in Computer Science, vol 13772. Springer, Cham. https://doi.org/10.1007/978-3-031-30258-9_9
  10. S. Axelsson, “The base-rate fallacy and its implications for the difficulty of intrusion detection,” in Proc. 6th ACM Conf. Computer and Communications Security (CCS), Singapore, 1999, pp. 1–7, doi: 10.1145/319709.319710.
  11. F. Pendlebury, F. Pierazzi, R. Jordaney, J. Kinder, and L. Cavallaro, "TESSERACT: Eliminating experimental bias in malware classification across space and time," in Proc. 28th USENIX Security Symp., Santa Clara, CA, USA, 2019, pp. 729–746.
  12. D. Arp, E. Quiring, F. Pendlebury, A. Warnecke, F. Pierazzi, C. Wressnegger, L. Cavallaro, and K. Rieck, “Dos and don’ts of machine learning in computer security,” in Proc. 31st USENIX Security Symp., Boston, MA, USA, 2022, pp. 3971–3988.
  13. Bozorov S. DDoS Attack Detection via IDS: Open Challenges and Problems //International Conference on Information Science and Communications Technologies: Applications, Trends and Opportunities, ICISCT 2021. – 2021.
  14. N. Moustafa and J. Slay, “UNSW-NB15: A comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set),” in Proc. 2015 Military Communications and Information Systems Conf. (MilCIS), Canberra, Australia, 2015, pp. 1–6, doi: 10.1109/MilCIS.2015.7348942.
  15. Sharafaldin, A. H. Lashkari, and A. A. Ghorbani, “Toward generating a new intrusion detection dataset and intrusion traffic characterization,” in Proc. 4th Int. Conf. Information Systems Security and Privacy (ICISSP), Funchal, Portugal, 2018, pp. 108–116, doi: 10.5220/0006639801080116.
  16. Safarov, Furkat & Basak, Mainak & Nasimov, Rashid & Abdusalomov, Akmalbek & Cho, Young. (2023). Explainable Lightweight Block Attention Module Framework for Network-Based IoT Attack Detection. Future Internet. 15. 297. 10.3390/fi15090297.
  17. S. Jacobs, R. Beltiukov, W. Willinger, R. A. Ferreira, A. Gupta, and L. Z. Granville, “AI/ML for network security: The emperor has no clothes,” in Proc. 2022 ACM SIGSAC Conf. Computer and Communications Security (CCS), 2022, pp. 1537–1551.