Consideration of Network Anomaly Detection Models for DoS/DDoS Attacks in GNS3 Environments using Machine Learning

Abstract

In recent years, the rapid advancement of network technologies has led to increasingly sophisticated cyberattacks, posing significant challenges to traditional rule-based anomaly detection systems. This research proposes and verifies a dual-stage network anomaly detection pipeline that integrates machine learning models to identify both known and unknown threats effectively. The proposed system utilizes a Random Forest (RF) model for the classification of established attack patterns and an Autoencoder (AE) for the detection of unknown anomalies through a reconstruction-based approach.
To ensure a highly reproducible and realistic experimental environment, the virtual network simulator GNS3 was employed to construct a complex topology including attacker, client, and victim nodes. Network traffic data, comprising normal communications and DDoS attacks (SYN Flood and Slowloris), were collected and processed using the NFStream library to extract key statistical features.
The experimental results demonstrate the high performance of the proposed model, with the Random Forest achieving an accuracy of 1.00 in classifying known attacks even within complex traffic scenarios. Furthermore, the Autoencoder successfully identified anomalies with high precision (0.96 accuracy in GNS3-based experiments) by learning normal traffic baselines and detecting deviations through Mean Squared Error (MSE) analysis.