Deep Reinforcement Learning for Intrusion Detection and Response in Software-Defined Networks
Abstract
This dissertation presents a unified set of learning-based solutions for improving cybersecurity in Software-Defined Networking (SDN) environments. The main objectives are to provide fine-grained traffic flow information for accurate attack detection, jointly optimize feature set and machine learning model selection for intrusion detection, develop adaptive intrusion response mechanisms that minimize disruption to legitimate traffic, and address the shortage of labeled training data for machine learning-based security applications in SDN-based networks. To achieve these objectives, the dissertation makes four main contributions: DeepMatch: A fine-grained traffic flow measurement framework based on a Partially Observable Markov Decision Process (POMDP) and a Deep Dueling Neural Network-based control algorithm. DeepMatch dynamically adjusts flow rule matching strategies to maximize traffic flow granularity while proactively preventing flow-table overflow in switches in the SDN data plane. // DeepGuard: A machine learning-based intrusion detection framework that integrates DeepMatch to preserve important runtime traffic flow features. DeepGuard formulates the joint selection of feature sets and machine learning models as a learning-based optimization problem and applies Q-Learning to identify effective configurations for accurate and efficient attack detection. // DeepAir: An adaptive intrusion response framework that models response-policy selection as a Markov Decision Process (MDP). By leveraging a Double Deep Q-Network (DDQN), DeepAir learns effective response policies that mitigate malicious traffic in real time while reducing the negative impact on benign traffic forwarding and limiting policy deployment overhead. // DeepTransfer: A selective transfer learning framework that addresses the lack of labeled SDN attack detection data. DeepTransfer applies DDQN to guide the transfer of attack detection knowledge from legacy network environments to SDN-based target domains, thereby improving detection performance while reducing the risk of negative transfer across heterogeneous network domains. // Extensive experimental evaluations validate the effectiveness of the proposed frameworks under different security scenarios, particularly Denial-of-Service and Distributed Denial-of-Service (DoS/DDoS) attacks. The results demonstrate that the proposed solutions improve traffic flow visibility, feature set-model selection, intrusion detection accuracy, response adaptability, QoS preservation for benign traffic, and learning stability compared with existing approaches. Collectively, the contributions of this dissertation advance intelligent and adaptive cybersecurity mechanisms for SDN infrastructures.
Keywords
Not available
Specialisations
Similar theses
- Evaluating Self-Supervised Learning for Network Intrusion Detection Bryan A. Marin Samano · 2026
- Adaptive Machine Learning for Real-Time Intrusion Detection in Networks Anju Sobha · National College of Ireland · 2025
- GenVRAD: dataset generator for machine learning detection of vehicle-roadside attacks : a thesis in Computer Engineering Devaj Ramsamooj · 2024
- Quantum machine learning for enhanced intrusion detection Ameel Valjee · University of Cape Town · 2026
- Machine Learning-Based Network Intrusion Detection for Industrial Control Dehlaghi Ghadim, Alireza · Mälardalen University College · 2025
- Class Imbalance and Time-To-Detection in the Performance Analysis of Machine Learning-Based Intrusion Detection Systems Shabanfar, Mohammad Pasha · Concordia University · 2025
