Logo image
Back
Clustering Enabled Robust Intrusion Detection System for Big Data Using Hadoop-PySpark
Conference proceeding   Peer reviewed

Clustering Enabled Robust Intrusion Detection System for Big Data Using Hadoop-PySpark

Md Abdur Rahman and Hossain Shahriar
International Symposium on High Capacity Optical Networks and Enabling Technologies, pp.249-254
International Conference on Smart Communities: Improving Quality of Life using AI, Robotics and IoT (HONET), 20th (Boca Raton, Florida, USA, 12/04/2023–12/06/2023)
01/2024
DOI: https://doi.org/10.1109/HONET59747.2023.10374747

Metrics

1 Record Views

Abstract

big data framework data anomaly Hadoop-PySpark intrusion detection systems Big Data Cybersecurity Data Systems
In the era of burgeoning networks and the proliferation of novel devices, safeguarding computer security presents an ever-growing challenge. The increasing attack surface grants cybercriminals ample opportunities to exploit vulnerabilities within systems. While the internet fosters innovation across diverse sectors, it also exposes us to perils in the form of cyber attacks and nefarious activities. Consequently, the necessity to robustly identify and counteract these threats becomes paramount. Network Intrusion Detection Systems (NIDS) have a pivotal role in upholding system confidentiality, availability, and integrity by actively monitoring and responding to potential breaches. This study addresses the complex predicaments arising from managing copious data volumes and imbalanced class distributions in intrusion detection, a result of the exponential escalation of network traffic. A hybrid framework is proposed to bolster the accuracy of identifying minority classes within imbalanced datasets. This innovative hybrid framework combines K-means clustering and the Random Forest classifier, custom-crafted for big data processing through the utilization of Hadoop with PySpark. The proposed model shows its efficacy while evaluating training model by the NSL- KDD testing dataset, applying accuracy assessment as well as processing time evaluation. The outcomes of the proposed model, are measured with an accuracy, a precision, a recall, and an F1-score which are 0.9987, 0.9994, 0.9991, and 0.9986 respectively. These results outperforms existing established models in the detection of intrusions.

Details

Logo image