Logo image
Back
The Methodological Pitfall of Dataset-Driven Research on Deep Learning: An IoT Example
Conference proceeding   Peer reviewed

The Methodological Pitfall of Dataset-Driven Research on Deep Learning: An IoT Example

Tianshi Wang, Denizhan Kara, Jinyang Li, Shengzhong Liu, Tarek Abdelzaher and Brian Jalaian
MILCOM IEEE Military Communications Conference, pp.1082-1087
MILCOM22: IEEE Military Communications Conference (MILCOM) (Rockville, Maryland, USA, 11/28/2022–12/02/2022)
11/28/2022
DOI: https://doi.org/10.1109/MILCOM55135.2022.10017612
Web of Science ID: WOS:000968304600173

Metrics

4 Record Views
5 Times Cited - Web of Science

Abstract

In this paper, we highlight a dangerous pitfall in the state-of-the-art evaluation methodology of deep learning algorithms. It results in deceptively good evaluation outcomes on test datasets, whereas the underlying algorithms remain prone to catastrophic failure in practice. We illustrate the pitfall in the context of an Internet-of-Things (IoT) application example and show that it occurs despite the use of cross-validation that breaks down the data into separate training, validation, and testing sets. The pitfall is illustrated by designing two target detection and classification algorithms. One is based on a recently proposed neural network architecture for embedded AI, and the other is based on a traditional machine learning approach with domain-inspired feature engineering. The neural network approach outperforms the traditional one on test data. Yet, it fails in deployment. The mechanics behind the failure are explained and linked to the way the algorithms are trained. Suggestions are presented to avoid the pitfall. The paper is a "call to arms" to improve the evaluation methodology of machine learning algorithms for mission-critical systems.

Details

Logo image