Talha Javed, Muhammad Haseeb, Muhammad Abdullah, Mobin Javed
Abstract
In this paper, we re-implement a recent work published in Usenix Security 2018: “Acquistional Rule Based Engine for Discovering Internet-of-Things Devices”. The paper introduced an NLP-based engine for automatically identifying the type, vendor, and product of IoT devices given banner data as input. We report on our efforts to reproduce the original implementation of the engine, documenting ambiguities around implementation and evaluation details that we encountered, as well as how we addressed them in our work. We evaluate our implementation on two ground truth datasets, finding that it fails to achieve the accuracy reported by the original authors. Our findings highlight the importance of recent community efforts towards a culture of reproducibility by presenting an example of how ambiguities in a research paper combined with lack of access to the original datasets can significantly affect a faithful re-implementation and evaluation.