State of the art of network protocol reverse engineering tools

Julien Duchene (1, 2) , Colas Le Guernic (3, 1) , Eric Alata (2) , Vincent Nicomette (2) , Mohamed Kaâniche (2)

1 DGA.MI - DGA Maîtrise de l'information
2 LAAS-TSF - Équipe Tolérance aux fautes et Sûreté de Fonctionnement informatique
3 TAMIS - Threat Analysis and Mitigation for Information Security

Communication protocols enable structured information exchanges between different entities. A description, at different levels of detail, is necessary for many applications, such as interoperability or security audits. When such a description is not available, one can resort to protocol reverse engineering to infer the format of exchanged messages or a model of the protocol. During the past 12 years, several tools have been developed in order to automate, entirely or partially, the protocol inference process. Each of those tools has been developed with a specific application goal for the inferred model, leading to specific needs, and thus different strengths and limitations. After identifying key challenges, the paper presents a survey of protocol reverse engineering tools developed in the last decade. We consider tools focusing on the inference of the format of individual messages or of the grammar of sequences of messages. Finally, we propose a classification of these tools according to different criteria, that is aimed at providing relevant insights about the techniques used by each of these tools and comparatively to other tools, for the classification of messages, the inference of their format or of the grammar of the protocol. This classification also permits to identify technical areas that are not sufficiently explored so far and that require further development in the future.

Mots clés: Protocol inference; Network trace analysis; Reverse engineering; Data structure inference; Binary application analysis

Domaines: Cryptographie et sécurité [cs.CR]; Réseaux et télécommunications [cs.NI]

Julien Duchene, Colas Le Guernic, Eric Alata, Vincent Nicomette, Mohamed Kaâniche. State of the art of network protocol reverse engineering tools. Journal of Computer Virology and Hacking Techniques, 2018, 14 (1), pp.53-68. ⟨10.1007/s11416-016-0289-8⟩. ⟨hal-01496958⟩