Z-Wave Zerial Port Sniffer (Zpiffer) Project
Both the Z/IP Gateway and OpenZWave library can copy all serial port traffic to and from the ZWave chip on a UZB stick into a file. We would like to be able to decipher the byte stream, both at the ZWave serial function level and for embedded command classes and commands. The specification of the stream is fixed and amounts to a huge state machine, interpreting bytes as they come in: matching against identifiers or assigning to variables, branching down alternative paths, and looping through repeated sequences or until an array is exhausted. As output we want a human-readable breakdown of each frame, displayed either as text or in a GUI. We call the program the Zpiffer, for ZWave serial port sniffer. It complements the Zniffer: although the Zpiffer doesn't read packets as they fly through the air – the ultimate check on ZWave traffic – it doesn't require extra hardware and isn't limited to Windows machines.
Data flow in a ZWave system, with the zpiffer monitoring serial port traffic and writing a readable version, which can be followed in a GUI.
The core of the Zpiffer is a ZWave specification written in a compact design-specific language (DSL). We use lex to convert it to state machine tables, with states and transition rules. The Zpiffer program has a parser that reads the byte stream and follows the state machine, emitting a readable version of the identifiers and variables. It can decrypt S0 packets if the network key is know. There is a simple GUI that can read the text output; it supports searching, filtering, and sorting any of the frame information, including date/time, source or destination node, and serial function or command class or command.
The package is available as a tarball. It includes the ZWave specification, the programs and scripts for transforming it into a state machine, the Zpiffer parser, and ZpifferGUI graphical interface to the parsed output, as well as full documentation.
Compilation of the ZWave serial specification into an executable parser, the zpiffer. Circles represent executed programs, either compiled or scripted.
ZWave Specification and Design Specification Language
INS12350 (Serial API Host Appl. Prg. Guide) defines the serial function streams. Command classes are found in SDS13781 (Z-Wave Application Command Class Specification), SDS13782 (Z-Wave Management Command Class Specification), SDS13783 (Z-Wave Transport Encapsulation Command Class Specification), and SDS13784 (Z-Wave Network Protocol Command Class Specification). The specifications were translated into the DSL in Fall, 2019.
You can read a full description of the DSL here, or in the README in the package. It can be used to define any byte stream, not just ZWave. The idea is to match incoming bytes against known, named identifiers or to treat them as values. The DSL allows you to specify how to display the data (as integers, in hex, or as strings), to extract bits, or to combine bytes or bits into larger values. It also provides control flow mechanisms. The most common is to branch after matching from a list of alternatives. You can define groups of bytes with a known size, or consume all that are available (accounting for trailing sequences), and process them as a series or a single collection. You can define common sub-sequences that are substituted when encountered. ZWave specific elements handle the transmit/response/request paths in serial functions, and S0 encryption. Finally you can flag sequences as optional, which will not cause a parse error if omitted.
Specification of the SENSOR_MULTILEVEL command class. Identifiers are defined in the ZWave headers, except SENSORTYPE which is found later in the spec. GET has an optional payload with a 2 bit scale, REPORT also has bit-encoded precision and sizes, plus a value of size bytes.
State Machine
A state machine contains nodes, corresponding to identifiers and variables in the specification, and links between them (also called sequences) for transitions between states. These are stored in two separate header files, generated from the DSL specification in three steps. First, a lexer extracts the identifiers and variables into a preliminary table while leaving an intermediate version of the specification. The table is first run through the C pre-processor to substitute the ZWave header and local definitions, then a script to clean up references and simplify math expressions. Finally, another lexer extracts the sequences table from the intermediate spec and adds the links to get the final identifier (node) table. The tables are stored as structures in header files which are compiled into the zpiffer parser.
Graphical depiction of internal parse of SENSOR_MULTILEVEL_REPORT command. Arrows indicate advance of parsed byte. SENSORTYPE is parsed, then the precision, scale, and size are unpacked from the next byte with bit operations. The parser builds value in a loop from the next size bytes.
Zpiffer/Parser
The zpiffer program implements the state machine and interface to the byte stream. It opens and reads from the Z/IP Gateway or OpenZWave log file, and can run in a continual loop to monitor serial port traffic as it's generated. The logs store complete frames, at least one per line, so the parser doesn't technically advance byte by byte; it can take shortcuts to handle multi-byte values and arrays in one go. In principle the flow through the state machine is simple. The parser determines how many bytes are needed for the next identifier, variable, or array while verifying that enough are available. Values are then assembled, possibly using the previous byte or bits, and bit masks and shifts applied. They may be held for the next identifier, stored as group counters or sizes, or kept in a local memory. The values are then printed. Finally the state machine advances to the next identifier. This involves testing group counters or lengths, following references to defined identifiers, picking the correct alternative in a list, skipping optional bytes if not available, and setting up the counters or lengths when a group starts. The logic in this section is somewhat tricky, especially with groups and arrays which may be nested or empty, and with detecting errors in the byte stream. We must also maintain a small stack of frame positions when we start down an alternative branch or follow a reference to a local definition, so we know where to continue when it ends.
The Zpiffer has numerous command line arguments. It can write its output to the console, a file, or a UDP port. It can follow updates to the byte stream, or read through the log file once. Identifiers can be displayed with their full name from the header file, or an abbreviation, and the byte position within the frame can be added. Finally, if passed the name of a file that holds the 16 byte S0 network key, the parser will decrypt secure messages. There is a zpiffer man page in the package that describes all the options.
zpiffer output of an S0 exchange, with the nonce on the left and the decrypted BASIC_SET command on the right.
ZpifferGUI
You'll find a GUI for the parser output in the package. It continually reads the parser output from the Zpiffer program, breaks it into several columns of information, and displays frames in a table. Clicking on any entry will show the full parse; there are two windows for left and right mouse button clicks.
ZpifferUI main window, with log in table at top and parsed frames below.
Clicking on the column headers brings up a filtering window that will show only frames matching the conditions. Depending on the type of data, you can specify a condition on the values ("show all frames not sent to node 6"), list specific values (or check values off a list for the serial functions, command classes, or commands), give substrings to match within the value, or test if a value exists. Unlike the Zniffer, source and destination nodes are available only as part of a frame and not for every interaction.
Defining a filter for the Command Class column. You can pick from the list on the right, or type in byte values at top left. Instead of a list you can display a range of values at middle left, or only show frames with commands at bottom left.
You can search for the first or last frame with a string by typing Control-f in the column. Saving can include the whole log, only those frames that are displayed or filtered, or on or after a date and time.
The GUI is written in Tcl/Tk and should run on any platform.
Other Applications
Except for handling S0 and the transmit/response/request byte in serial frames, there is nothing specific to ZWave in the DSL. The parser does depend on the format of the log files. If you avoid the first and modify file handling for the second, you can re-use the parser for any application. Just write a specification, generate the state machine, and compile a new parser.