Decoding Sensor Measurements From Hex Payloads Using Node.js

Node.js

In recent weeks I was decoding sensor measurements. The measurements were coming from various sensors. The payloads of the sensors were in hex format.

The first step of the process is to collect the sensor’s documentation. The documentations usually contain a description of the payload format.

An example payload might look like this:

Pattern:

AABBCCCCDD

Concrete example:

919F003D01

Where:

  • A - A unique identifier for the sensor model
  • B - A unique identifier for the message type
  • C - Temperature measurement
  • D - Air pressure measurement

Details of each portion:

  • A - 1 Byte, Unsigned
  • B - 1 Byte, Unsigned
  • C - 2 Bytes, Unsigned, Big endian, Celsius
  • D - 1 Byte, Unsigned, Atm

Some details can change between different portions of the payload.

Size

The payloads are usually in hex format. As a rule of thumb, two characters in the hex format represent 1 Byte, aka. 8 bits.

Signedness

This signedness determines what ranges of values can be represented with a certain number of bytes. Usually, if a number is signed its actively mentioned in the documentation, otherwise you can assume it’s unsigned.

Endianness

The endianness determines how the bytes should be ordered. Either from left to right or right to left. If not explicitly stated in the documentation it usually means big-endian. If a portion is only 1-byte long the endianness does not matter, since endianness means byte ordering.

Unit of measurement

In the case of measurements, the documentation must specify the unit of measurement it uses.

This is usually not a single unit, instead a portion of an unit. For example: 1 / 16 of a degree Celsius.

This ratio is basically the resolution of the sensor. In this case the sensor can sense temperature difference in 0.0625 increments.

Node.js implementation

There is a great package called binary-parser that can handle binary data elegantly.

It can streamline endianness, signedness and much more. The input of the parser is Buffer so first you must convert your hex string. The output is the parsed object.

1
2
3
4
5
6
7
8
9
10
const Parser = require('binary-parser').Parser;
 
const sensorParser = new Parser()
  .uint8("modelId")
  .uint8("messageId")
  .uint16be("temperature")
  .uint8("airPressure")
 
const buffer = Buffer.from("919F003D01", "hex");
const measurements = sensorParser.parse(buffer);

This produces an object with the following format:

1
2
3
4
5
6
{
	modelId: 145,
	messageId: 159,
	temperature: 61, // in 1 / 16 C
	airPressure: 1	 // int Atm
}

Formatters

We can handle unit of measurements with the built in formatters.

In our example the temperature is sent in 1 / 16 degree Celsius but we want to receive values as Celsius.

1
2
3
4
5
6
7
8
9
10
11
const temperatureFormatter = (temperature) => {
	return {
		temperature / 16; // Alternative tempearture * 0.0625
	}
}

const sensorParser = new Parser()
  .uint8("modelId")
  .uint8("messageId")
  .uint16be("temperature", { formatter: temperatureFormatter})
  .uint8("airPressure");

This produces:

1
2
3
4
5
6
{	
	modelId: 145,
	messageId: 159,
	temperature: 3.8125, // in C
	airPressure: 1	 // int Atm
}

Variable length portions

Some payload formats have variable length internal portions.

AABBBBCC

Where

  • A : First value we need
  • B : A variable length portion that has no information for us
  • C : Second value we need
  • D : Third value we need

We can handle this situation with an offset and the seek method.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
const variableLengthParser = (buffer) =>{
	const parser = new Parser()
	.uint8('first')
	.saveOffset('currentOffset')
	.seek(function() {
		const secondByteLength = 1;
		const thirdByteLength = 1;
		return { buffer.byteLength - this.currentOffset - ( secondByteLength + thirdByteLength )}
	})
	.uint8('second')
	.uint8('third');

	return parser.parse(buffer);
}

In this case, we need an encapsulating function that allows us to reference the buffer itself. After the first argument, the offset is saved. Then inside the seek function, the number of steps are calculated until the end of the variable-length portion.

For that, we need the total buffer length and sizes of portions coming after the variable-length portion.

Skipping bits

Some payloads have bits that represent a certain state of the payload.

As an example, let’s say the 1st bit of the 2nd byte is a special signal bit we need.

1
2
3
new Parser
.uint8()
.bit1('specialBit')

One potential problem if we need to get the first and third bit of an 2 Byte portion that is big-endian.

Since big endian has reverse byte order we need to get the bits from the end:

1
2
3
4
5
new Parser
.bit13()
.bit1('third')
.bit1()
.bit1('first')

Slicing

Some payload formats contain both hex and ascii portions.

Example

3D01

Where the first two characters are the hex representation of the number 61 and the second two characters literally represent 1.

In these cases splicing the string is the best option we have.

Multiple units of measurement

If we have multiple sensors each sending measurements in different units we need to convert them into a single unit.

We can use the convert-units package and write a utility function to handle this.

1
2
3
4
5
6
7
8
9
10
11
const temperatureConverter = ({ unit: currentUnit , value }) => {
	const temperatureUnit = 'c';
	if (convert().from(currentUnit).possibilities().includes(temperatureUnit){
	return convert(value).from(currentUnit).to(temperatureUnit)
  } else {
		return value;
	}
}

const measurements = { temperature { unit: 'K', value: 273.15 }};
const temperatureInCelsius = temperatureConverter(measurements.temperature)

The temperatureConverter takes in a unit as parameter. Checks if it’s possible to convert it to the the selected temperature unit (C). Finally, if it’s possible returns the converted value.

Useful resources