Data not published, CRC error

[bortek]

I am using Landis E360 meter and see that the data is coming but not published in HASS. I see in the logs that CRC checksum is failing.

[19:01:21][D][data:259]: Program Started
[19:01:21][D][data:263]: Read 24 bytes from UART
[19:01:21][D][data:293]: [22.7.0]: 0000.000 kW
[19:01:21][D][data:263]: Read 24 bytes from UART
[19:01:21][D][data:293]: [41.7.0]: 0001.019 kW
[19:01:21][D][data:263]: Read 24 bytes from UART
[19:01:21][D][data:293]: [42.7.0]: 0000.000 kW
[19:01:21][D][data:263]: Read 24 bytes from UART
[19:01:21][D][data:293]: [61.7.0]: 0001.110 kW
[19:01:21][D][data:263]: Read 24 bytes from UART
[19:01:21][D][data:293]: [62.7.0]: 0000.000 kW
[19:01:21][D][data:263]: Read 26 bytes from UART
[19:01:21][D][data:293]: [23.7.0]: 0000.210 kVAr
[19:01:21][D][data:263]: Read 26 bytes from UART
[19:01:21][D][data:293]: [24.7.0]: 0000.000 kVAr
[19:01:21][D][data:263]: Read 26 bytes from UART
[19:01:21][D][data:293]: [43.7.0]: 0000.200 kVAr
[19:01:21][D][data:263]: Read 26 bytes from UART
[19:01:21][D][data:293]: [44.7.0]: 0000.000 kVAr
[19:01:21][D][data:263]: Read 26 bytes from UART
[19:01:21][D][data:293]: [63.7.0]: 0000.274 kVAr
[19:01:21][D][data:263]: Read 26 bytes from UART
[19:01:21][D][data:293]: [64.7.0]: 0000.000 kVAr
[19:01:21][D][data:263]: Read 20 bytes from UART
[19:01:21][D][data:293]: [32.7.0]: 235.9 V
[19:01:21][D][data:263]: Read 20 bytes from UART
[19:01:21][D][data:293]: [52.7.0]: 238.9 V
[19:01:21][D][data:263]: Read 20 bytes from UART
[19:01:21][D][data:293]: [72.7.0]: 240.2 V
[19:01:21][D][data:263]: Read 20 bytes from UART
[19:01:21][D][data:293]: [31.7.0]: 004.7 A
[19:01:21][D][data:263]: Read 20 bytes from UART
[19:01:21][D][data:293]: [51.7.0]: 004.3 A
[19:01:21][D][data:263]: Read 20 bytes from UART
[19:01:21][D][data:293]: [71.7.0]: 004.7 A
[19:01:21][D][data:263]: Read 6 bytes from UART
[19:01:21][D][crc:275]: CRC: 1490 = ADF2. PASS = NO

Sometimes I see this in the logs

[19:01:11][V][app:081]: A component took a long time in a loop() cycle (0.30 s).
[19:01:11][V][app:082]: Components should block for at most 20-30ms in loop().

And this as well but it is discusses in issue #3

[19:00:52][E][uart_esp8266:167]: Reading from UART timed out at byte 0!

Any ide what could be wrong. Perhaps checksum is calculated differently on E360?

I use short cable like 8cm from the port to the board and it is a shielded cable with shield on the ground so the data loss due to is should not be a cause.

[bortek]

According to http://www.nodo-domotica.nl/images/8/86/DSMR.pdf

CRC is a CRC16 value calculated over the preceding characters in the data message (from
“/” to “!” using the polynomial: x16+x15+x2
+1). CRC16 uses no XOR in, no XOR out and is
computed with least significant bit first. The value is represented as 4 hexadecimal characters (MSB first).

Is this how the CRC is calculated in the code?

I also read on the forums (http://domoticx.com/p1-poort-slimme-meter-hardware/ )that other use different resistors. 1K instead of 10K and 4.7K. I will git it a try too.

[bortek]

Yet another finding on another forum. Quoting it here.

I fixed the cable was not suitable for a Landis E360. For this I had an ISKRA AM550.
So I opened it and put a 1K resistor between + 5v and RXD. There is also an existing resistor (10K) between the RTS and RXD. Now it works like a charm
Yes! Also soldered that 1k resistor. Now it works again. I had the exact same switch from an ISKRA AM550 -> Landis E360. How did you find out this was supposed to be it?

Source: https://gathering.tweakers.net/forum/list_message/60757450#60757450

But it does not really make sense if looking at the schematic on this project since RTS is already shorted to +5V and putting those two resistors (it will be in parallel) gives us one resistor of 9 Ohms.

[bortek]

Here they use it like this

To connect my Landys and Gyr E350 (ZCF110), I used a 10K resistor connected between DATA (RXD) and RTS.

https://github.com/fliphess/esp8266_p1meter

[bortek]

Yet another schematic just like this one also with inverter but they use both 1K resistors

https://gathering.tweakers.net/forum/list_message/61171042#61171042

[bortek]

Another suggestion of using differente serial parameters

SER.bytesize = serial.SEVENBITS
SER.parity = serial.PARITY_EVEN

I am going to beat it, this is just a matter of time and testing...
:)

[bortek]

E360 seems to use ESMR5.0 protocol and not DSMR perhaps crc calculation is different om it. Cant find ESMR5 specs.

[bortek]

Another finding about CRC from this page and further down in the comments
https://willem.aandewiel.nl/index.php/2019/04/09/dsmr-logger-v4-slimme-meter-uitlezer/

Willy,
I don't know what you're doing or what your lineup is.
I suspect that you yourself copy a telegram and use it as input. If so, remember that each line of a telegram must be terminated with a CR and LF. One wrong character in your telegram changes the CRC. An empty line also consists of at least a CR and LF!

Willy Verbiest says:
20 December 2019 at 19:04
Dear Willem,
I changed the existing const char raw [] definition from the parse example of Matthijs Kooijman to the received telegram from Fluvius and it looks like this:
// Data to parse (Fluvius mail)
const char raw [] =
“ / FLU5 \ 253770234_A \ r \ n ”
“ \ r \ n ”
“ 0-0: 96.1.4 (50213) \ r \ n ”

Willy
Change this line:

“/FLU5\253770234_A\r\n”

to

“/FLU5\\253770234_A\r\n”

And then try parse again.

A backslash has a special meaning.

Willy Verbiest says
21 December 2019 at 18:41
Dear Willem
_Fantastic it works! I used the original crc16_update from ( https://github.com/PaulStoffregen/cores/blob/master/teensy3/util/crc16.h )
and added the extra \ teg and get the same CRC number as sent in the P1 telegram of my digital meter (it is not smart).
Thank you.

I found a companion/specs for DSRM 5 http://www.nodo-domotica.nl/images/8/86/DSMR.pdf and CRC calculation algorithm is the same.

So what about this extra \ in the header of the message. Maybe it is being dropped out due to it is a special character and needs to be escaped and thus we need to add it? How to do that?

[bortek]

More ideas to try from https://willem.aandewiel.nl/index.php/2020/02/28/restapis-zijn-hip-nieuwe-firmware-voor-de-dsmr-logger/

Somehow some brilliant mind thought it was a good idea to use backslashes in the Smart Meter Telegram’s. It was/is not!
If your meter sends a backslash (which is for most computer’s an “escape char”) you have to “escape the escape char”. In other words: Place a backslash in from of every backslash in the telegram (so “\” becomes “\” and if there is something like this “\\” place a backslash in front of ever backslash in the telegram so it becomes “\\\”).

[bortek]

Today's finding. CRC fail rate is about 80% while 20% of the time CRC is correct. After analyzing lots of debug output i found that Data is coming or being read from the port using chunks. If there data is read in several chunks then the CRC fails. But if the data comes in as one big chunk the CRC is correct. Below is output of two different telegrams.

In the first part you see several chunks separated by =====Reading Data===== headline and thus CRC fail. The second telegram is only one chunk and CRC becomes OK. The data in both still looks ok.

One possible explanation could be that there are some extra characters being added or missing between chunks that make CRC fail. Next question is to find what causes the multiple chunks and if that is normal what are the characters (if any) that garble CRC.

[22:37:51][D][data:259]: =====Reading Data=====
[22:37:51][D][data:263]: Read 10 bytes from UART
[22:37:51][D][data:263]: Read 1 bytes from UART
[22:37:51][D][data:263]: Read 25 bytes from UART
[22:37:51][D][data:263]: Read 28 bytes from UART
[22:37:51][D][data:303]: [1.8.0]: 00000627.435 kWh
[22:37:51][D][data:263]: Read 28 bytes from UART
[22:37:51][D][data:303]: [2.8.0]: 00000000.000 kWh
[22:37:51][D][data:263]: Read 30 bytes from UART
[22:37:51][D][data:303]: [3.8.0]: 00000058.614 kVArh
[22:37:51][D][data:263]: Read 30 bytes from UART
[22:37:51][D][data:303]: [4.8.0]: 00000008.355 kVArh
[22:37:51][D][data:263]: Read 23 bytes from UART
[22:37:51][D][data:303]: [1.7.0]: 0003.025 kW
[22:37:51][D][data:263]: Read 23 bytes from UART
[22:37:51][D][data:303]: [2.7.0]: 0000.000 kW
[22:37:51][D][data:263]: Read 25 bytes from UART
[22:37:51][D][data:303]: [3.7.0]: 0000.608 kVAr
[22:37:51][D][data:263]: Read 25 bytes from UART
[22:37:51][D][data:303]: [4.7.0]: 0000.000 kVAr
[22:37:51][D][data:263]: Read 24 bytes from UART
[22:37:51][D][data:303]: [21.7.0]: 0001.000 kW
[22:37:51][D][data:259]: =====Reading Data=====
[22:37:51][D][data:263]: Read 24 bytes from UART
[22:37:51][D][data:303]: [22.7.0]: 0000.000 kW
[22:37:51][D][data:263]: Read 24 bytes from UART
[22:37:51][D][data:303]: [41.7.0]: 0001.058 kW
[22:37:51][D][data:263]: Read 24 bytes from UART
[22:37:51][D][data:303]: [42.7.0]: 0000.000 kW
[22:37:51][D][data:263]: Read 24 bytes from UART
[22:37:51][D][data:303]: [61.7.0]: 0000.966 kW
[22:37:51][D][data:263]: Read 24 bytes from UART
[22:37:51][D][data:303]: [62.7.0]: 0000.000 kW
[22:37:51][D][data:263]: Read 26 bytes from UART
[22:37:51][D][data:303]: [23.7.0]: 0000.171 kVAr
[22:37:52][D][data:259]: =====Reading Data=====
[22:37:52][D][data:263]: Read 26 bytes from UART
[22:37:52][D][data:303]: [24.7.0]: 0000.000 kVAr
[22:37:52][D][data:263]: Read 26 bytes from UART
[22:37:52][D][data:303]: [43.7.0]: 0000.193 kVAr
[22:37:52][D][data:263]: Read 26 bytes from UART
[22:37:52][D][data:303]: [44.7.0]: 0000.000 kVAr
[22:37:52][D][data:263]: Read 26 bytes from UART
[22:37:52][D][data:303]: [63.7.0]: 0000.243 kVAr
[22:37:52][D][data:259]: =====Reading Data=====
[22:37:52][D][data:263]: Read 26 bytes from UART
[22:37:52][D][data:303]: [64.7.0]: 0000.000 kVAr
[22:37:52][D][data:263]: Read 20 bytes from UART
[22:37:52][D][data:303]: [32.7.0]: 237.7 V
[22:37:52][D][data:263]: Read 20 bytes from UART
[22:37:52][D][data:303]: [52.7.0]: 235.6 V
[22:37:52][D][data:263]: Read 20 bytes from UART
[22:37:52][D][data:303]: [72.7.0]: 242.8 V
[22:37:52][D][data:263]: Read 20 bytes from UART
[22:37:52][D][data:303]: [31.7.0]: 004.2 A
[22:37:52][D][data:263]: Read 20 bytes from UART
[22:37:52][D][data:303]: [51.7.0]: 004.5 A
[22:37:52][D][data:259]: =====Reading Data=====
[22:37:52][D][data:263]: Read 20 bytes from UART
[22:37:52][D][data:303]: [71.7.0]: 004.1 A
[22:37:52][D][data:263]: Read 6 bytes from UART
[22:37:52][I][crc:285]: CRC: F7C9 = F976. PASS = NO

[22:38:11][D][data:259]: =====Reading Data=====
[22:38:11][D][data:263]: Read 10 bytes from UART
[22:38:11][D][data:263]: Read 1 bytes from UART
[22:38:11][D][data:263]: Read 25 bytes from UART
[22:38:11][D][data:263]: Read 28 bytes from UART
[22:38:11][D][data:303]: [1.8.0]: 00000627.451 kWh
[22:38:11][D][data:263]: Read 28 bytes from UART
[22:38:11][D][data:303]: [2.8.0]: 00000000.000 kWh
[22:38:11][D][data:263]: Read 30 bytes from UART
[22:38:11][D][data:303]: [3.8.0]: 00000058.618 kVArh
[22:38:11][D][data:263]: Read 30 bytes from UART
[22:38:11][D][data:303]: [4.8.0]: 00000008.355 kVArh
[22:38:11][D][data:263]: Read 23 bytes from UART
[22:38:11][D][data:303]: [1.7.0]: 0003.018 kW
[22:38:11][D][data:263]: Read 23 bytes from UART
[22:38:11][D][data:303]: [2.7.0]: 0000.000 kW
[22:38:11][D][data:263]: Read 25 bytes from UART
[22:38:11][D][data:303]: [3.7.0]: 0000.603 kVAr
[22:38:11][D][data:263]: Read 25 bytes from UART
[22:38:11][D][data:303]: [4.7.0]: 0000.000 kVAr
[22:38:11][D][data:263]: Read 24 bytes from UART
[22:38:11][D][data:303]: [21.7.0]: 0000.998 kW
[22:38:11][D][data:263]: Read 24 bytes from UART
[22:38:11][D][data:303]: [22.7.0]: 0000.000 kW
[22:38:11][D][data:263]: Read 24 bytes from UART
[22:38:11][D][data:303]: [41.7.0]: 0001.055 kW
[22:38:11][D][data:263]: Read 24 bytes from UART
[22:38:11][D][data:303]: [42.7.0]: 0000.000 kW
[22:38:11][D][data:263]: Read 24 bytes from UART
[22:38:11][D][data:303]: [61.7.0]: 0000.964 kW
[22:38:11][D][data:263]: Read 24 bytes from UART
[22:38:11][D][data:303]: [62.7.0]: 0000.000 kW
[22:38:11][D][data:263]: Read 26 bytes from UART
[22:38:11][D][data:303]: [23.7.0]: 0000.168 kVAr
[22:38:11][D][data:263]: Read 26 bytes from UART
[22:38:11][D][data:303]: [24.7.0]: 0000.000 kVAr
[22:38:11][D][data:263]: Read 26 bytes from UART
[22:38:11][D][data:303]: [43.7.0]: 0000.192 kVAr
[22:38:11][D][data:263]: Read 26 bytes from UART
[22:38:11][D][data:303]: [44.7.0]: 0000.000 kVAr
[22:38:11][D][data:263]: Read 26 bytes from UART
[22:38:11][D][data:303]: [63.7.0]: 0000.243 kVAr
[22:38:11][D][data:263]: Read 26 bytes from UART
[22:38:11][D][data:303]: [64.7.0]: 0000.000 kVAr
[22:38:11][D][data:263]: Read 20 bytes from UART
[22:38:11][D][data:303]: [32.7.0]: 237.5 V
[22:38:11][D][data:263]: Read 20 bytes from UART
[22:38:11][D][data:303]: [52.7.0]: 237.0 V
[22:38:11][D][data:263]: Read 20 bytes from UART
[22:38:11][D][data:303]: [72.7.0]: 243.1 V
[22:38:11][D][data:263]: Read 20 bytes from UART
[22:38:11][D][data:303]: [31.7.0]: 004.2 A
[22:38:11][D][data:263]: Read 20 bytes from UART
[22:38:11][D][data:303]: [51.7.0]: 004.5 A
[22:38:11][D][data:263]: Read 20 bytes from UART
[22:38:11][D][data:303]: [71.7.0]: 004.1 A
[22:38:11][D][data:263]: Read 6 bytes from UART
[22:38:11][I][crc:285]: CRC: 5CC3 = 5CC3. PASS = YES

[psvanstrom]

Great finds @bortek ! I've have not had time to look at this but I will do that now. You're probably on to something with the reading of chunks, seems that the E360 outputs data a bit different than the Sagemcom reader.

I've updated the code so that the full telegram that has been read is logged by debug logging each raw line read. Could you update your microcontroller and post a telegram here? Would make it a bit easier to try to understand what's going on.

[bortek]

Thanks for getting back! I though you left us alone hehe. Yes I think there is something fishy about the way E360 data chunks.

My controller that is located "at landet" has hanged after I have tried to add delay in the loop so I cannot do anything at the moment. I need to disconnect and manually flash new code. I will travel till landet on tuesday and will do the tests then.

[jimborr]

I am in the same situation using the E360 and getting random CRC error.. Using this version of code seems to work much better for me, no CRC errors at all any more.

Thanks!!

[bortek]

I have tried this specific branch called E360 which adds a delay during reading. But that did not make any difference. Therefore I wonder if you have made any other adjustments to the code or device?

[Anteus]

Also on a E360 but only getting CRC errors. Tried with 3s delay but no luck maybe something with my wiring?

[bortek]

Can you try with 40ms ? I have heard that one guy managed to get it working with it.