Special character handling - MQTT error

[marcvl]

I'm getting following MQTT error when publishing to a topic with a payload that contains the 'degree' character (º
MASCULINE ORDINAL INDICATOR - Unicode: U+00BA, UTF-8: C2 BA):

[24941] MQTTException -> [Errno 104] ECONNRESET
reconnect start ...
reconnect finish ...

The MQTT message goes to Home Assistant where it also generates an error:

Logger: homeassistant.components.mqtt.discovery
Source: components/mqtt/discovery.py:109
Integration: MQTT (documentation, issues)
First occurred: 6:04:58 PM (1 occurrences)
Last logged: 6:04:58 PM*

*Unable to parse JSON core2-temp: '{"avty_t": "~status", "pl_avail": "on", "val_tpl": "{{value_json.temperature}}", "dev_cla": "temperature", "unit_of_meas": "ºC", "~": "core2/env2/", "dev": {"mdl": "Core 2", "mf": "M5Stack", "name": "Core 2 Thermostat", "ids": ["12234"]}, "name": "Core2 Temperature", "stat_t": "~state", "pl_not_avail": "off", "uniq_id": "12234"'

Strangely, the JSON seems to be missing closing "}" which might explain the HA error. My code has the right JSON formatting. Just removing the 'degree' symbol will solve the issue. So, there seems to be something strange with how special characters are handled. Or maybe I'm doing something wrong?

[robalstona]

You may try replace celsius character with this string \0xb0

[marcvl]

How would I do that?

My current code is: KEY_UNIT_OF_MEASUREMENT: "ºC",

Also, I believe the correct Unicode character is 0xba, not 0xb0.

I checked Mosquitto output to see exactly what Core2 is putting out. It's very strange. When I add the 'degree' symbol, the JSON is missing a closing parenthesis (which is why HA is complaining). I'm showing you below the Mosquitto output for the temperature sensor and the humidity sensor. Both are formatted in the same way. The temperature sensor has the degree symbol.

homeassistant/sensor/core2/core2-temp/config {"avty_t": "~status", "pl_avail": "on", "val_tpl": "{{value_json.temperature}}", "dev_cla": "temperature", "unit_of_meas": "ºC", "~": "core2/env2/", "dev": {"mdl": "Core 2", "mf": "M5Stack", "name": "Core 2 Thermostat", "ids": ["12234"]}, "name": "Core2 Temperature", "stat_t": "~state", "pl_not_avail": "off", "uniq_id": "12234"
homeassistant/sensor/core2/core2-humid/config {"avty_t": "~status", "pl_avail": "on", "val_tpl": "{{value_json.humidity}}", "dev_cla": "humidity", "unit_of_meas": "%", "~": "core2/env2/", "dev": {"mdl": "Core 2", "mf": "M5Stack", "name": "Core 2 Thermostat", "ids": ["12234"]}, "name": "Core2 Humidity", "stat_t": "~state", "pl_not_avail": "off", "uniq_id": "122347"}

And here's the same Mosquitto output, when I remove the degree symbol.

homeassistant/sensor/core2/core2-temp/config {"avty_t": "~status", "pl_avail": "on", "val_tpl": "{{value_json.temperature}}", "dev_cla": "temperature", "unit_of_meas": "C", "~": "core2/env2/", "dev": {"mdl": "Core 2", "mf": "M5Stack", "name": "Core 2 Thermostat", "ids": ["12234"]}, "name": "Core2 Temperature", "stat_t": "~state", "pl_not_avail": "off", "uniq_id": "12234"}
homeassistant/sensor/core2/core2-humid/config {"avty_t": "~status", "pl_avail": "on", "val_tpl": "{{value_json.humidity}}", "dev_cla": "humidity", "unit_of_meas": "%", "~": "core2/env2/", "dev": {"mdl": "Core 2", "mf": "M5Stack", "name": "Core 2 Thermostat", "ids": ["12234"]}, "name": "Core2 Humidity", "stat_t": "~state", "pl_not_avail": "off", "uniq_id": "122347"}

[marcvl]

Ok, so I tried two alternatives:

KEY_UNIT_OF_MEASUREMENT: chr(176) + "C",
KEY_UNIT_OF_MEASUREMENT: chr(186) + "C",

The first one represents 0xb0 and the second one 0xba.

Both result in the same error message ([24400] MQTTException -> [Errno 104] ECONNRESET) and a missing } at the end of the JSON.

Maybe the json.dumps function, which I'm using, needs a special parameter to indicate unicode characters?

[marcvl]

So my last comment provided the solution....

The correct way of doing this is :

1/ Specify the Unicode character in the payload with chr(186) where 186 is the character number (in this case representing MASCULINE ORDINAL INDICATOR - Unicode: U+00BA, UTF-8: C2 BA.

2/ Ensure that your json.dumps applies utf-8 encoding before sending it to MQTT as follows:
m5mqtt.publish(topic, json.dumps(payload).encode('utf-8'))

Problem solved.