Getting data out of smart meters

For the tariff I am on, I needed a smart meter - there are various concerns about smart meters, but having actually got them, I want to be able to make use of them for the one thing they are good at. That is accurate usage data in real time.

Yes, for electricity a current clamp also works, but how "calibrated" is that exactly? Getting the data from the actual smart meter is much nicer.

But how?

Well, it turns out that there is an in home display you can buy which links to your smart meter and also to your wifi, and will send data via MQTT!

Slightly redacted MQTT smart meter data

It seems I can get electricity updates every 10 seconds, and gas every half an hour. It even includes electricity export data (although only a cumulative figure).

Next step will be putting data in to an SQL database.

And yes, it shows export real time as well :-

What is this marvel?

It is the Glow from Hilderbrand. They have been quite good sorting it out - it was to be an SMETS1 unit, but with new smart meters it is the SMETS2 that I have.

So far, I am impressed.

Home automation for geeks

This is mainly about bridging some of the "trendy apply shit" and "command line forever" types of working for home automation.

Background: I have done quite a bit for home automation here at the new house. There are thousands of IoT devices (smart sockets, lights, switches, all sorts). They all come with their own app, cloud service, and feature set, but thankfully most work using ESP8266 chipset and can be "reflashed" with tasmota. Tasmota is an open source IoT application, and very flexible. It is not that tasmota is "better" than the systems that come with these devices (though it usually is), more that almost any device from almost any manufacturer can be made to work in the same way with the same config. The main one I have been using of late is the Shelly 1, which is very easy to reflash and has a relay and switch input. I do, however, have a number of my own devices, including environmental monitors. The tasmota devices can do lots of automation and send messages to each other as needed. I have, however, tried to keep it simple so a light with a Shelly 1 will have the switch connected and will work the light based on the switch even with no WiFi or Internet working!

These all work over WiFi and connect to an MQTT server. MQTT allows devices to talk to each other using a "topic" (simple text string) rather than that pesky IP addressing stuff :-) But it is very much in the school of "command line" over "GUI".

Bring on apple!

I use an iPhone, but I have not bought in to their home automation. As I say, it is more "command line" for me. That is until now. After much nagging from my son, I have managed to link my Shelly 1 and other devices, including my own environmental sensors, to my iPhone!

• I have MQTT running on a Raspberry Pi in my loft. It is just apt install mosquitto. You may want to set up some security too.
• I installed homebridge. There is a Raspberry Pi image even. I just followed the instructions and it worked. Again, set a password!
• I installed the add on homebridge-tasmota plug in (just search on the plug in tab on the web config for homebridge).
• I sent a cmnd/tasmotas/SetOption30 1 to turn on Home Assistant Discovery on all my tasmotas.
• I added the homebridge to me "home" on my iPhone. It asked what room each device was in. It warns it is not an approved bridge :-)

It is worth pointing out it is not perfect. If devices are added afterwards they appear in the "room" where the iPhone thinks the homebridge is located. You can move them to the right room, and rename them, but it is not quite as stable - resetting things can move it back very easily. So I recommend getting on homebridge first then adding the bridge. It asked where each devices it when you do that. You can reset the bridge and re-add if needed.

As for my environmental monitors, I added the necessary home assistant discovery messages, and it works. I had to do a bit of trial and error, and reading the homebridge-tasmota plugin code, but managed to make it see Temperature, Humidity and CO2 properly. Again, it seems better if these are seen by the homebridge before adding the bridge to the iPhone.

But yes, it works. I am all GUI'd up now!

Inveo Nano

The temperature sensors I am using are the Inveo Nano PoE sensors. They are quite industrial and cost around £100. In hindsight I realise I could now make a wifi connected temperature sensor for around £2.50 or add a temperature sensor module to my new light switches for a pence, but possibly not as good.

Update: It looks like I could connect a DS18B20 (the sensor the Nano uses) to any of my new Sonoff light switches with incredible ease. So yes, for the cost of the sensor (under £3) I could have put temperature sensor attached to the new light switches and saved £100. Arrrg!

They work quite well. Slight hassle setting up as on a fixed IP not DHCP, but documented well enough, and they will do DHCP. They have several ways of working including SNMP.

I set one up in the bedroom, and this is what I have been testing with for a while now. A cron every minute got the temperature by SNMP and fed in to my air-con control stuff using MQTT.

However, the other two that I had ordered finally arrived yesterday, and they don't work. Arrrg! The SNMP simply does not respond. Yes, I check the manual and the config (it is one tick box to enable SNMP), but nothing. Only difference is new ones are version 1.18 and old one is version 1.17. I was pulling my hair out.

Then I spotted another option: MQTT. When I first installed one I had not looked at MQTT at all, but now I am using it, and in fact feeding the SNMP result in by MQTT. So I tried that.

What is odd is that it allows MQTT to be turned on, but it connects to a pre-set cloud service :-

This is a bit odd, as it seems to mean I could poke temperatures in to their MQTT cloud service for any sensors if I know the MAC. Anyway, the DNS provided on the FireBrick on the VLAN on which I have these things changed the IP for their MQTT server to mine, and now it reports to me. All I had to do was allow my air-con control code to be configurable as to the topic on which it listens for temperature. Sorted! Shame about the SNMP but I don't care now :-)

It is quite nice - it sends temperature every minute, and if it changes (e.g. 0.1 degree change). This had a rather unexpected side effect of making my air-con control too good. It reacted instantly, turning off the compressor when temperature hit the target so the code. It was also trying to keep the average temperature at target and could never win as it never went over target! As a result my room was within 0.2℃ total variation all night which is amazing!

I also noticed that the air-con seems to not try the compressor again for 5 minutes, which probably makes sense, and ensures I am not wearing anything out. As a result, every 5 minutes, it runs the compressor for 20 to 30 seconds or so - whatever needed to reach target temp, and stops. It does work rather well.

I have since tweaked it so it is not trying to set ever hotter temperatures to move that on to the target temperature :-)

Setting the temperature

I have upgraded the aircon in my bedroom as the old one broke, and have a nice new Daikin one (as I blogged). I can control over wifi, which is good.

Even though this is way better than the old unit I had, it still suffers from a couple of problems.

Firstly, the good thing it does is finely control the power and fan so that as it gets close to target temperature it lowers the power a bit, and ends up staying on target pretty closely without oscillations. Excellent.

One problem it cannot solve though is the there will be a temperature gradient in the room, not just top to bottom, but also potentially quite a lot if you have a window open a bit to let in some fresh air and ensure the CO2 levels are sensible. Outside was actually below 0 this morning!

The other small problem is that the "auto" mode is not good at some times of year when it will wait until several degrees over target before switching.

To solve the first problem I have an SNMP thermometer with a sensor on the side of my bed where I sleep. This allows me to nicely graph the temperature for a start.

This also lets me take control of the air-con.

First attempts

The first attempt involved doing the job for the aircon - setting the power based on the temperature. However, I cannot do that directly, I have to set the target temperature relative to current temperature to force it to work out a power setting. This took several iterations to get reasonably right.

At one point I was in fact oscillating so much I even had it switching to cooling after it overshot on heating, and then back again later.

But finally after a lot of testing (mostly leaving it for an hour or so on latest algorithm while I did other stuff) I got it oscillating within around ±0.5℃ or so. I was pleased, and slept on it (in a nice temperature controlled room).

On this graph, y-axis is temp in ℃ intervals, x-axis is time in hours, black is target, red is measured temp, light red is setting temp on aircon. The green is reported temp from aircon which I suspect is a separate sensor and way off. So mostly the measured temp was within around ±0.5℃ once it had settled a bit at the start.

Then I had a brainwave..

Finally working (well, so far)

The aircon itself does a better job, holding a much more stable temperature without much oscillation. So use that! I decided to keep a moving average of how far out the setting temperature was from the measured temperature, and use that to add an offset to the target set on the aircon. It only allows settings to 0.5℃ so I added error dither as well.

This works a treat, and allows me to set the aircon allowing for the temperature gradient in the room automatically. The only minor change I made was to offset the averages by a few minutes to allow for the fact that the set temperature does not immediately change the room temperature - so as to better track the actual temperature difference. I was somewhat fooled by how much it changed during the day when sun light hit the room, etc, but I should have expected that.

Here, by using the aircon to control the level we have a smoother temperature. This was in the evening as the outside temperature dropped and so the level to which the aircon had to be set was gradually increased to maintain the measured temperature. Again, within 0.5℃.

The result is all on GitHub (here). It includes MQTT daemon as well (which is how I tell it the air temp from SNMP).

Update: This worked well when the aircon is fighting a window that is ajar. Now I have my new fans installed and windows slows it is oscillating ±1℃, because it is effectively turning on and off at a low level. So I am working on the older method if I find we are doing that. We'll see how it goes. Github updated as I go.

MQTT

The trade off between security and convenience is a complex one for IoT. The irony of my recent investigation in to a few things was that the most secure (the Withings Sleep monitor) is in fact impossible for me to take control of directly and hence I cannot secure my personal data. The least secure (the Daikin air-con) is one of the easiest to take control of (local http) and so most useful to me.

I can take care of the security myself with separate IoT VLAN and hidden SSID and firewalling and so on, and I would rather not hand both my personal data, and control of my home, to third party "cloud" services if I can avoid it. I appreciate security of WiFI is always an issue, but at least it is mostly a physically constrained issue (i.e. in range of my WiFi) which is a lot tighter than cloud services in China, etc.

The SONOFF switches were interesting. I was a bit against re-flashing them initially, but in fact I think this is working really well. I have refreshed a few with Tasmota and it just works. The Tasmota provides a simple (optional) web interface and talks MQTT. It seems there are a few common open source projects for these, and Tasmota was recommended - thank you.

I had not quite appreciated how MQTT works, and actually it is quite fun. A simple messaging protocol via a hub (MQTT broker. The Tasmota stuff has loads of commands allowing you to configured everything via MQTT. You can even have one switch send MQTT commands to other switches (via the broker) so easy to create two way pairs of light switches, etc, with no control application.

It took me maybe half an hour to add MQTT support to my door entry and alarm system (SolarSystem) using the mosquitto library. This means I can add config to do things like turn on the lights when I open the door to my office, etc.

But MQTT is an odd beast in some ways - it is designed to be really simple for the small simple code of embedded devices, but you can use it with TLS and usernames and passwords as well. This is almost at odds as TLS is a big bit of code for an embedded device (it can be done). Also, the management of certificates is a complication for managing simple IoT. In some ways a simpler approach of a local firewalled VLAN is a lot easier, but then the comms within that LAN are all plain text and not secure. This is an area where an actual local area LAN, locked down, makes some sense.

Of course, I am not new to any sort of home automation - I had a door entry system back in the late 80's using a home made mag card reader connected to a wire wrapped 6502 board I made. But I have not really got on the bandwagon for some of this cheap modern kit like the SONOFFs. My approach, as usual, is to understand the nuts and bolts of these things, so at present a simple MQTT broker (mosquitto) is all I need, but I may go for something like home assistant some time.

However, I do like MQTT, so I will probably make an MQTT client for my Daikin air-con at some point (and put on GitHub). It may be that I add MQTT to some other code and systems as well.