Environmental sensors

For my next little project I am thinking of making a good little environmental sensor box, mainly for CO₂ and temperature as they are the two things I have some control over.

ESP8266 based, why not. But I have been looking at the sensors you can get, and bought a few.

Are all CO₂ sensors crap?

Well, I tried a few, including this CJMCU module (CCS811 based) and this AppliedSensor module.

Sorry to say, but both are crap! And the detail is in the small print - they estimate CO₂ from other sensors, and are very easily confused. I have some isopropyl alcohol and you use that anywhere near them and they quickly say 18000ppm CO₂. Even without such factors they swing wildly all over the place.

It seems you need an NDIR based sensor, and I found this one which seems to actually work! This SCD30 sensor seems to be the business, with CO₂, temperature and humidity (floating point!) via I²C.


Next step - sort a suitable display and box.

I can then use the temperature to control the air-conditioning, and the CO₂ to control extractor fan speed.

Door control

My door control and alarm system has made some progress.

The main change is making access control a bit more distinct from alarm system. The doors at the office were "falling off" the WiFi occasionally, even if only for a few seconds at a time, a few times a day.

I suspect that can be fixed, and it makes a huge difference what channels we use. The AP report something like 60 interfering APs.

So the change is to make the doors work autonomously which means the door controller has to be able to authenticate the fob/card used but also know enough to tell if it is allowed access.

I have actually created six levels of integration and autonomy for door control linking to alarm system.

This means things like times of day, depending on day of week, which doors they can access, and expiry dates and the like (i.e. some way to be sure a lost card is now invalid).

However, the DESFire cards are a challenge at best. I cannot have any operation that produces an error as that kills the authentication. So I have to only read files that exist, and only read the right length of the file, etc.

My original design was to have files for various aspects of security. Allowed doors; Barred doors; Start and end times; Start date/time; expiry date/time. The idea wast to get the file ID list and decide what to check depending on what files existing.

But even that is a challenge if any files can be variable length. The allowed/barred lists are like this, as are other files. So you have to check the file size for each file first.

Before you know it you are doing dozens of operations to the card, each taking tens of milliseconds.

So I ended up making a system that uses a single "access file", which I expect to exist. I ask the size and then read it. In that file I encode the various things I need like allowed and barred doors, and access times. But it is just those two operations on the card and hence a quick operation.

The end result is an autonomous door control system on an ESP8266 based door controller. It even has expiry update so that a card not used for X days expires. I obviously have blacklisting, but that also zaps the file if a blacklisted card is used.

The next step is some key roll-over logic as well as extra belt-and-braces old card expiry logic.

This is giving a fast enough process for door control. Yay!

DoH and VPNs and trust

We live in a strange world - where trust is a complex issue.

Once upon a time we would all trust the "authorities", i.e. the police and our own governments, but increasingly we live in a world where a lot of people have good reason (not criminal reasons, even) not to trust people.

The Internet is an especially complicated area where international players of all sorts come in to play, with commercial and political and criminal reasons to cause you concern.

The Internet protocols have been built on a lot of trust, but now we see some new mechanisms to help, two of these being DoH and VPNs.

DoH

DNS over https is one element, with DNSSEC being another. Using DoH means you use an https request to some external server to make your DNS requests.

An https request looks much like any other, and could as easily be your accessing facebook as accessing a DoH server. It is not something that can be snooped on, or selectively blocked.

If you do not trust your ISP to provide "clean" DNS without filtering or snooping, DoH allows you to choose someone else to trust. This is the problem, you have to trust someone, but you have a choice of who you trust.

In addition to DoH, you can also use DNSSEC to validate the accuracy of the responses. Using DoH means someone in the middle cannot snoop, or easily do any selective blocking. But whoever offers the DoH service could.

VPN

A VPN provider works in much the same way - you effectively choose a different "ISP" to provide your Internet access via the ISP you use. Again, choosing who to trust.

I was surprised how popular our own (unencrypted) L2TP service has been at A&A. In time we'll be offering IPsec based virtual ISP services too, I am sure.

Browsers doing DoH

Mozilla are working on using DoH in browsers, which means someone (like an ISP) cannot snoop, or selectively block, DNS requests. It is sad that this is even necessary. Note that AAISP do not filter or block any DNS, and have no plans to.

Oddly this upset ISPA, who has considered making Mozilla their "Internet villain" this year for DoH work.

This seems odd. If an ISP has an order to block some DNS, then they cannot block DoH, but so what? they are complying still!

I was surprised ISPA took that stance, even as a joke award for Internet villain. I can only hope they do not select them as the villain.

So A&A have donated the same amount as an ISPA membership, £2,940, to Mozilla. We have not been ISPA members for some time, but this is the first time I felt ISPA were perhaps taking views I did not really agree with.

We all benefit from the work of Mozilla so much every day, this seemed well worthwhile.

Amazon Yesterday

Amazon do a lot right, but sometimes they do some of the simple things wrong.

Getting it right

They are quite good at showing when they can deliver, clearly, e.g. "Get it by tomorrow if you order within next 5 hours", etc.

They are quite good at then managing to deliver on the agreed date.

Caught out

However, I have been caught out by changing delivery address (e.g. to Wales) and the dates change. This is actually quite good of Amazon, as they have worked out they cannot do the same delivery date, but I have been caught out and not noticed the change.

Things can go wrong

Obviously things can go wrong and they miss the agreed date. Nobody is perfect. They are pretty bad at then sorting it quickly or telling you when it will arrive, but thankfully this is not that often.

Getting the simple things wrong

What does bug me though is some of the silly things they get wrong. They will have "get it by [tomorrow]" on the buy now page but when you order the confirmation may magically change to the day later. If you don't spot it, you can be waiting in all day for a delivery that won't come!

I have also seen, and documented, cases where the delivery date changes from one screen to the next - e.g. the list of orders may say "arriving today" but you click on it for details and it is "by 9pm tomorrow". How the hell do they get something that simple so wrong.

Amazon Yesterday

You will have seen the excellent spoof https://www.youtube.com/watch?v=HA_gwzx39LQ for Amazon Yesterday shipping.

Whilst time travel is not really possible, their app is actually offering this. I just recorded it on my phone - with Amazon actually offering to have the item arrive yesterday.

Now how the hell do they get that so wrong?

What if I eat it all?

How is it that food labelling does not require a column for "what if I eat it all?".

How hard is that?

This is a typical example of food labelling, and it shows values for 100g and a 23g "serving" on a packet of crisps that is 95g.

I have an A-level in maths but it even takes me a moment to work out that I just ate 50.35g of carbohydrates.

Seriously, how hard is it to actually state what is in the actual packet you have? Surely that should be a basic requirement?

FFS 23g is not even a nice fraction of 95g. It is obviously intended for me, and 3.13 of my close friends to share...

Soft RS485 UART using interrupts on ESP8266

My non technical readers can skip this :-)

Background

As you will have noticed, I have made a complete access control and alarm system based on a Raspberry Pi but working over RS485 bus connections to standard Honeywell Galaxy parts like RIO, Max Readers, and Keypads.

As you will have noticed, I have now started making WiFi connected parts for the alarm system based on ESP8266 parts (ESP-01, ESP-12F, ESP-12S). These include a small ESP-01 based board with RS485 driver that fits inside a Galaxy keypad talking RS485 to the keypad.

As you may have noticed, I have NFC readers sorted and working with secure NXP MIFARE DESFire EV1 cards. These are way more secure than poxy proxy cards used by Max Readers.

Challenge

The concern some people have raised is the reliance on WiFi. It will do the job, but it is not that hard to disrupt externally. Now, much like using a screwdriver on a Max Reader, it will set off the alarm to disrupt it, but it is perhaps a tad easier. One has to consider if it is easier than simply forcing the door or breaking a window, which is always the benchmark for any security :-) But it is a concern.

So can I make the door controller, which does the secure cards, look like a Max Reader and do RS485 to a conventional Galaxy system? I think I can...

Approach

Initially, for the keypad system, I used the built in serial drivers on the ESP8266. These are good, interrupt driven, well supported, except...

The code on the ESP8266 using Arduino is the NONOS SDK, and means one thread for all. It can be delayed a bit for all sorts of reasons, even many milliseconds. For serial that is all handled behind the scenes on interrupts this is no problem, until is it RS485!

The problem is that for RS485 you have to enable and disable the line driver around your transmission. Initially I simply set a time to disable after poking the message in the Tx queue. But that could be delayed and mean after transmitting we held the line driven for too long.

The solution was to wait in the main loop task until sent, and then disable the driver. This is not nice, as any other operations of the device are delayed, but for the keypad driver application there is nothing else. So it works well.

This will not work for the card reader as it has to, well, talk to the card reader, and inputs, and outputs for the door control, and so on. Also, if I am a slave device (unlike the keypad where I am master) I am expected to answer polls promptly and not hang around for 10ms randomly.

My plan - interrupts! Make a soft serial RS485 UART on an ESP8266 to do the grunt in the background.

UARTs

A UART (Universal Asynchronous Receive/Transmit) is a device to send and receive old fashioned serial byte / character data. The concept dates back to what? the 50s or 60s. It is simple...

Each byte, or character, to be sent is coded as a series of 1 or 0 on a bus. What 1 and 0 mean in terms of current or voltage depends on the flavour of system (RS232, RS485, RS422, etc). Each bit time (1/Baud rate) the line has a 0 or a 1. The line is idle (1) and then the byte/character starts with a start bit (0), bits in the byte (low bit first), then any parity bit, and then stop bit (or bits) which are 1.

Obviously both ends have to agree the rate (Baud rate) for bits, and byte size, and parity present and if even or odd, and even number of stop bits.

The Galaxy uses 9600 Baud (so a bit every 104⅙µs), 8 bit data, no parity and one stop bit. Simples!

Sending is easy, you need some sort of timer, and you set 1 or 0 on the output to do the start bit, data bits, and stop bit. The Arduino ESP8266 SoftwareSerial sends in real time from the command using careful timers, it seems. I plan to use interrupts.

Receiving is a tad harder. You have to find the start bit for each byte, and then sample each bit, ideally in the middle. But once you have the bits you shift in the data bits and check the stop bit is 1 (else an error). Not that hard?

Edge interrupt on rx?

Interestingly the SoftwareSerial code simply interrupts on each edge and looks at time from last edge to work out the received data (I think). That is one approach but I decided to clock in bits on interrupt - I just need to sample mid bit somehow.

My initial approach was to set an interrupt on the falling edge of the data line to set the recurring timer half a bit later. However, this has all sorts of challenges, no least of which is race conditions on a regular per-bit interrupt and the edge interrupt. I did work, but not 100%. I am sure I could have solved the race conditions, but actually there is a good reason to stick to simple regular interrupts.

So the approach I took was just a regular interrupt, and this was 3 times per bit. This means ⅔ of the interrupts just count a sub bit and return in most cases, so very light weight in terms of time. But what it does mean is that I could in fact have multiple instances of RS485 busses running on this one interrupt if all the same Baud rate. I have not yet coded that though. It is, however, nice and simple and working for one instance well.

Why 3?

The 3 times per bit is simple. For Tx it makes no odds, skip 2 out of 3 interrupts and set the line to 1 or 0 for the next bit to send. For Rx, it is the same for sampling, except when waiting for start bit. If waiting for the start bit you check for the line going 0 every interrupt, and if it does set the sub bit counter to sample one interrupt later and then every 3 interrupts.

This is not as clean as an edge interrupt adjusting the timing to the microsecond, but it does mean that we can have sampling of the centre of the bit, ±⅙ of a bit which is more than adequate. Doing 2 times bit rate is no good as you could be sampling in the middle or right at the end. Hardware UARTs usually do 16x bit rates, but 3x is fine in my opinion.

This is how I have coded soft UARTs for over 20 years dating back to PIC16C84s, realy!

Messages

The way the RS485 bus works on Galaxy systems is that messages are sent as a sequence of bytes with no gaps, and then a gap - no length indicator in the message (but there is a checksum). So I coded the system to handle this, and even, if we are a slave, auto respond with current status response.

This allows the application to work on a message level, and if it is slow, we answer polls promptly anyway with current status. The driver also does the checksum.

Interrupts on ESP8266

Now the challenge is making it work on an ESP8266. They are rather special. They are a weird mix of old and new. The ESP-12S has 4M of flash! As an embedded controller that is huge! But it has around 80k of RAM (yes, k) and I had a BBC micro with more in the 80s (32k main and 64k second processor). The fact it does TCP at all is mental, but it does, and it does WiFi too.

The usual pressures on coding are complex - embedded systems have pressures on code space, RAM, execution time, power, all sorts.

In this case interrupts are special. The processor has a cache for the code, but this cannot be used by an ISR (presumably some interrupt process handles cache fails). This means the ISR has to be in RAM.

The trick is marking the code you use for the ISR with ICACHE_RAM_ATTR attribute. The RAM used is small, so not only does the ISR have to be fast it has to be small in code space too.

Thankfully a UART is simple code in terms of space and time. Small TARDIS code, even?

This worked, well, sort of, it worked mostly if started after about a second after boot. It turns out that using digitalRead, digitalWrite and pinMode is bad as these Arduino functions are not marked ICACHE_RAM_ATTR. Instead you need to find the underlying GPIO_REG_WRITE and GPIO_REG_READ functions. That then works reliably. The GPIO access is also faster, which is good, but very processor specific.

The result is working RS485, on interrupt, on an ESP8266!

I even added a clock track (blue on that trace) for debug - toggling an output for tx and rx clocking per bit.

It works!

This was deployed as new code to talk to the keypad, but means I can also act as a slave and pretend to be a Max Reader and provide a fob ID derived from the NFC card ID when checked using AES handshake.

It is also quite important as I am using the hardware serial at 115200 Baud to talk to the PN532 NFC card reader!

P.S. I have tested now as both master talking to Keypad, and as slave, pretending to be a Max Reader.

Choosing interface for PN532 (I²C, SPI or HSU) to ESP8266

I have been working with ESP8266 controllers and NFC readers. My preferred reader is the PN532 based Elechouse reader. As I have noted before, buy from Elechouse direct as there are a lot of bad copies!

However, when connecting to the processor I have a choice of interface. It is one of the rather nice features of the PN532. There are three interface types I²C, SPI or HSU.

So how to choose...

I²C

I²C stands for Inter-Integrated Circuit, so the acronym is IIC, but someone thought it would be fun to treat letters in an acronym in the same way as algebraic variables and write II as I². Arrg!

It is a simple bus for connecting devices and uses a clock and data line. Wikipedia has details (here). Both clock and data lines have a passive pull-up resistor and are actively driven low by master or slave. The protocol is well defined, and works well. It allows multiple devices on one bus and is very popular for a lot of peripherals.

Looking at the Elechouse web site I saw this: "On-board level shifter, Standard 5V TTL for I2C and UART, 3.3V TTL SPI" which rather concerned me. The ESP8266 is all 3.3V, and I did not want to have to mess about. This immediately put me off using I²C. However, looking at the schematic, the level shifter is completely optional, in that the same two pins are used for SPI, I²C and HSU and connections have these at the 3.3V level direct from the PN532 (via 100Ω) as well as via the level shifter, so it can be used for HSU and I²C if you want.

✓ Only two wires, simplifies wiring and frees up GPIO pins
✘ ESP8266 does not have hardware I²C support, so uses more processor and is slower
✓ ESP8266 has software I²C which means almost any GPIO pins can be used for clock and data, making PCB layout much simpler.
✘ Data sheet suggested level shifted to 5V (which turns out to be optional)
✘ Both clock and data use passive pull up resistors, meaning they can be subject to more interference on long lines

SPI

SPI stands for Serial Peripheral Interface. Wikipedia has details (here). It is more complex than I²C in the way it works - using 4 wires not 2, and allowing different configurations. It is also quite fast, allowing several MHz operation.

The ESP8266 has hardware support for SPI, which is good, but the downside of that is that specific GPIO pins are used for 3 of the 4 signals, so PCB layout is more complex.

✘ Four wires, so more to connect from controller to reader and uses more GPIO pins

✓ ESP8266 has hardware support, so allows faster working

✘ ESP8266 has hardware support, so the pin assignments are fixed making PCB layout more complex

✓ Lines are driven (from master or from slave) rather than using passive pull ups

✓ 3.3V direct connection to ESP8266

HSU

HSU stands for High Speed UART. This is simple old fashioned serial communications, on a Tx and Rx line, and defaults to 115,200 Baud.

The ESP8266 has hardware support for UART, and will work at these speeds. The ESP8266 library has good, interrupt based, serial handling. Unlike SPI there is actually a choice of pins that can be used, but this is still a tad limited. Of course the ESP8266 uses the UART for other things like programming and debug, but there is an additional Tx available for debug (Serial1).

✓ Only two wires, simplifies wiring and frees up GPIO pins

✓ ESP8266 has good hardware and library support for serial UART

✘ Pin usage is a fixed, but a choice of two sets

✘ Data sheet suggested level shifted to 5V (which turns out to be optional)
✘ Serial on ESP8266 is used for programming and debug
✓ Tx and Rx are driven (one each way) rather than passive pull up resistors
✓ The same connector can use used for programming and connection to PN532, reducing number of headers needed on the board

What did I do?

I was initially put off by the level convertor comment, leaving SPI as the only option. The first project I made using an NFC reader was connected to weighing scales, and used the serial on the ESP8266 to talk to the scales, so that rules out HSU. So SPI it is.

There are 4 connections needed (so 6 wires including power and ground). This means you can use 5 pins in a row on the ESP8266 (VCC, D13, D12, D14, D16) and GND to make a 6 way connector. Sadly the pins are not quite the same order as on the Elechouse PN532 board. This just means the connectors need wiring carefully.

This works well. I then used the same design for the door controller.

Hindsight

Firstly, the fact that the level convertors are optional means I²C and HSU are an option. I could have, for example, used I²C on a smaller ESP-01 for the scales rather than using an ESP-12F and SPI. However, the use of passive pull-ups is still a concern if the reader is on any length of wire from the controller as may be the case on a door controller. I could have used HSU, as the ESP8266 libraries have a soft serial mode, which can work at the scales 9,600 Baud rate, so could have used Tx/Rx for PN532 and one of the other pins to talk to the scales serial interface, and used an ESP-01.

I have since tried HSU, which just works, and is only 4 wires to the reader (power, ground, Tx and Rx). I was concerned that it would be slower (115,200 Baud rather than 2MHz) but actually it is slightly quicker (28ms rather than 30ms for InListPassiveTarget on same card), and a lot of the time is the PN532 operation and the NFC card itself. The Baud rate can also be configured!

It also turns out that had I used the more usual GPIO15 as the SPI SS pin not GPIO16 I would have actually wired the PN532 such that it could be used for HSU or SPI, as the PN532 uses MOSI for Tx and SS for Rx and the ESP8266 alternative Tx/Rx for UART0 are on those pins!

In practice, what I have done is make a 4 pin header with GND, 3V3, Tx and Rx. This is the same order as the 4 pins on the Elechouse board, but is also a header that can be used for programming the ESP8266.

I am slightly kicking myself for not trying HSU and I²C to start with, but to be honest I was just pleased to get things working. Freeing up two extra GPIO pins is nice, as is having fewer wires to reader, and pins in the same order both ends of the cable. The libraries do, however, make it very easy to change interface. Just remember to set the switches on the Elechouse board to match what you are using.

I knocked up a library

So I have made a library from scratch today for NXP MIFARE DESFire cards using AES.

Been a long day.

But here it is: https://github.com/revk/DESFireAES

Have fun.

P.S. Updated and tested and, and made an ESP8266 version....

NXP MIFARE DESFire EV1 for access control

Earlier this week I set myself the task of understanding the NXP MIFARE DESFire cards.

This was more of a challenge than you may expect because the manufacturers don't publish the protocol (except under NDA). Why?!?

• It does not stop people getting the data, as is apparent!
• It means there is also misinformation out there which cannot easily be corrected by reference to the official manual.
• It puts people off using the product, which cannot be good for sales.
• It means subtle details in the official manual will not be known, which could lead to failures in edge cases, making the product look bad.
• It does not help security - as any secure system must not rely on secret specifications.

I have written up what I have found out so far: PDF on github

So, why not just use one of the few (unofficial) libraries out there?

• I probably will for the host application on linux anyway.
• Coding it myself helped me learn more about the cards and how they work and what they can do.
• It was fun!
• The code on my ESP8266 devices to read cards does not need what is in a library - half of it is for handling legacy formats, and DES and 2KDES and 3KDES whereas I need a few specific operations using AES only.

The result - I have managed quite a lot, and made the door control system use cards which cannot simply be copied (unlike the 125kHz proxy tags, which can be, very very easily).

I have even managed to convert from DES to AES for the master key, and crucially I have documented detailed examples of this, and the CMAC logic, and so on, in the manual I have written. A lot of what I have learned has come from other sources as listed at the top, but I have gone through testing things directly and coding stuff myself to confirm things as well.

I have a cyclic record file on the card logging usage (door ID, and timestamp). I still need to add some handling for expiry, and time period controls on the card, maybe.

I am a tad concerned over response times. I am selecting the app, authenticating, getting file list, writing a usage log record and updating a counter, and you notice the slight delay using the card, which is a concern. This may be something that can be improved, not sure. I have not worked out how much is the card, and how much is the ESP8266 doing AES!

But it is working! I am even checking the CMAC on responses to confirm no man-in-the-middle upgrade attack (making one card seem like another once it has done AES handshake).

Happy to take comments and update any mistakes in the manual, but I hope it is useful to people.

P.S. speed improvements by tweaking what operations I do and in what order. Now works very slickly.

P.P.S. I made my own linux/C library (here) and light weight ESP8266 functions (here).

Tipping the scales

I am getting really good at upgrading the scales to do WiFi now, and read cards. The card of choice is a Monzo card which works perfectly well as an ID for weighing yourself. Notably Amex have an ID of 00000000 which is useless.

Really neat PCB, locking molex connectors, cables, the lot. It is working well. Marsden kindly sent the service manual so I can set to "kg" and "st/lb" modes for those that like old-school weights.

The system has a QR code on the scales, which allows you to register any card against your email address. You then weigh yourself and tap the card on the scales to record the weight.

When you tap the card it beeps to confirm it is working, and keep beeping until a stable weight is recorded over WiFi to the server. Simple UI if ever I made one!

Sadly, I have fried a couple of boards whilst doing these upgrades, but Marsden do spare parts (not cheap) and whilst I have a few clues what I may have done, I am not 100% sure of how I killed them. I think, with care, and anti-static, and being neat (avoiding any shorts, or diodes backwards) I can do this reliably now. Obviously very unofficial so I cannot expect any help from Marsden. They do make very nice scales though.

The next challenge is to make the web site that holds the data have graphs and sharing options and so on. At present it is just a list of weights. But even that is working for relatives using this solution.

This is turning in to the new fat-shamers club or something, not sure.

But what was the first thing on the web site? after registering the domain? well, it was getting an excellent lawyer to put together the privacy policy for us and ensuring we are GDPR compliant.

You have to love any lawyer that covers standing on scales with your pet cat as part of the privacy policy. Thank you Neil.