2019-09-14

Brexit countdown clock

Having made the environmental sensor, I have now cut down the design to just the processor and OLED display so I can make other things.

One example is a Brexit clock. I have put the code and PCB and 3D case on GitHub (here). It may need regular code updates, obviously :-)

It did raise one issue though. Most of the Brexit clocks I see, like Sky News's (here) seem to be an hour out. Brexit is at 11pm but they show days plus time that would be midnight.

Now, I know why... Someone is being lazy and simply dividing time of Brexit minus time now by 86400 and ignoring the clock change. One of the days between now and Brexit is 25 hours long, but that does not stop it being "one day".

My clock gets it right, obviously.

There are other uses for the app - and as I am taking one of my grandsons on a his first cruise, I thought he may like this...


He heard the phones in the suite have a button labeled "pizza" :-)
He is already packing!

P.S. This is why I have a milling machine and a 3D printer! I came up with the idea over coffee this morning at around 9am. I had to design a new PCB and a new case, mill the PCB, 3D print the case, and assemble it all, as well as make the code. I had a working version by around noon. My grandson is going to love this, I am sure!

2019-09-12

Environmental sensor

For my latest little R&D project I have made a new environmental sensor. These should help us see how good the air quality is in the office, and as usual, I have published this as an open source project for anyone else to make the same.

This proved to be a bit of a challenge, as always. I have already managed to get my head around the ESP32 and the Espressif ESP-IDF now, and have a set of  MQTT based tools as a base (here).

Hardware

I used an ESP32-WROOM-32 again, as they are really nice. I also used an SCD30 CO₂ sensor, which is the most expensive bit at around £40 from RS. I got a nice 128x128 OLED display from Amazon, and a DS18B20 temperature sensor.

One challenge was space - I decided to make the the PCB the same size as the OLED PCB and sandwich the CO₂ sensor in between. This was definitely a challenge to get connectors to fit!

I also decided to try the Molex SPOX connectors as they seem relatively cheap and easy to use.

The other aspect which is especially challenging is power. Most of the bits I have done to date have been alarm and access control and so use 12V on screw terminals. As it happens the unit (pictured above) takes 12V which is in the wall for the door control. But for use in bedrooms I need power to these and so I went for a micro USB. This is simply because that is a really easy and readily available and cheap way to provide a device with power.

The challenge itself is that the connectors have tiny tiny 0.635mm spaced contacts, and are also mechanically crap and come off the board. I ended up getting some connectors that I could superglue to the PCB, and thankfully the power contract are the end, so even though milling and soldering at 0.635mm pitch is just about possible (surprisingly) I can avoid this by actually only soldering the end pins for power. I decided to make the 3D printed case design fit tightly around the connector as well for extra mechanical support, except that means removing from the case pulls the connector off the PCB!

In short, some interesting new hardware challenges.

Software

There are a lot of libraries around for ESP32 and ESP-IDF already, which is nice - but perhaps not as many as for the ESP8266/Arduino based environment.

That said, these are simple enough devices, so actually I ended up doing the I₂C interface for the CO₂ monitor myself using the ESP-IDF directly.

I used a library for the DS18B20 though, and it seems they have been quite clever using the IR remote control hardware to make the one wire bus work using DMA. Impressive. I decided to use a DS18B20 as (a) it allows the thermometer to be positioned where you want, (b) it is away from the components that could get slightly warm, and (c) it seems way more accurate than the one built in to the SCD30 CO₂ sensor. However, the code is happy to use the SCD30 if you don't want to use a DS18B20. Similarly, the code is happy to work without a display, but I am quite pleased with the display, to be honest.

For the display, there is a good Adafruit GFX library, but I ended up doing it myself. The commands for the SSD1327 OLED controller are simple enough, and only a few lines of code to send a whole frame buffer over I₂C. As such, I decided to make some 16 grey level renders of my existing font designs. There is also a logo (configurable).

I found the CO₂ reading was liable to change quite quickly. This is great, except just standing looking at it caused it to change a lot due to your breath. So I ended up damping the reading (biased average with last reading) to remove spikes.

I also decided to round the values for reporting on MQTT so that it is not flooding with updates every second. Configurable rounding, preset to CO₂ at multiples of 10ppm, RH at whole percent, and Temp at 0.1℃. I also added some hysteresis to the reporting.

Obviously I have my air-con control code working with it - just had to tell it a different topic to watch for on MQTT, but I added logging of CO₂ and R/H as well just for fun.

And finally I added settings so it can send MQTT to turn a fan on or off based on CO₂ levels.

It is interesting to see how CO₂ changes during the day.

Overall I am very pleased with it. It is all on GitHub (here) with PCB milling, 3D printed case design, and code. Have fun.

P.S. The OLED displays are made of glass, and very very very easy to accidentally break. Especially if the 3D case design is a fraction out!


P.P.S. I have one by my bed, and it is quite bright, so I now have an MQTT contrast command tied in to the sleep tracking stuff so when I get in to bed it goes dim :-)


2019-09-04

Front door locks, at last

Even though I have had electronic door locks for something like 30 years, since well before I was married, my wife, for some reason, wanted a "normal" front door for the house. First units were made from a cassette player head (Sony Walkman, I think) and read mag cards (bank cards).

This is crazy (IMHO, but obviously I am "wrong"). For a long time it was a typical double glazing unit, but she then got a custom made wooden door. Nice!

Even though custom made, she was not keen on me insisting on electronic locks. I am not sure why.

Anyway, I did manage to get good euro profile actual locks, hard to pick, etc, and you need a 3D printer to make a fake key! Yeh, I know!!

The problem is we got some keys, and could get more, but with 5 kids and their various family you run out, especially if any are staying with you or visiting enough to have a key.

The end result was rather annoying for me. In practice the house is never empty, but having a key in a box outside the door is not a good security measure! Not quite a key under the mat, but damn close!

So finally I got a locking spindle.

I fitted it and routed wires in the mechanism to the hinge, etc.

The mechanical lock still works, obviously.

Unless deadlocked (as I say, never nobody here), the door can be opened by using the handle inside. Yes, that too is multi-point locking, but you just turn the handle.

Now the outside handle also works, if 12V is applied to the locking spindle. Suitable amounts of super glue, etc, make it hard to dismantle and operate otherwise. The door supplier has just used a split spindle which could easily be opened with a screw driver and replaced - crazy!

But this little device was simple, and around £60. There are options for a completely new multi-point locking strip with motors in it, but they are a lot more work and cost. This was surprisingly simple.

I already had a card reader on the wall near the front door, now an NFC reader using an ESP32 (inside) to check cards. That simply set the alarm, previously, but now can be used to "engage" the locking spindle.

End result is that I can now easily allow some access at some times of day, using DESFire cards, or even just bank cards, if and when I like, and no keys in a box outside. We have someone staying with us for a few days, and they were suitably gobsmacked that I just enabled their bank card to work the front door for a few days!

I am surprised more locks are not like this to be honest. The tech is way better now, but still, it has always been so much better than simple mechanical keys.

PCB milling

The new CNC machine is working well, so I thought it worth explaining a bit more on the PCB milling. I am getting pretty good at it now.

Drilling

I 3D printed a support for the PCBs, holding them a couple of mm off the bed. This allows me to drill through the PCB and mill around the outline. Designs on thingiverse.

Milling

I previously mentioned milling bits, and there are lots you can buy from Amazon and the like. However, the best one I have found so far is one from RS components (part 382002). It is not cheap (£40), but don't be fooled - it works, and works, and works. I did manage to finally blunt one after a lot of PCBs, but really it is very good. The small end means it does not snap off easily either.

I am using a 0.5mm drill to make holes, and a 1mm drill to mill around the edges.

Design rules

I am now milling for the finer pitch used by the ESP32-WROOM-32 SoC modules. So I have to be able to mail quite fine tracks with 1.27mm spacing.

What I have found is that, with that bit, the simplest rule is to ensure isolation tracks are at least 1mm apart. This is quite easy, using inkscape, I just set the tracks to 1mm wide and see if any over lap. I have a layer for the parts, a layer for the drilled holes, a layer for the edge mill, and a layer for the isolation tracks. I can then simply save each layer as EPS and convert to an NC G code file.

Faster milling

I also updated my code to take the EPS output from inkscape and make a mill file. It now sorts the lines, and makes paths with a much more optimal routing. On GitHub.

2019-09-01

Pololu D24V5F3

These are really nice power supply modules from Pololu..

They run from anything up to 36V and regulate down to a nice 3.3V. They cope if I reverse the polarity. They are tiny (0.4″ × 0.5″). They just work.

They cost a few dollars, and actually cost more than the ESP32-WROOM-32 (at current pre-brexit style exchange rates), but they really just work, and I am using them everywhere.

What I did not work out, when ordering a 100 of them, is that they are so recyclable! I am changing controllers from ESP8266 to ESP32, but the power modules can be easily desoldered and re-used. I have so many I am thinking of listing them on Amazon.

I whole heartedly recommend them, very nice, work well, don't get fried when you cock up.

2019-08-31

ESP32 Interrupts

Well, it has gone well... A week in Greece and a week back home, and I have my alarm system / access control code working.

In the process I have released an ESP32 / ESP-IDF based PN532 library which works with my DESFireAES library, and also a VL53L0X library.

The only bit that has proved problematic is the 9600 Baud RS485 Galaxy Bus library. It worked well on the ESP8266 using around 28800Hz timer interrupt to bit bash RS485. I coded the same on the ESP32, with much faster processor, and, well, two processors. It is being "iffy". It is close enough for now, but I need to fix.

I could go for using one of the three UARTs which includes some RS485 stuff, but that too needs some interrupts to get timing right, I expect. Getting the UART interrupts working was even more challenging than a simple timer interrupt.

I am working on updating my door controllers now. The new design is nice, and smaller than the old design.


So I'll read up more on interrupts :-)

2019-08-21

ESP32 progress

I am, of course, reinventing wheels, yet again with my move to ESP32 for IoT and access control stuff.

It is going well - I am actually on holiday in Greece, but whilst my wife and daughter sit on sun beds, I have been messing with code. Yes, I took a laptop, programming cable, some ESP32 boards, and even a hand held oscilloscope on holiday with me. Sorry. I spent 4 hours on the plane reading the data sheet on my iPad!

My plan is to get my ESP32 development environment and toolkit working like I had on the ESP8266, and build up my "Solar System" alarm and door control module code.

In some ways this is a nice project - it is often good to rework and redesign something for the second time as you have learned lessons and can do things better. It is also nice that my code effectively has a specification to work to. The previous code was being specified, designed, and coded all at once. But now I have a system which works with the ESP8266 modules and should work just the same with the ESP32 modules.

Progress

I am making good progress. I have the build environment on Mac and linux working. The instructions were very clear. I am actually ssh'd to my linux box to develop now. I have the basic support system which allows me to connect to the WiFi, and MQTT server, and do over the air s/w updates, and store settings in non volatile storage. These are all based on the ESP-IDF tools, which seem quite good.

I have hit a few snags - for example I cannot do certificate pinning on the TLS using the SHA1 hash of the certificate, it seems. I have to include the PEM format certificate itself. This seems needlessly complicated, but not a big change.

I have a snag with MQTT that it seems not to pass on messages with zero length data for some reason. I may have to work that out and do a pull request to the library.

I also have to restructure the design in some case - for example, previously I did the settings in a block of memory that was flashed. Now I am using the provided NVS library for settings. It seems good as it does not have to re-flash a block on every change - it incrementally updates a log of settings in flash as it goes. Quite sensible really. However, it seems to lack a means to enumerate the settings that are there - I have to ask for a setting by name. As a result I have changed my code structure to register settings from the application with my library. This has meant a redesign, but as always it makes it much better in the long run.

Update: As redesigns go - I ended up re-doing it yet again, and pretty slick now.

I should be able to sort the basic GPIO (inputs and outputs) really easily, and hence use almost all of my existing door control code with that. It looks like I can compile my linux version of DESFire stuff for NFC too, so no need for an ESP specific version (if needs ESP specific AES calls).

There will be some challenges with the PN532 library and VL53L0X library I expect. But they are not rocket science, thankfully. The Honeywell Galaxy RS485 bus should just work if I can work out the timer interrupt logic. Saldy each of these is simple in theory but getting my head around the specific way of working for the ESP32 takes an unknown amount of time. Interrupts were a challenge on the ESP8266!

PCB design

This had me kicking myself the night before we left on holiday. I have made up several ESP32 based PCBs. A general break out board; a door controller board; and several Galaxy keypad boards. I had checked I could load code on the door controller. It worked.

But I realised, on reading more of the data sheet, that 6 of the pins on the bottom of the module (ESP32-WROOM-32) are connected to the internal flash and so unusable. Why even have them come out of the module then? They are literally in the way of tracking the usable pins out of the chip. I had the same issue on the ESP-12F and taped over the pads. The ESP-12S sensibly does not have them, but then has a GND pad you have to tape over if you have tracks under it. The ESP32-WROOM-32 has both unusable pins on the end and a GND pad in the way. Arrrg! Even worse, it is only 6 pins of 10 at the end that you cannot use, so not actually easy to tape over just the 6 and still sensibly solder to others. I had read a web site on "safe" GPIO pins, but that documented the ESP32 chip not the ESP32-WROOM-32 module, and for that the pins 6-11 are "safe" to use as GPIO. There are other pins that are iffy - i.e. needs to be high or low at boot. Those I knew about.

The end result is that the keypad boards I made would simply not work! So I was left with three boards in total to take on holiday. The door controller would not "work" for the GPIO pins, but they are not connected to anything that would stop the chip actually working. So usable for developing code. I have managed to trash one board already with a flash / secure boot mix up!

The first thing I did when I got here was re-work my designs on the basis of taping all 10 connectors on the end of the chip. If I had not, it would have been bugging me all holiday. I had to be a bit creative in some cases, but I managed to redesign both boards. I'll mill them when I get back (I don't think my wife would have let me bring the milling machine on holiday for some reason).

Door controller PCB

Update: The RS485 is a challenge. I made a low level bit bashed solution for ESP8266, which I could use. It uses two pins, a Tx/Rx and a DE/RE pin. But the UARTs in the ESP32 have hardware support for RS485 as do the drivers, and are better done with separate Tx, Rx, and DE (RE permanently enabled). So I may re-track the PCB and make a driver using the UART. Again, a redesign of both PCB and code. Overall this is dragging on longer than expected.

Security

Security with any IoT stuff is important, and reading the data sheets on the ESP32 and using the build environment, I can see that Espressif have done their homework. Nothing is 100% secure, but they are really trying hard here - which is good news as a lot of IoT stuff is totally crap!

The chip has a secure boot mode. This means the first stage bootloader in ROM checks the signature on the bootloader in flash before running it. The second stage bootloader checks the signature on the application. Doing an OTA (over the air) s/w update also checks the signature, even. This means you cannot load wrong code on the chip!

It also supports encrypted flash. This means that I can have the code, and all my settings (like login details to the MQTT server, and WiFi) securely stored in the flash. Even pulling it off the board won't get you the data.

It has a set of eFuses, which are built in to the chip. They can only be "blown", so are bits you cannot unset once set. These include bits that stop you reading the eFuses, but they are wired internally in the chip hardware allowing it to encrypt and decrypt the flash. You can even set a fuse to disable the JTAG debug port. This means you can really lock it down so that you cannot get credentials out of the chip or the flash. It might not stop someone de-capping the chip package and using a microscope to read the fuses, but it will stop most conceivable attacks.

They have built in AES, SHA, BIGNUM (RSA) and TRNG hardware as well, and the API makes using https and TLS simple and encouraged.

In short, security seems to have been done very well, and thought about at the outset in the hardware and software design. This is almost unheard of in the IoT world, which is generally pretty shit at security. Well done Espressif.

Update: I said nothing is 100% and a recent tweet suggests hackers have managed to crack it using clock or power glitching, but Espressif have replied saying they are adding hardware changes in the next release out soon! So they do seem to be trying, which is way better than most IoT stuff.

Opportunities

Once I have this starting point - code and PCBs - and it is working, then (apart from updating my existing door controllers), I have to think of other uses for these chips. They are packed with stuff - included a dedicated separate ultra low power processor. The opportunities for very low power devices are interesting. Even just the bluetooth looks fun - I could make it that door entry using a key fob needs to be able to see my phone in range, for example. I am sure there are a lot of opportunities!

They have some interesting direct AP-less WiFi modes in the API as well, and some low rate long range (e.g. 1km) WiFi functions too. There is even a MESH WiFi system. This all makes for some very interesting possibilities to make systems that do not rely on infrastructure such as access points.

I still have not made my environmental monitoring device yet, so that may well be the next project after this.

2019-08-15

ESP-01, ESP-12F, ESP-12S, ESP32

As I mentioned, I am trying to move my IoT development stuff to ESP32, and notably move from Arduino IDE to the native Espressif ESP32-IDF using mostly C.

This poses a slight problem. The development environments are sufficiently different that it is something of a pain in the arse to maintain both. Probably just about possible, but not simple. So I'd like to basically abandon the ESP8266 / Arduino stuff (I'll leave on GitHub) and start working only on the ESP32 based stuff.

This poses a slight snag as one of the use cases for this IoT stuff is a very small device placed inside a Honywell Galaxy alarm system keypad.

This is based on an ESP-01 module which is ESP8266 based. It is not ideal as not CE marked, no RF can, very limited IO pins, but it is small. It comes with a 2x4 0.1" pitch header and is just the right size to fit a Pololu 3V3 regulator between it and the PCB, which is rather neat.

This use case has the ESP-01, a 3V3 regulator, an RS485 SO8 driver, and a small 2x2 milli-grid header for programming and debugging.

The whole thing fits in the small space inside the keypad, as per the picture on the right.

As a result I assumed I'd be stuck with, at the very least, my keypad code on the ESP8266 / Arduino based code.

The other IoT devices are generally based on either ESP-12F, or preferably ESP-12S modules. These are only slightly narrower than the ESP32-WROOM-32 modules I am now using, and so easy enough to make alternative designs using the ESP32.

The ESP32, however, whilst larger, has a lot of IO pins. It is possible to track these pins so that you have quite a few usable IO pins, and GND and 3V3, all on one end of the chip if you try :-


I have actually tracked Tx/Rx programming pins, and GND and 3V3 "through" some unused GPIO pins, which allows its all to come out of the bottom end and keep the whole thing quite narrow. Obviously this is fine as long as I don't make those pins an output :-)

Once again, the new milling machine is doing a great job, and I was able to solder it.


And, to my delight, it fits, and the keyboard PCB fits over it with no problems!


So, yes, I can actually move away from ESP8266 now. I have to migrate and re-work the various tools and modules I have made, but once done, I can continue development on ESP32 only. Yay!

P.S. Always tweaking :-)


2019-08-14

Moving to ESP32

Someone already told me that once I move to ESP32 I won't want to go back to ESP8266.

Well, apart from the fact the ESP8266 is available in a tiny ESP-01 package, I think they may be right. The ESP32 in a WROOM-32 package is slightly wider than the ESP12 package, but because the GPIO pins can be mapped any way you want, pretty much, the tracking takes less space. So overall I was able to make an equivalent board in less space using ESP32.

The big challenge for me with ESP32 was the concern over the fine pitch - 0.9mm contacts at 1.27mm spacing. I was not sure I could either mill a PCB or solder a board at that pitch reliably.

I was wrong, I can!

New door controller using ESP32

Fixing the X axis backlash on my mill was a big help as well. So yes, I can use the ESP32 modules.

The other concern was the price, but it seems that getting a proper ESP32 is as cheap, if not cheaper, than an ESP12S. Yes, some cheap ESP12S can be got, without CE marks, and missing FCC ID, and not quite working well on WiFi. But £2.91 for ESP32-WROOM-32 from a reputable supplier is pretty good.

There are advantages to the ESP32, obviously, including much more RAM, faster, dual processor, more GPIO, more UARTs, and so on. It is a good comprehensive IoT device.

The other big change I want to make is moving away from Arduino IDE. Yes, Arduino makes things easy, but it turns out the Espressif IDF for ESP32 is pretty good. It has the key things like https, MQTT, OTA, and all sorts as standard.

So my plan is to make an ESP32 IDF based version of my door controller, including PN532, VL53L0X, etc. This should not be too hard, and a lot of my existing code can easily be ported to a simple C code environment.

I am, however, impressed with the documentation - not only of the APIs in the IDF, but also the technical reference manual. It actually details all of the registers clearly.

The other impressive thing is Espressif seem to to be trying to encourage some best practice in IoT. They have secure boot and signed images as a standard feature. Even https libraries, as standard. It means that making a reasonably secure IoT device is easy. Well done.

I managed to get the build environment working on Mac and linux with no problems, following the clear instructions. I built the "hello world" example, and flashed it with no problem.

I am off on a 4 hour flight shortly, so I have the 669 page PDF of the technical reference manual on an iPad for some light reading on the flight.

Of course, once I have this off the ground, I can look at Bluetooth, and CAN bus, and all sorts of new fun things.

2019-08-10

New milling machine

I had a CNC 3018 PRO machine from Amazon.
  • It needed assembling, but I've done that now. It is something to bear in mind if buying one though.
  • It is a bit of a "home made" style, with the PCB bolted to the frame, etc.
  • The one I got had a bent screw for X axis, which limited the usable space a bit.
  • It has a working area 30x18cm.
  • I was able connect a Z-axis probe (well, contact to PCB copper clad board).
  • It works well, and I used bCNC on my Mac to work it.
  • It comes with a controller box, which is not needed really!
  • It is surprisingly cheap.
So, especially given the bent screw, I decided to get a replacement, and went for a CNC 3020 from Amazon.
  • This is a much more "industrial" unit - proper drag chains and cables. Looks good but costs a bit more.
  • Work area (you guessed it) 30x20cm
  • Comes pre-built
  • Has a boxed controller and power supply, with nice connectors to the machine.
However, it has a challenge or two - the main one is that it works with some very specific software (a pirate copy of which is apparently supplied with it). It has parallel (!) and USB, but the USB only works with that software and only on a windows machine - it does not even appear on a Mac even in a debug log! The software did not look too good anyway! This is a bit of a bugger.

The upgrade

My solution was to upgrade the controller to a TingG controller. This adds a bit to the cost (especially with the customs/VAT and admin charge from US). But was not actually at all hard to do.

The old controller is bolted to a heat sink, and can simply be unscrewed.

The new controller is slightly smaller. I needed to drill/tap an M3 hole in the heat sink for one of the screws so that I could screw it to the heat sink. I also cut an extra hole in the case for the USB connector.

Also, I decided to use the heat sink, which meant something between the new controller board and the heatsink. I got some small copper block / heat sinks and thermal tape - and I used them between the back of the new board and the existing heatsink.

The TinyG is designed to work without the need for a heatsink, but this can not really do any harm.


The connectors on the old board were not the same, though the new board has pads which means I could have changed them. I decided instead to use the screw terminals on the new board, fitting bootlace crimps on the wires. I had to swap the cables to make them fit, and use the motors 2, 3 and 4 as closest to edge of the case. But they all fit!


I also connected the emergency stop to the reset pins. I have not worked out how to connect the spindle drive to the new board yet - that is a challenge for another time. The case has a button and a dial for that which works for now.

I also got a 6 way (plus ground) chassis DIN connector and wired up all 6 end stops (X/Y/Z Min/Max).


This meant various micro switches, and super glue on the actual machine.


And also, the Z-Min I wired to croc clips to allow me to use contact to PCB for homing Z axis.

The TinyG has good documentation, covering setting the current, etc. I was able to set the axis to the right motors, and set the polarity and travel per rotation, etc.

However, the bCNC code I was using did not like the TinyG, and I ended up installing CNC.js which works nicely on my Mac. It does lack the multiple point Z axis levelling of bCNC sadly, but works.

I did have to configure the homing for Z axis a tad as it is designed for a switch from which you back off. For PCB contact you get the exact Z home, and do not need offsets. But this is all well documented.

Slack (backlash)

However, even though it is a really good solid construction and seems to have no play, I found there was an issue with the X axis. It seems to have around 0.1mm slack on it. This means if you move right to a point it is actually around 0.05mm short of it, same if moving left (i.e. is then 0.05mm right of it). Make a row of left and right moves and lines and you see the problem clearly. This test shows it well - centre is left and right moves before each line, but left is all moves from the right (apart from first, bottom left) and right is all moves from the left. Repeatable slack!

Oddly the Y axis is absolutely spot on!

This is not really enough of a problem to cause issues making PCBs, but is annoying so I wanted to fix it.

I checked for anything loose, and also checked the TinyG for any options, but to no avail.

My solution (now updated to the eps2gcode tools) is software to compensate for slack. This seems to work.

Overall I am quite happy with the result.


P.S. Tightening this nut a bit fixed it.



P.P.S. After reading the TinyG docs I am actually running at 4 micro steps not 8, so 1/200th mm spacing, which I think will be more than adequate for anything I am milling. I suspect 1/100th mm would be fine but actually that is a lot more noisy. Now that the backlash is sorted maybe I'll go back to 8.

2019-08-08

Body cams are weird #GDPR

Body cams are weird - you walk in to a shop with one on, especially with a big WARNING: VIDEO/AUDIO RECORDING sign on it, and you get strange looks. But at the same time, half a dozen CCTV pointing at you, is "normal".

I have yet to be challenged on this, and I am not sure I really want the altercation, but I imagine if asked to "turn that off" it would be valid to say "I will, if you turn that off [point to CCTV]".

Let me stress, I am not a lawyer or GDPR expert, but I'll try and make sure this post is technically accurate and update any errors pointed out to me. And thank you Neil for the links, case references, and definitions and correction of my many typos.

Domestic purposes

One of the things that is outside the scope of GDPR is processing of personal data by a natural person in the course of a “purely personal or household activity”, which is commonly called processing “for purely domestic purposes" This is a tad complicated in interpretation though, and there is some confusion on this.

There has been a judgement on fixed position CCTV cameras recording in a continuous loop in people's homes (domiciles!) that says that it is only domestic purposes if there is no view of any area outside the property boundary. i.e. if the CTTV covers the road, or neighbour's property, it is not domestic purposes and hence you need a lawful basis, a privacy notice, to comply with data subject rights, and everything else that goes with GDPR compliance.

Personally (and I think I am not alone on this), I think this ruling is wrong: The cost seems to have confused the setting of the processing with the purpose of the processing. The classic example of domestic purposes, an "address book", will have names and addresses and phone numbers of people not within the boundary of my property. So clearly it is not the data subjects that define "domestic purposes" but the "purpose" of the processing (the clue is in the words used!). But as we have a ruling, that becomes risky to try and argue that CCTV covering the road is domestic purposes.

Indeed, for security of my home (domestic purposes) I want recordings of people "casing the place" (on the road or opposite my home) as that may have much better, daylight, views of faces to use when later the same vehicle is caught on camera relating to the place being burgled.

Dash cams, and helmet cams, were at that point considered "OK" though (or, at least, there has been no case law yet involving them), which leads to a whole series of scenarios where one cycles home and puts the helmet cam on the wall of the house, and then connects to power, and so on - at what point does it change from "OK" as a helmet cam covering the road, to not "OK" as a CCTV covering the road?

So, to be clear, if you have seen my body cam and found this post because the URL on the label, my purpose in recording is domestic purposes - I record my cycle rides mainly. Just like a dash cam in a car.

Continuous recording

There is some draft guidance on the matter of dash cams which actually suggests that dash cams must not  be continuously recording!

Yes, you read that right - it seems to suggest that you have to start recording just before the "accident" you want to record. I know, that is utter madness.

Of course there are plenty of people who will jump in and point out the technical solution is a pre-recording buffer. That is only saved if you "press the button" (after the accident), or the camera detects a collision, even, which some can!

But that *IS* processing personal information, even if it just goes in to RAM and is replaced after 5 minutes. An example of why that is processing is simple - imagine a dash cam where the camera owner is the cause of an accident. They have a choice about processing of that personal data in the RAM of their camera - they can choose to "press the button" and save the (incriminating) evidence, or not. The data subject can, if they are quick, ask for a copy of that data, at which point they have to "press the button" as otherwise they are deleting personal data after being given (what used to be called) a subject access request. Clearly the data in the pre-record buffer is personal data, and there are data processing decisions to be made in relation to that data.

We can only hope that such a nonsense paragraph is removed from this draft guidance (or, even better, replaced with something more akin to common sense, that someone trying to protect themselves from an accident, or put themselves in a good position if they do suffer an accident, is “domestic purposes”, even if it films someone else in a public setting).

Transition from domestic purposes to commercial

There is then one area about which I am really unsure. If I have personal data, recorded purely for domestic purposes, e.g. my address book, but I then tweet that!

In another recent case, the Court of Justice of the European Union has held that "since [someone] published the video in question on a video website on which users can send, watch and share videos, without restricting access to that video, thereby permitting access to personal data to an indefinite number of people, the processing of personal data at issue in the main proceedings does not come within the context of purely personal or household activities”.

Applying this to a dashcam or helmet cam, even if my filming would be considered to be “domestic purposes”, posting the resulting video on Twitter or Facebook (e.g. to highlight bad driving, or just as part of a journal of my day), unless I lock down access to the video, it looks like I would be outside the scope of domestic purposes.

Given how many people live out their lives online, engaging in the type of chat that one might previously have had in a pub or coffee shop, this feels like it could entail the imposition of GDPR — which, let’s face it, is not something normal people should have to understand — on the general public, for very common activities. It seems to me like rather too much of an interference with people's private lives.

Somewhere in between

Sharing a family video (e.g. my grandson riding a bike for the first time) with family is clearly domestic purpose.

I assume doing so over iMessage is still so, as I do not hand the video to Apple (end to end encryption).

What of posting to a small (family) group on Facebook? My "purpose" is clearly domestic, but Facebook have the image and they do things with images that are not domestic purposes.

What does that mean for GDPR I wonder? I assume that this means that Facebook needs to comply with GDPR, but I do not (since I am not sharing with an indefinite number of people).

2019-07-29

🚲 Paint only cycle lines

I shared a slightly amusing video of an "incident" on my way back from costa (here) and a friend of mine commented that there was a cycle lane next to the road on which I was cycling, and asked why I was not using it.

I think he genuinely does not realise why I would not use that cycle lane. So I wanted to try and explain a little.

Firstly, and importantly, there are quite a lot of original cycle lanes in Bracknell. The town planners included these, and they go places that are different to the roads (see the tree lines on the image on the right, for example). A lot of them are a proper road surface, wide enough for bikes to pass each way, with proper curbs and separate pedestrian paths along side. They link sensibly to roads, and notable go easily under the major road junctions. These I use, when they go where I want to go. Sadly, even though they have clearly distinct pedestrian paths along side you still get pedestrians meandering in the cycle path, which is a nuisance, and seems to be for no obvious reason or advantage to the pedestrian.

However, like a lot of places, there are a lot of cycle lanes in "paint only". I.e. they are former foot paths that now have paint and signs. Some are split with cycling one side and pedestrian the other side, but quite often it is just shared use on what used to be the foot path.



So, I want to try and explain to my friend, who drives a Tesla. It is a nice car, and if I get a car it will probably be a Tesla. They are quiet, and quite impressive.

Dedicated Tesla lanes

So imagine the council were going to make dedicated lanes for Teslas. Now, ideally these would be specially suited to such a car, smooth road surface, good white lines so the auto-pilot can see them clearly, fences to stop pedestrians accessing the lane, maybe even a special higher speed limit. If you have a special Tesla lane that gave you some advantage over the road you would use it.

Except...
  • The special lane is actually a bit narrow
  • It has a poor road surface
  • Access to the lane is not seamless - it has a bit of a step up, or some cobbles, or a bollard you have to go around, or even a chicane / kink so you have to manoeuvre to get on to and off the lane.
  • The lane has numerous obstacles (signs, bins, bus stops), with just enough space to get around them, but you have to watch out for them and manoeuvre around them.
  • Some of the signs are two pole sign where the sign you have to go between, but the sign is low enough you are not sure you can drive under, or some Telsas won't fit under.
  • The lane is actually two way, but not mostly wide enough to pass other vehicles, so you have to co-ordinate with the other driver to pass where you can, slowing down, moving over, etc.
  • There are pedestrians allowed on the lane, and they don't hear you coming (Teslas are quiet) so you have to beep the horn, and they get all indignant. Even when there is space to pass them, you have to slow down in case they step in front of you or wave their arms about for no reason.
  • The lane does not actually go very far, it stops at the next side road, and you have to get off the lane and cross a side road (giving way). You may even have fewer rights than a pedestrian (who has priority over cars driving in to the side road, not that many drivers realise that). Then you can get on to a new special Tesla lane the other side of the side road.
  • Actually, often there is no follow on lane, you have to re-join the main road, or perhaps there is another lane the other side of the main road.
You could suffer all of this, or you could just use the main road, as you are entitled to anyway, and which goes the same place as the special lane (without all the hassle and giving way at each side road, etc).

As a Tesla driver, would you use these special lanes?


2019-07-18

Environmental sensors

For my next little project I am thinking of making a good little environmental sensor box, mainly for CO₂ and temperature as they are the two things I have some control over.

ESP8266 based, why not. But I have been looking at the sensors you can get, and bought a few.

Are all CO₂ sensors crap?

Well, I tried a few, including this CJMCU module (CCS811 based) and this AppliedSensor module.

Sorry to say, but both are crap! And the detail is in the small print - they estimate CO₂ from other sensors, and are very easily confused. I have some isopropyl alcohol and you use that anywhere near them and they quickly say 18000ppm CO₂. Even without such factors they swing wildly all over the place.

It seems you need an NDIR based sensor, and I found this one which seems to actually work! This SCD30 sensor seems to be the business, with CO₂, temperature and humidity (floating point!) via I²C.


Next step - sort a suitable display and box.

I can then use the temperature to control the air-conditioning, and the CO₂ to control extractor fan speed.

2019-07-13

Door control

My door control and alarm system has made some progress.

The main change is making access control a bit more distinct from alarm system. The doors at the office were "falling off" the WiFi occasionally, even if only for a few seconds at a time, a few times a day.

I suspect that can be fixed, and it makes a huge difference what channels we use. The AP report something like 60 interfering APs.

So the change is to make the doors work autonomously which means the door controller has to be able to authenticate the fob/card used but also know enough to tell if it is allowed access.

I have actually created six levels of integration and autonomy for door control linking to alarm system.

This means things like times of day, depending on day of week, which doors they can access, and expiry dates and the like (i.e. some way to be sure a lost card is now invalid).

However, the DESFire cards are a challenge at best. I cannot have any operation that produces an error as that kills the authentication. So I have to only read files that exist, and only read the right length of the file, etc.

My original design was to have files for various aspects of security. Allowed doors; Barred doors; Start and end times; Start date/time; expiry date/time. The idea wast to get the file ID list and decide what to check depending on what files existing.

But even that is a challenge if any files can be variable length. The allowed/barred lists are like this, as are other files. So you have to check the file size for each file first.

Before you know it you are doing dozens of operations to the card, each taking tens of milliseconds.

So I ended up making a system that uses a single "access file", which I expect to exist. I ask the size and then read it. In that file I encode the various things I need like allowed and barred doors, and access times. But it is just those two operations on the card and hence a quick operation.

The end result is an autonomous door control system on an ESP8266 based door controller. It even has expiry update so that a card not used for X days expires. I obviously have blacklisting, but that also zaps the file if a blacklisted card is used.

The next step is some key roll-over logic as well as extra belt-and-braces old card expiry logic.

This is giving a fast enough process for door control. Yay!

2019-07-04

DoH and VPNs and trust

We live in a strange world - where trust is a complex issue.

Once upon a time we would all trust the "authorities", i.e. the police and our own governments, but increasingly we live in a world where a lot of people have good reason (not criminal reasons, even) not to trust people.

The Internet is an especially complicated area where international players of all sorts come in to play, with commercial and political and criminal reasons to cause you concern.

The Internet protocols have been built on a lot of trust, but now we see some new mechanisms to help, two of these being DoH and VPNs.

DoH

DNS over https is one element, with DNSSEC being another. Using DoH means you use an https request to some external server to make your DNS requests.

An https request looks much like any other, and could as easily be your accessing facebook as accessing a DoH server. It is not something that can be snooped on, or selectively blocked.

If you do not trust your ISP to provide "clean" DNS without filtering or snooping, DoH allows you to choose someone else to trust. This is the problem, you have to trust someone, but you have a choice of who you trust.

In addition to DoH, you can also use DNSSEC to validate the accuracy of the responses. Using DoH means someone in the middle cannot snoop, or easily do any selective blocking. But whoever offers the DoH service could.

VPN

A VPN provider works in much the same way - you effectively choose a different "ISP" to provide your Internet access via the ISP you use. Again, choosing who to trust.

I was surprised how popular our own (unencrypted) L2TP service has been at A&A. In time we'll be offering IPsec based virtual ISP services too, I am sure.

Browsers doing DoH

Mozilla are working on using DoH in browsers, which means someone (like an ISP) cannot snoop, or selectively block, DNS requests. It is sad that this is even necessary. Note that AAISP do not filter or block any DNS, and have no plans to.

Oddly this upset ISPA, who has considered making Mozilla their "Internet villain" this year for DoH work.

This seems odd. If an ISP has an order to block some DNS, then they cannot block DoH, but so what? they are complying still!

I was surprised ISPA took that stance, even as a joke award for Internet villain. I can only hope they do not select them as the villain.

So A&A have donated the same amount as an ISPA membership, £2,940, to Mozilla. We have not been ISPA members for some time, but this is the first time I felt ISPA were perhaps taking views I did not really agree with.

We all benefit from the work of Mozilla so much every day, this seemed well worthwhile.

Amazon Yesterday

Amazon do a lot right, but sometimes they do some of the simple things wrong.

Getting it right

They are quite good at showing when they can deliver, clearly, e.g. "Get it by tomorrow if you order within next 5 hours", etc.

They are quite good at then managing to deliver on the agreed date.

Caught out

However, I have been caught out by changing delivery address (e.g. to Wales) and the dates change. This is actually quite good of Amazon, as they have worked out they cannot do the same delivery date, but I have been caught out and not noticed the change.

Things can go wrong

Obviously things can go wrong and they miss the agreed date. Nobody is perfect. They are pretty bad at then sorting it quickly or telling you when it will arrive, but thankfully this is not that often.

Getting the simple things wrong

What does bug me though is some of the silly things they get wrong. They will have "get it by [tomorrow]" on the buy now page but when you order the confirmation may magically change to the day later. If you don't spot it, you can be waiting in all day for a delivery that won't come!

I have also seen, and documented, cases where the delivery date changes from one screen to the next - e.g. the list of orders may say "arriving today" but you click on it for details and it is "by 9pm tomorrow". How the hell do they get something that simple so wrong.

Amazon Yesterday

You will have seen the excellent spoof https://www.youtube.com/watch?v=HA_gwzx39LQ for Amazon Yesterday shipping.

Whilst time travel is not really possible, their app is actually offering this. I just recorded it on my phone - with Amazon actually offering to have the item arrive yesterday.

Now how the hell do they get that so wrong?

2019-07-02

What if I eat it all?

How is it that food labelling does not require a column for "what if I eat it all?".

How hard is that?

This is a typical example of food labelling, and it shows values for 100g and a 23g "serving" on a packet of crisps that is 95g.

I have an A-level in maths but it even takes me a moment to work out that I just ate 50.35g of carbohydrates.

Seriously, how hard is it to actually state what is in the actual packet you have? Surely that should be a basic requirement?

FFS 23g is not even a nice fraction of 95g. It is obviously intended for me, and 3.13 of my close friends to share...

2019-06-14

Soft RS485 UART using interrupts on ESP8266

My non technical readers can skip this :-)

Background

As you will have noticed, I have made a complete access control and alarm system based on a Raspberry Pi but working over RS485 bus connections to standard Honeywell Galaxy parts like RIO, Max Readers, and Keypads.

As you will have noticed, I have now started making WiFi connected parts for the alarm system based on ESP8266 parts (ESP-01, ESP-12F, ESP-12S). These include a small ESP-01 based board with RS485 driver that fits inside a Galaxy keypad talking RS485 to the keypad.

As you may have noticed, I have NFC readers sorted and working with secure NXP MIFARE DESFire EV1 cards. These are way more secure than poxy proxy cards used by Max Readers.

Challenge

The concern some people have raised is the reliance on WiFi. It will do the job, but it is not that hard to disrupt externally. Now, much like using a screwdriver on a Max Reader, it will set off the alarm to disrupt it, but it is perhaps a tad easier. One has to consider if it is easier than simply forcing the door or breaking a window, which is always the benchmark for any security :-) But it is a concern.

So can I make the door controller, which does the secure cards, look like a Max Reader and do RS485 to a conventional Galaxy system? I think I can...

Approach

Initially, for the keypad system, I used the built in serial drivers on the ESP8266. These are good, interrupt driven, well supported, except...

The code on the ESP8266 using Arduino is the NONOS SDK, and means one thread for all. It can be delayed a bit for all sorts of reasons, even many milliseconds. For serial that is all handled behind the scenes on interrupts this is no problem, until is it RS485!

The problem is that for RS485 you have to enable and disable the line driver around your transmission. Initially I simply set a time to disable after poking the message in the Tx queue. But that could be delayed and mean after transmitting we held the line driven for too long.

The solution was to wait in the main loop task until sent, and then disable the driver. This is not nice, as any other operations of the device are delayed, but for the keypad driver application there is nothing else. So it works well.

This will not work for the card reader as it has to, well, talk to the card reader, and inputs, and outputs for the door control, and so on. Also, if I am a slave device (unlike the keypad where I am master) I am expected to answer polls promptly and not hang around for 10ms randomly.

My plan - interrupts! Make a soft serial RS485 UART on an ESP8266 to do the grunt in the background.

UARTs

A UART (Universal Asynchronous Receive/Transmit) is a device to send and receive old fashioned serial byte / character data. The concept dates back to what? the 50s or 60s. It is simple...



Each byte, or character, to be sent is coded as a series of 1 or 0 on a bus. What 1 and 0 mean in terms of current or voltage depends on the flavour of system (RS232, RS485, RS422, etc). Each bit time (1/Baud rate) the line has a 0 or a 1. The line is idle (1) and then the byte/character starts with a start bit (0), bits in the byte (low bit first), then any parity bit, and then stop bit (or bits) which are 1.

Obviously both ends have to agree the rate (Baud rate) for bits, and byte size, and parity present and if even or odd, and even number of stop bits.

The Galaxy uses 9600 Baud (so a bit every 104⅙µs), 8 bit data, no parity and one stop bit. Simples!

Sending is easy, you need some sort of timer, and you set 1 or 0 on the output to do the start bit, data bits, and stop bit. The Arduino ESP8266 SoftwareSerial sends in real time from the command using careful timers, it seems. I plan to use interrupts.

Receiving is a tad harder. You have to find the start bit for each byte, and then sample each bit, ideally in the middle. But once you have the bits you shift in the data bits and check the stop bit is 1 (else an error). Not that hard?

Edge interrupt on rx?

Interestingly the SoftwareSerial code simply interrupts on each edge and looks at time from last edge to work out the received data (I think). That is one approach but I decided to clock in bits on interrupt - I just need to sample mid bit somehow.

My initial approach was to set an interrupt on the falling edge of the data line to set the recurring timer half a bit later. However, this has all sorts of challenges, no least of which is race conditions on a regular per-bit interrupt and the edge interrupt. I did work, but not 100%. I am sure I could have solved the race conditions, but actually there is a good reason to stick to simple regular interrupts.

So the approach I took was just a regular interrupt, and this was 3 times per bit. This means ⅔ of the interrupts just count a sub bit and return in most cases, so very light weight in terms of time. But what it does mean is that I could in fact have multiple instances of RS485 busses running on this one interrupt if all the same Baud rate. I have not yet coded that though. It is, however, nice and simple and working for one instance well.

Why 3?

The 3 times per bit is simple. For Tx it makes no odds, skip 2 out of 3 interrupts and set the line to 1 or 0 for the next bit to send. For Rx, it is the same for sampling, except when waiting for start bit. If waiting for the start bit you check for the line going 0 every interrupt, and if it does set the sub bit counter to sample one interrupt later and then every 3 interrupts.

This is not as clean as an edge interrupt adjusting the timing to the microsecond, but it does mean that we can have sampling of the centre of the bit, ±⅙ of a bit which is more than adequate. Doing 2 times bit rate is no good as you could be sampling in the middle or right at the end. Hardware UARTs usually do 16x bit rates, but 3x is fine in my opinion.

This is how I have coded soft UARTs for over 20 years dating back to PIC16C84s, realy!

Messages

The way the RS485 bus works on Galaxy systems is that messages are sent as a sequence of bytes with no gaps, and then a gap - no length indicator in the message (but there is a checksum). So I coded the system to handle this, and even, if we are a slave, auto respond with current status response.

This allows the application to work on a message level, and if it is slow, we answer polls promptly anyway with current status. The driver also does the checksum.

Interrupts on ESP8266

Now the challenge is making it work on an ESP8266. They are rather special. They are a weird mix of old and new. The ESP-12S has 4M of flash! As an embedded controller that is huge! But it has around 80k of RAM (yes, k) and I had a BBC micro with more in the 80s (32k main and 64k second processor). The fact it does TCP at all is mental, but it does, and it does WiFi too.

The usual pressures on coding are complex - embedded systems have pressures on code space, RAM, execution time, power, all sorts.

In this case interrupts are special. The processor has a cache for the code, but this cannot be used by an ISR (presumably some interrupt process handles cache fails). This means the ISR has to be in RAM.

The trick is marking the code you use for the ISR with ICACHE_RAM_ATTR attribute. The RAM used is small, so not only does the ISR have to be fast it has to be small in code space too.

Thankfully a UART is simple code in terms of space and time. Small TARDIS code, even?

This worked, well, sort of, it worked mostly if started after about a second after boot. It turns out that using digitalRead, digitalWrite and pinMode is bad as these Arduino functions are not marked ICACHE_RAM_ATTR. Instead you need to find the underlying GPIO_REG_WRITE and GPIO_REG_READ functions. That then works reliably. The GPIO access is also faster, which is good, but very processor specific.

The result is working RS485, on interrupt, on an ESP8266!


I even added a clock track (blue on that trace) for debug - toggling an output for tx and rx clocking per bit.

It works!

This was deployed as new code to talk to the keypad, but means I can also act as a slave and pretend to be a Max Reader and provide a fob ID derived from the NFC card ID when checked using AES handshake.

It is also quite important as I am using the hardware serial at 115200 Baud to talk to the PN532 NFC card reader!

P.S. I have tested now as both master talking to Keypad, and as slave, pretending to be a Max Reader.

2019-06-09

Choosing interface for PN532 (I²C, SPI or HSU) to ESP8266

I have been working with ESP8266 controllers and NFC readers. My preferred reader is the PN532 based Elechouse reader. As I have noted before, buy from Elechouse direct as there are a lot of bad copies!

However, when connecting to the processor I have a choice of interface. It is one of the rather nice features of the PN532. There are three interface types I²C, SPI or HSU.

So how to choose...

I²C

I²C stands for Inter-Integrated Circuit, so the acronym is IIC, but someone thought it would be fun to treat letters in an acronym in the same way as algebraic variables and write II as I². Arrg!

It is a simple bus for connecting devices and uses a clock and data line. Wikipedia has details (here). Both clock and data lines have a passive pull-up resistor and are actively driven low by master or slave. The protocol is well defined, and works well. It allows multiple devices on one bus and is very popular for a lot of peripherals.

Looking at the Elechouse web site I saw this: "On-board level shifter, Standard 5V TTL for I2C and UART, 3.3V TTL SPI" which rather concerned me. The ESP8266 is all 3.3V, and I did not want to have to mess about. This immediately put me off using I²C. However, looking at the schematic, the level shifter is completely optional, in that the same two pins are used for SPI, I²C and HSU and connections have these at the 3.3V level direct from the PN532 (via 100Ω) as well as via the level shifter, so it can be used for HSU and I²C if you want.

✓ Only two wires, simplifies wiring and frees up GPIO pins
✘ ESP8266 does not have hardware I²C support, so uses more processor and is slower
✓ ESP8266 has software I²C which means almost any GPIO pins can be used for clock and data, making PCB layout much simpler.
✘ Data sheet suggested level shifted to 5V (which turns out to be optional)
✘ Both clock and data use passive pull up resistors, meaning they can be subject to more interference on long lines

SPI

SPI stands for Serial Peripheral Interface. Wikipedia has details (here). It is more complex than I²C in the way it works - using 4 wires not 2, and allowing different configurations. It is also quite fast, allowing several MHz operation.

The ESP8266 has hardware support for SPI, which is good, but the downside of that is that specific GPIO pins are used for 3 of the 4 signals, so PCB layout is more complex.

✘ Four wires, so more to connect from controller to reader and uses more GPIO pins
✓ ESP8266 has hardware support, so allows faster working
✘ ESP8266 has hardware support, so the pin assignments are fixed making PCB layout more complex
✓ Lines are driven (from master or from slave) rather than using passive pull ups
✓ 3.3V direct connection to ESP8266

HSU

HSU stands for High Speed UART. This is simple old fashioned serial communications, on a Tx and Rx line, and defaults to 115,200 Baud.

The ESP8266 has hardware support for UART, and will work at these speeds. The ESP8266 library has good, interrupt based, serial handling. Unlike SPI there is actually a choice of pins that can be used, but this is still a tad limited. Of course the ESP8266 uses the UART for other things like programming and debug, but there is an additional Tx available for debug (Serial1).

✓ Only two wires, simplifies wiring and frees up GPIO pins
✓ ESP8266 has good hardware and library support for serial UART
✘ Pin usage is a fixed, but a choice of two sets
✘ Data sheet suggested level shifted to 5V (which turns out to be optional)
✘ Serial on ESP8266 is used for programming and debug
✓ Tx and Rx are driven (one each way) rather than passive pull up resistors
✓ The same connector can use used for programming and connection to PN532, reducing number of headers needed on the board

What did I do?

I was initially put off by the level convertor comment, leaving SPI as the only option. The first project I made using an NFC reader was connected to weighing scales, and used the serial on the ESP8266 to talk to the scales, so that rules out HSU. So SPI it is.

There are 4 connections needed (so 6 wires including power and ground). This means you can use 5 pins in a row on the ESP8266 (VCC, D13, D12, D14, D16) and GND to make a 6 way connector. Sadly the pins are not quite the same order as on the Elechouse PN532 board. This just means the connectors need wiring carefully.

This works well. I then used the same design for the door controller.

Hindsight

Firstly, the fact that the level convertors are optional means I²C and HSU are an option. I could have, for example, used I²C on a smaller ESP-01 for the scales rather than using an ESP-12F and SPI. However, the use of passive pull-ups is still a concern if the reader is on any length of wire from the controller as may be the case on a door controller. I could have used HSU, as the ESP8266 libraries have a soft serial mode, which can work at the scales 9,600 Baud rate, so could have used Tx/Rx for PN532 and one of the other pins to talk to the scales serial interface, and used an ESP-01.

I have since tried HSU, which just works, and is only 4 wires to the reader (power, ground, Tx and Rx). I was concerned that it would be slower (115,200 Baud rather than 2MHz) but actually it is slightly quicker (28ms rather than 30ms for InListPassiveTarget on same card), and a lot of the time is the PN532 operation and the NFC card itself. The Baud rate can also be configured!

It also turns out that had I used the more usual GPIO15 as the SPI SS pin not GPIO16 I would have actually wired the PN532 such that it could be used for HSU or SPI, as the PN532 uses MOSI for Tx and SS for Rx and the ESP8266 alternative Tx/Rx for UART0 are on those pins!

In practice, what I have done is make a 4 pin header with GND, 3V3, Tx and Rx. This is the same order as the 4 pins on the Elechouse board, but is also a header that can be used for programming the ESP8266.

I am slightly kicking myself for not trying HSU and I²C to start with, but to be honest I was just pleased to get things working. Freeing up two extra GPIO pins is nice, as is having fewer wires to reader, and pins in the same order both ends of the cable. The libraries do, however, make it very easy to change interface. Just remember to set the switches on the Elechouse board to match what you are using.

2019-06-02

I knocked up a library

So I have made a library from scratch today for NXP MIFARE DESFire cards using AES.

Been a long day.

But here it is: https://github.com/revk/DESFireAES

Have fun.

P.S. Updated and tested and, and made an ESP8266 version....

2019-06-01

NXP MIFARE DESFire EV1 for access control

Earlier this week I set myself the task of understanding the NXP MIFARE DESFire cards.

This was more of a challenge than you may expect because the manufacturers don't publish the protocol (except under NDA). Why?!?

  • It does not stop people getting the data, as is apparent!
  • It means there is also misinformation out there which cannot easily be corrected by reference to the official manual.
  • It puts people off using the product, which cannot be good for sales.
  • It means subtle details in the official manual will not be known, which could lead to failures in edge cases, making the product look bad.
  • It does not help security - as any secure system must not rely on secret specifications.

I have written up what I have found out so far: PDF on github

So, why not just use one of the few (unofficial) libraries out there?
  • I probably will for the host application on linux anyway.
  • Coding it myself helped me learn more about the cards and how they work and what they can do.
  • It was fun!
  • The code on my ESP8266 devices to read cards does not need what is in a library - half of it is for handling legacy formats, and DES and 2KDES and 3KDES whereas I need a few specific operations using AES only.

The result - I have managed quite a lot, and made the door control system use cards which cannot simply be copied (unlike the 125kHz proxy tags, which can be, very very easily).

I have even managed to convert from DES to AES for the master key, and crucially I have documented detailed examples of this, and the CMAC logic, and so on, in the manual I have written. A lot of what I have learned has come from other sources as listed at the top, but I have gone through testing things directly and coding stuff myself to confirm things as well.

I have a cyclic record file on the card logging usage (door ID, and timestamp). I still need to add some handling for expiry, and time period controls on the card, maybe.

I am a tad concerned over response times. I am selecting the app, authenticating, getting file list, writing a usage log record and updating a counter, and you notice the slight delay using the card, which is a concern. This may be something that can be improved, not sure. I have not worked out how much is the card, and how much is the ESP8266 doing AES!

But it is working! I am even checking the CMAC on responses to confirm no man-in-the-middle upgrade attack (making one card seem like another once it has done AES handshake).

Happy to take comments and update any mistakes in the manual, but I hope it is useful to people.

P.S. speed improvements by tweaking what operations I do and in what order. Now works very slickly.

P.P.S. I made my own linux/C library (here) and light weight ESP8266 functions (here).

2019-05-24

Tipping the scales

I am getting really good at upgrading the scales to do WiFi now, and read cards. The card of choice is a Monzo card which works perfectly well as an ID for weighing yourself. Notably Amex have an ID of 00000000 which is useless.

Really neat PCB, locking molex connectors, cables, the lot. It is working well. Marsden kindly sent the service manual so I can set to "kg" and "st/lb" modes for those that like old-school weights.

The system has a QR code on the scales, which allows you to register any card against your email address. You then weigh yourself and tap the card on the scales to record the weight.

When you tap the card it beeps to confirm it is working, and keep beeping until a stable weight is recorded over WiFi to the server. Simple UI if ever I made one!

Sadly, I have fried a couple of boards whilst doing these upgrades, but Marsden do spare parts (not cheap) and whilst I have a few clues what I may have done, I am not 100% sure of how I killed them. I think, with care, and anti-static, and being neat (avoiding any shorts, or diodes backwards) I can do this reliably now. Obviously very unofficial so I cannot expect any help from Marsden. They do make very nice scales though.

The next challenge is to make the web site that holds the data have graphs and sharing options and so on. At present it is just a list of weights. But even that is working for relatives using this solution.

This is turning in to the new fat-shamers club or something, not sure.

But what was the first thing on the web site? after registering the domain? well, it was getting an excellent lawyer to put together the privacy policy for us and ensuring we are GDPR compliant.

You have to love any lawyer that covers standing on scales with your pet cat as part of the privacy policy. Thank you Neil.