ESP-01, ESP-12F, ESP-12S, ESP32

As I mentioned, I am trying to move my IoT development stuff to ESP32, and notably move from Arduino IDE to the native Espressif ESP32-IDF using mostly C.

This poses a slight problem. The development environments are sufficiently different that it is something of a pain in the arse to maintain both. Probably just about possible, but not simple. So I'd like to basically abandon the ESP8266 / Arduino stuff (I'll leave on GitHub) and start working only on the ESP32 based stuff.

This poses a slight snag as one of the use cases for this IoT stuff is a very small device placed inside a Honywell Galaxy alarm system keypad.

This is based on an ESP-01 module which is ESP8266 based. It is not ideal as not CE marked, no RF can, very limited IO pins, but it is small. It comes with a 2x4 0.1" pitch header and is just the right size to fit a Pololu 3V3 regulator between it and the PCB, which is rather neat.

This use case has the ESP-01, a 3V3 regulator, an RS485 SO8 driver, and a small 2x2 milli-grid header for programming and debugging.

The whole thing fits in the small space inside the keypad, as per the picture on the right.

As a result I assumed I'd be stuck with, at the very least, my keypad code on the ESP8266 / Arduino based code.

The other IoT devices are generally based on either ESP-12F, or preferably ESP-12S modules. These are only slightly narrower than the ESP32-WROOM-32 modules I am now using, and so easy enough to make alternative designs using the ESP32.

The ESP32, however, whilst larger, has a lot of IO pins. It is possible to track these pins so that you have quite a few usable IO pins, and GND and 3V3, all on one end of the chip if you try :-

I have actually tracked Tx/Rx programming pins, and GND and 3V3 "through" some unused GPIO pins, which allows its all to come out of the bottom end and keep the whole thing quite narrow. Obviously this is fine as long as I don't make those pins an output :-)

Once again, the new milling machine is doing a great job, and I was able to solder it.

And, to my delight, it fits, and the keyboard PCB fits over it with no problems!

So, yes, I can actually move away from ESP8266 now. I have to migrate and re-work the various tools and modules I have made, but once done, I can continue development on ESP32 only. Yay!

P.S. Always tweaking :-)

Moving to ESP32

Someone already told me that once I move to ESP32 I won't want to go back to ESP8266.

Well, apart from the fact the ESP8266 is available in a tiny ESP-01 package, I think they may be right. The ESP32 in a WROOM-32 package is slightly wider than the ESP12 package, but because the GPIO pins can be mapped any way you want, pretty much, the tracking takes less space. So overall I was able to make an equivalent board in less space using ESP32.

The big challenge for me with ESP32 was the concern over the fine pitch - 0.9mm contacts at 1.27mm spacing. I was not sure I could either mill a PCB or solder a board at that pitch reliably.

I was wrong, I can!

New door controller using ESP32

Fixing the X axis backlash on my mill was a big help as well. So yes, I can use the ESP32 modules.

The other concern was the price, but it seems that getting a proper ESP32 is as cheap, if not cheaper, than an ESP12S. Yes, some cheap ESP12S can be got, without CE marks, and missing FCC ID, and not quite working well on WiFi. But £2.91 for ESP32-WROOM-32 from a reputable supplier is pretty good.

There are advantages to the ESP32, obviously, including much more RAM, faster, dual processor, more GPIO, more UARTs, and so on. It is a good comprehensive IoT device.

The other big change I want to make is moving away from Arduino IDE. Yes, Arduino makes things easy, but it turns out the Espressif IDF for ESP32 is pretty good. It has the key things like https, MQTT, OTA, and all sorts as standard.

So my plan is to make an ESP32 IDF based version of my door controller, including PN532, VL53L0X, etc. This should not be too hard, and a lot of my existing code can easily be ported to a simple C code environment.

I am, however, impressed with the documentation - not only of the APIs in the IDF, but also the technical reference manual. It actually details all of the registers clearly.

The other impressive thing is Espressif seem to to be trying to encourage some best practice in IoT. They have secure boot and signed images as a standard feature. Even https libraries, as standard. It means that making a reasonably secure IoT device is easy. Well done.

I managed to get the build environment working on Mac and linux with no problems, following the clear instructions. I built the "hello world" example, and flashed it with no problem.

I am off on a 4 hour flight shortly, so I have the 669 page PDF of the technical reference manual on an iPad for some light reading on the flight.

Of course, once I have this off the ground, I can look at Bluetooth, and CAN bus, and all sorts of new fun things.

New milling machine

I had a CNC 3018 PRO machine from Amazon.

• It needed assembling, but I've done that now. It is something to bear in mind if buying one though.
• It is a bit of a "home made" style, with the PCB bolted to the frame, etc.
• The one I got had a bent screw for X axis, which limited the usable space a bit.
• It has a working area 30x18cm.
• I was able connect a Z-axis probe (well, contact to PCB copper clad board).
• It works well, and I used bCNC on my Mac to work it.
• It comes with a controller box, which is not needed really!
• It is surprisingly cheap.

So, especially given the bent screw, I decided to get a replacement, and went for a CNC 3020 from Amazon.

• This is a much more "industrial" unit - proper drag chains and cables. Looks good but costs a bit more.
• Work area (you guessed it) 30x20cm
• Comes pre-built
• Has a boxed controller and power supply, with nice connectors to the machine.

However, it has a challenge or two - the main one is that it works with some very specific software (a pirate copy of which is apparently supplied with it). It has parallel (!) and USB, but the USB only works with that software and only on a windows machine - it does not even appear on a Mac even in a debug log! The software did not look too good anyway! This is a bit of a bugger.

The upgrade

My solution was to upgrade the controller to a TingG controller. This adds a bit to the cost (especially with the customs/VAT and admin charge from US). But was not actually at all hard to do.

The old controller is bolted to a heat sink, and can simply be unscrewed.

The new controller is slightly smaller. I needed to drill/tap an M3 hole in the heat sink for one of the screws so that I could screw it to the heat sink. I also cut an extra hole in the case for the USB connector.

Also, I decided to use the heat sink, which meant something between the new controller board and the heatsink. I got some small copper block / heat sinks and thermal tape - and I used them between the back of the new board and the existing heatsink.

The TinyG is designed to work without the need for a heatsink, but this can not really do any harm.

The connectors on the old board were not the same, though the new board has pads which means I could have changed them. I decided instead to use the screw terminals on the new board, fitting bootlace crimps on the wires. I had to swap the cables to make them fit, and use the motors 2, 3 and 4 as closest to edge of the case. But they all fit!

I also connected the emergency stop to the reset pins. I have not worked out how to connect the spindle drive to the new board yet - that is a challenge for another time. The case has a button and a dial for that which works for now.

I also got a 6 way (plus ground) chassis DIN connector and wired up all 6 end stops (X/Y/Z Min/Max).

This meant various micro switches, and super glue on the actual machine.

And also, the Z-Min I wired to croc clips to allow me to use contact to PCB for homing Z axis.

The TinyG has good documentation, covering setting the current, etc. I was able to set the axis to the right motors, and set the polarity and travel per rotation, etc.

However, the bCNC code I was using did not like the TinyG, and I ended up installing CNC.js which works nicely on my Mac. It does lack the multiple point Z axis levelling of bCNC sadly, but works.

I did have to configure the homing for Z axis a tad as it is designed for a switch from which you back off. For PCB contact you get the exact Z home, and do not need offsets. But this is all well documented.

Slack (backlash)

However, even though it is a really good solid construction and seems to have no play, I found there was an issue with the X axis. It seems to have around 0.1mm slack on it. This means if you move right to a point it is actually around 0.05mm short of it, same if moving left (i.e. is then 0.05mm right of it). Make a row of left and right moves and lines and you see the problem clearly. This test shows it well - centre is left and right moves before each line, but left is all moves from the right (apart from first, bottom left) and right is all moves from the left. Repeatable slack!

Oddly the Y axis is absolutely spot on!

This is not really enough of a problem to cause issues making PCBs, but is annoying so I wanted to fix it.

I checked for anything loose, and also checked the TinyG for any options, but to no avail.

My solution (now updated to the eps2gcode tools) is software to compensate for slack. This seems to work.

Overall I am quite happy with the result.

P.S. Tightening this nut a bit fixed it.



P.P.S. After reading the TinyG docs I am actually running at 4 micro steps not 8, so 1/200th mm spacing, which I think will be more than adequate for anything I am milling. I suspect 1/100th mm would be fine but actually that is a lot more noisy. Now that the backlash is sorted maybe I'll go back to 8.

Body cams are weird #GDPR

Body cams are weird - you walk in to a shop with one on, especially with a big WARNING: VIDEO/AUDIO RECORDING sign on it, and you get strange looks. But at the same time, half a dozen CCTV pointing at you, is "normal".

I have yet to be challenged on this, and I am not sure I really want the altercation, but I imagine if asked to "turn that off" it would be valid to say "I will, if you turn that off [point to CCTV]".

Let me stress, I am not a lawyer or GDPR expert, but I'll try and make sure this post is technically accurate and update any errors pointed out to me. And thank you Neil for the links, case references, and definitions and correction of my many typos.

Domestic purposes

One of the things that is outside the scope of GDPR is processing of personal data by a natural person in the course of a “purely personal or household activity”, which is commonly called processing “for purely domestic purposes" This is a tad complicated in interpretation though, and there is some confusion on this.

There has been a judgement on fixed position CCTV cameras recording in a continuous loop in people's homes (domiciles!) that says that it is only domestic purposes if there is no view of any area outside the property boundary. i.e. if the CTTV covers the road, or neighbour's property, it is not domestic purposes and hence you need a lawful basis, a privacy notice, to comply with data subject rights, and everything else that goes with GDPR compliance.

Personally (and I think I am not alone on this), I think this ruling is wrong: The cost seems to have confused the setting of the processing with the purpose of the processing. The classic example of domestic purposes, an "address book", will have names and addresses and phone numbers of people not within the boundary of my property. So clearly it is not the data subjects that define "domestic purposes" but the "purpose" of the processing (the clue is in the words used!). But as we have a ruling, that becomes risky to try and argue that CCTV covering the road is domestic purposes.

Indeed, for security of my home (domestic purposes) I want recordings of people "casing the place" (on the road or opposite my home) as that may have much better, daylight, views of faces to use when later the same vehicle is caught on camera relating to the place being burgled.

Dash cams, and helmet cams, were at that point considered "OK" though (or, at least, there has been no case law yet involving them), which leads to a whole series of scenarios where one cycles home and puts the helmet cam on the wall of the house, and then connects to power, and so on - at what point does it change from "OK" as a helmet cam covering the road, to not "OK" as a CCTV covering the road?

So, to be clear, if you have seen my body cam and found this post because the URL on the label, my purpose in recording is domestic purposes - I record my cycle rides mainly. Just like a dash cam in a car.

Continuous recording

There is some draft guidance on the matter of dash cams which actually suggests that dash cams must not  be continuously recording!

Yes, you read that right - it seems to suggest that you have to start recording just before the "accident" you want to record. I know, that is utter madness.

Of course there are plenty of people who will jump in and point out the technical solution is a pre-recording buffer. That is only saved if you "press the button" (after the accident), or the camera detects a collision, even, which some can!

But that *IS* processing personal information, even if it just goes in to RAM and is replaced after 5 minutes. An example of why that is processing is simple - imagine a dash cam where the camera owner is the cause of an accident. They have a choice about processing of that personal data in the RAM of their camera - they can choose to "press the button" and save the (incriminating) evidence, or not. The data subject can, if they are quick, ask for a copy of that data, at which point they have to "press the button" as otherwise they are deleting personal data after being given (what used to be called) a subject access request. Clearly the data in the pre-record buffer is personal data, and there are data processing decisions to be made in relation to that data.

We can only hope that such a nonsense paragraph is removed from this draft guidance (or, even better, replaced with something more akin to common sense, that someone trying to protect themselves from an accident, or put themselves in a good position if they do suffer an accident, is “domestic purposes”, even if it films someone else in a public setting).

Transition from domestic purposes to commercial

There is then one area about which I am really unsure. If I have personal data, recorded purely for domestic purposes, e.g. my address book, but I then tweet that!

In another recent case, the Court of Justice of the European Union has held that "since [someone] published the video in question on a video website on which users can send, watch and share videos, without restricting access to that video, thereby permitting access to personal data to an indefinite number of people, the processing of personal data at issue in the main proceedings does not come within the context of purely personal or household activities”.

Applying this to a dashcam or helmet cam, even if my filming would be considered to be “domestic purposes”, posting the resulting video on Twitter or Facebook (e.g. to highlight bad driving, or just as part of a journal of my day), unless I lock down access to the video, it looks like I would be outside the scope of domestic purposes.

Given how many people live out their lives online, engaging in the type of chat that one might previously have had in a pub or coffee shop, this feels like it could entail the imposition of GDPR — which, let’s face it, is not something normal people should have to understand — on the general public, for very common activities. It seems to me like rather too much of an interference with people's private lives.

Somewhere in between

Sharing a family video (e.g. my grandson riding a bike for the first time) with family is clearly domestic purpose.

I assume doing so over iMessage is still so, as I do not hand the video to Apple (end to end encryption).

What of posting to a small (family) group on Facebook? My "purpose" is clearly domestic, but Facebook have the image and they do things with images that are not domestic purposes.

What does that mean for GDPR I wonder? I assume that this means that Facebook needs to comply with GDPR, but I do not (since I am not sharing with an indefinite number of people).

🚲 Paint only cycle lines

I shared a slightly amusing video of an "incident" on my way back from costa (here) and a friend of mine commented that there was a cycle lane next to the road on which I was cycling, and asked why I was not using it.

I think he genuinely does not realise why I would not use that cycle lane. So I wanted to try and explain a little.

Firstly, and importantly, there are quite a lot of original cycle lanes in Bracknell. The town planners included these, and they go places that are different to the roads (see the tree lines on the image on the right, for example). A lot of them are a proper road surface, wide enough for bikes to pass each way, with proper curbs and separate pedestrian paths along side. They link sensibly to roads, and notable go easily under the major road junctions. These I use, when they go where I want to go. Sadly, even though they have clearly distinct pedestrian paths along side you still get pedestrians meandering in the cycle path, which is a nuisance, and seems to be for no obvious reason or advantage to the pedestrian.

However, like a lot of places, there are a lot of cycle lanes in "paint only". I.e. they are former foot paths that now have paint and signs. Some are split with cycling one side and pedestrian the other side, but quite often it is just shared use on what used to be the foot path.



So, I want to try and explain to my friend, who drives a Tesla. It is a nice car, and if I get a car it will probably be a Tesla. They are quiet, and quite impressive.

Dedicated Tesla lanes

So imagine the council were going to make dedicated lanes for Teslas. Now, ideally these would be specially suited to such a car, smooth road surface, good white lines so the auto-pilot can see them clearly, fences to stop pedestrians accessing the lane, maybe even a special higher speed limit. If you have a special Tesla lane that gave you some advantage over the road you would use it.

Except...

• The special lane is actually a bit narrow
• It has a poor road surface
• Access to the lane is not seamless - it has a bit of a step up, or some cobbles, or a bollard you have to go around, or even a chicane / kink so you have to manoeuvre to get on to and off the lane.
• The lane has numerous obstacles (signs, bins, bus stops), with just enough space to get around them, but you have to watch out for them and manoeuvre around them.
• Some of the signs are two pole sign where the sign you have to go between, but the sign is low enough you are not sure you can drive under, or some Telsas won't fit under.
• The lane is actually two way, but not mostly wide enough to pass other vehicles, so you have to co-ordinate with the other driver to pass where you can, slowing down, moving over, etc.
• There are pedestrians allowed on the lane, and they don't hear you coming (Teslas are quiet) so you have to beep the horn, and they get all indignant. Even when there is space to pass them, you have to slow down in case they step in front of you or wave their arms about for no reason.
• The lane does not actually go very far, it stops at the next side road, and you have to get off the lane and cross a side road (giving way). You may even have fewer rights than a pedestrian (who has priority over cars driving in to the side road, not that many drivers realise that). Then you can get on to a new special Tesla lane the other side of the side road.
• Actually, often there is no follow on lane, you have to re-join the main road, or perhaps there is another lane the other side of the main road.

You could suffer all of this, or you could just use the main road, as you are entitled to anyway, and which goes the same place as the special lane (without all the hassle and giving way at each side road, etc).

As a Tesla driver, would you use these special lanes?

Environmental sensors

For my next little project I am thinking of making a good little environmental sensor box, mainly for CO₂ and temperature as they are the two things I have some control over.

ESP8266 based, why not. But I have been looking at the sensors you can get, and bought a few.

Are all CO₂ sensors crap?

Well, I tried a few, including this CJMCU module (CCS811 based) and this AppliedSensor module.

Sorry to say, but both are crap! And the detail is in the small print - they estimate CO₂ from other sensors, and are very easily confused. I have some isopropyl alcohol and you use that anywhere near them and they quickly say 18000ppm CO₂. Even without such factors they swing wildly all over the place.

It seems you need an NDIR based sensor, and I found this one which seems to actually work! This SCD30 sensor seems to be the business, with CO₂, temperature and humidity (floating point!) via I²C.


Next step - sort a suitable display and box.

I can then use the temperature to control the air-conditioning, and the CO₂ to control extractor fan speed.

Door control

My door control and alarm system has made some progress.

The main change is making access control a bit more distinct from alarm system. The doors at the office were "falling off" the WiFi occasionally, even if only for a few seconds at a time, a few times a day.

I suspect that can be fixed, and it makes a huge difference what channels we use. The AP report something like 60 interfering APs.

So the change is to make the doors work autonomously which means the door controller has to be able to authenticate the fob/card used but also know enough to tell if it is allowed access.

I have actually created six levels of integration and autonomy for door control linking to alarm system.

This means things like times of day, depending on day of week, which doors they can access, and expiry dates and the like (i.e. some way to be sure a lost card is now invalid).

However, the DESFire cards are a challenge at best. I cannot have any operation that produces an error as that kills the authentication. So I have to only read files that exist, and only read the right length of the file, etc.

My original design was to have files for various aspects of security. Allowed doors; Barred doors; Start and end times; Start date/time; expiry date/time. The idea wast to get the file ID list and decide what to check depending on what files existing.

But even that is a challenge if any files can be variable length. The allowed/barred lists are like this, as are other files. So you have to check the file size for each file first.

Before you know it you are doing dozens of operations to the card, each taking tens of milliseconds.

So I ended up making a system that uses a single "access file", which I expect to exist. I ask the size and then read it. In that file I encode the various things I need like allowed and barred doors, and access times. But it is just those two operations on the card and hence a quick operation.

The end result is an autonomous door control system on an ESP8266 based door controller. It even has expiry update so that a card not used for X days expires. I obviously have blacklisting, but that also zaps the file if a blacklisted card is used.

The next step is some key roll-over logic as well as extra belt-and-braces old card expiry logic.

This is giving a fast enough process for door control. Yay!

DoH and VPNs and trust

We live in a strange world - where trust is a complex issue.

Once upon a time we would all trust the "authorities", i.e. the police and our own governments, but increasingly we live in a world where a lot of people have good reason (not criminal reasons, even) not to trust people.

The Internet is an especially complicated area where international players of all sorts come in to play, with commercial and political and criminal reasons to cause you concern.

The Internet protocols have been built on a lot of trust, but now we see some new mechanisms to help, two of these being DoH and VPNs.

DoH

DNS over https is one element, with DNSSEC being another. Using DoH means you use an https request to some external server to make your DNS requests.

An https request looks much like any other, and could as easily be your accessing facebook as accessing a DoH server. It is not something that can be snooped on, or selectively blocked.

If you do not trust your ISP to provide "clean" DNS without filtering or snooping, DoH allows you to choose someone else to trust. This is the problem, you have to trust someone, but you have a choice of who you trust.

In addition to DoH, you can also use DNSSEC to validate the accuracy of the responses. Using DoH means someone in the middle cannot snoop, or easily do any selective blocking. But whoever offers the DoH service could.

VPN

A VPN provider works in much the same way - you effectively choose a different "ISP" to provide your Internet access via the ISP you use. Again, choosing who to trust.

I was surprised how popular our own (unencrypted) L2TP service has been at A&A. In time we'll be offering IPsec based virtual ISP services too, I am sure.

Browsers doing DoH

Mozilla are working on using DoH in browsers, which means someone (like an ISP) cannot snoop, or selectively block, DNS requests. It is sad that this is even necessary. Note that AAISP do not filter or block any DNS, and have no plans to.

Oddly this upset ISPA, who has considered making Mozilla their "Internet villain" this year for DoH work.

This seems odd. If an ISP has an order to block some DNS, then they cannot block DoH, but so what? they are complying still!

I was surprised ISPA took that stance, even as a joke award for Internet villain. I can only hope they do not select them as the villain.

So A&A have donated the same amount as an ISPA membership, £2,940, to Mozilla. We have not been ISPA members for some time, but this is the first time I felt ISPA were perhaps taking views I did not really agree with.

We all benefit from the work of Mozilla so much every day, this seemed well worthwhile.

Amazon Yesterday

Amazon do a lot right, but sometimes they do some of the simple things wrong.

Getting it right

They are quite good at showing when they can deliver, clearly, e.g. "Get it by tomorrow if you order within next 5 hours", etc.

They are quite good at then managing to deliver on the agreed date.

Caught out

However, I have been caught out by changing delivery address (e.g. to Wales) and the dates change. This is actually quite good of Amazon, as they have worked out they cannot do the same delivery date, but I have been caught out and not noticed the change.

Things can go wrong

Obviously things can go wrong and they miss the agreed date. Nobody is perfect. They are pretty bad at then sorting it quickly or telling you when it will arrive, but thankfully this is not that often.

Getting the simple things wrong

What does bug me though is some of the silly things they get wrong. They will have "get it by [tomorrow]" on the buy now page but when you order the confirmation may magically change to the day later. If you don't spot it, you can be waiting in all day for a delivery that won't come!

I have also seen, and documented, cases where the delivery date changes from one screen to the next - e.g. the list of orders may say "arriving today" but you click on it for details and it is "by 9pm tomorrow". How the hell do they get something that simple so wrong.

Amazon Yesterday

You will have seen the excellent spoof https://www.youtube.com/watch?v=HA_gwzx39LQ for Amazon Yesterday shipping.

Whilst time travel is not really possible, their app is actually offering this. I just recorded it on my phone - with Amazon actually offering to have the item arrive yesterday.

Now how the hell do they get that so wrong?

What if I eat it all?

How is it that food labelling does not require a column for "what if I eat it all?".

How hard is that?

This is a typical example of food labelling, and it shows values for 100g and a 23g "serving" on a packet of crisps that is 95g.

I have an A-level in maths but it even takes me a moment to work out that I just ate 50.35g of carbohydrates.

Seriously, how hard is it to actually state what is in the actual packet you have? Surely that should be a basic requirement?

FFS 23g is not even a nice fraction of 95g. It is obviously intended for me, and 3.13 of my close friends to share...