Hayes AT

A long long time ago, in galaxy far far away (no, scratch that bit), a standard evolved for talking to modems, well, specifically Hayes modems, but it was widely adopted.

It works over a serial link by sending commands starting AT for (attention). Now, I am pretty sure they were reasonably consistent, back in the day, but like many things this standard has evolved and been bastardised.

We now have things like this SIM800 module, which is basically a mobile phone in a can, with no display or keypad. It talks on a serial bus and talks Hayes AT commands, but as modified by various GSM bodies.

Now, some AT commands are consistent. You send AT and some options, and get some data followed by OK. Good.

There is a reasonably consistent format for commands and settings and queries, mostly. There are then AT+ functions which are used for a lot of the mobile stuff.

But it is far from consistent. The AT+CIPSEND, after the > sends "SEND OK" not "SENT OK" or "OK", and AT+CIPSHUT sends "SHUT OK". Why are these not just "OK" like the rest? I am also pretty sure I had one case of a response with no OK (but cannot find it now).

Then we have the asynchronous messages from the device. Most start with a +, like "+CREG: 0,1", so can be recognised if you are expecting another response. But some are just text like "SMS Ready". These smack of debug messages left in the code.

Half the interface was clearly designed for a person sat at a terminal typing stuff, and the other half was designed for talking to a machine!

But when you get to some of the higher level functions, like establishing a UDP "connection", it gets even more special. For a start, you look for "CONNECT OK" not simply "OK", and mostly at that point it seems to avoid sending asynchronous messages (though I cannot be sure). Some still can be sent, like "+PDP: DEACT" when it loses the connection. What you do get though, with no prompt, header, or indication of length is the raw binary content of any UDP packets you receive on the connection. So send a packet containing "+PDP: DEACT" and I'll think the connection has dropped!

What would have been sensible would have been a line something like "+DATA=N" and then N bytes of data, or some such, but no, it is simply the raw UDP packet.

Then we have some interesting little bugs and discrepancies. One that took a while to work but was that the length of UDP data I can send is limited. Not to 65535 which is what IP limits to, but to 1472 which is the available UDP in a 1500 byte IPv4 packet. Except it is not. It is actually 1460, any more says "ERROR". Oh, and it won't allow me to send a zero length UDP packet - no idea why, and as NAT is often used this is something I may want to send as a keep alive. I ended up having to send one byte.

But there is an even more special feature. When you receive a UDP packet, as I say, it arrives as the raw binary data via serial. I am relieved to see no mangling of any bytes, which is good. Except it truncates the last byte of the message. Yep, I have to send an extra dummy byte on my packets to the device. WTF?

I could save myself from a lot of this by coding PPP, pretending to dial on an old style modem and talking PPP which the device fakes and converts to packet data. This is a serious consideration, to be honest.

However, after a lot of messing about, my GPS trackers are sending (and receiving, when needed) tracking data over UDP over mobile - yay!

GPS trackers

I am playing around with GPS trackers (I'll post more on hardware for this later). This is for commercial vehicles, not anything covert, and I am not going to even touch on GDPR in this post :-)

Reducing the number of points

One of the challenges is managing how often we sample location, and if/how we reduce the number of points. There are a number of reasons to do this:-

Local Storage

One reason to reduce the number of points stored is where the tracker is storing the points - either for download later or for transmission when mobile coverage comes back, etc. The local storage of a small embedded device is likely to be limited. The GPS module I am using has several ways to do this (I have yet to work out how to do other than simple interval). Storing every second, which it will happily do, runs out of space in a few hours.

Transmission

Transmitting the data over a mobile network has costs. Yes, there are SIM packages with unlimited data, but if you have a fleet of hundreds of vehicles you can do better than each of them on a package for tens of pounds a month and unlimited data. There are packages with much lower monthly costs but data usage costs, and there are foreign SIM packages with roaming for better coverage and higher data costs. Reducing the data transmission can get the costs down to pence a month. This means not only reducing the number of points but finding ways to reduce the overhead of transmission of the data itself.

My plan is to pack data in to an encrypted UDP packet, with a simple resend packet sent back if the data has not arrived at the expected time. I.e. an entirely NAK based protocol so that normally it is one packet in one direction, probably every 10 minutes or so. This means really low mobile data usage.

Database storage

Reducing the number of points also helps with back-end storage. Hundreds of vehicles tracked every second means a lot of data. Simply increasing the interval loses useful data so you ideally need a smarter way to reduce the data.

Display

Perhaps a small point, with computers and networks being so fast, but reducing the number of points sent to a browser to display a route can make things much more responsive.

Where to reduce points

At present I have code to reduce the points when pulling from a database and sending to a browser, with an option to also delete the removed points from the database. But ideally you want to do this in the tracker itself - that way you have all of the benefits for local storage and transmission as well.

This also creates an extra advantage - if you have a means to reduce data points effectively you can start by collecting much more data in the tracker. The GPS modules I am using can do 1/10 second fixes, so why not start with this, and reduce points from there.

How to reduce points

I pondered how to do this, and came upon with something based on removing points within a certain distance. I then found there is a standard algorithm which is basically very similar to what I came up with. The Ramer Douglas Peucker algorithm.

It recursively takes a set of points. If all intermediate points are within a specified margin of the straight line between start and end, then those intermediate points are removed. If not, the one furthest away is kept and the point from start to this, and from this to end, are recursively processed. The end result is a path of straight lines where all the removed points must have been within the specified margin of that line.

This works well, and removes points on long straight roads but keeps points driving around a roundabout!

Even setting a margin as small as 1m massively reduces the number of points, and what is even better is that you can start with say 10 times as many points (e.g. 1/10 second fixes) and still end up with similar final number - just that the points you select are more precise (i.e. the exact edge of a turn, not just nearest second).

Of course I won't have a full set of points in the tracker, I'll have to work on time periods, like 10 minutes, collect data, and then apply the algorithm. This means there will be points based purely on time as the end points for the chunks of data I process.

Distances

One issue is that the raw data is lat/lon and not ground distances. There are ways to work out exact ground distances around the curved surface of the Earth, but a simpler way is to just assume we can map lat/lon on to a flat plane. What I do is, for each stage of the algorithm, take all lat/lon relative to the middle, and convert to metres. For latitude this is simply multiple my 111111 (yes, apparently this comes from French definition of a metre). For longitude it is 111111*cos(lat), remembering to convert to radians for cos(). This means I can then work out distance from a line in metres with minimal error. Of course as the algorithm recurses and processes smaller line segments, the error because the Earth is not flat (really, it is not!) reduces.

3D

For a change, I am not talking of 3D printing, but we have altitude as well. So I am including altitude in the calculations. This means that a straight road that has a peak hill gets a point kept at the top of the hill!

4D

Yes, 4D printing would be cool, but no, I am now including time in the algorithm. That way we get points if a vehicle stops and starts or changes speed. The only issue here is deciding how time works as a unit in the distance calculation, so I need a metres/second. I have made this small, treating a 1 second deviation from the line between two points in space and time as a small fraction of a metre, hence you have to change speed or stop for many seconds to create a point you keep. It seems to work well.

Dropping summer time

It seems moves are afoot to stop the clock changes, and I rather like the idea.

The issue, for me, is that people talk of "permanent summer time" as the answer, sort of forgetting that the clocks are set based on lots of measurement at the Greenwich observatory and hence why we have Greenwich Mean Time. It is summer time that is the "anomaly" and it only makes sense to me to be on GMT all year.

This assumes stopping clock changes is a good idea, and that seems to be a matter of debate, but one which is leaning towards the idea of not doing it.

So, some thoughts on why, to my mind, it is obvious we stick to GMT all year.

The matter of definition

It is called GMT, and is generally the high sun at the middle of the day at 12:00. It would be crazy, IMHO, to have the mean time over Greenwich as Greenwich Mean Time plus one hour.

What would we call it?

We can't call it GMT and it makes no sense to call it BST as not only summer time. It won't be UTC if +1, so actually we need a new name. That is more of a pain that you realise as it is a lot of software and all sorts. UKT maybe?

It only matters in winter anyway!

In summer we have more than enough daylight. Yes, moving some "unused" daylight from early morning to evening is vaguely helpful, or was before the invention of street lights. But what matters is Winter.

We tried a permanent summer time in 1970 and it did not go well. We know GMT in Winter "works" well enough and fits around schools and so on.

Changing what we do in Winter (by making permanent UTC+1) would be way more problematic as we simply have less daylight to play with. Changing in summer (by stopping BST) is not an issue as we have daylight to spare, so to speak.

Give UK an advantage?

There are a lot of things that log in UTC, and confusion over UTC and local time happen all the time. Heck, even the Amazon app gets the order date wrong all summer because of this. BT get this wrong in their XML in lots of places. There is an advantage for the a country that stays on UTC+0 as their time zone as all UTC logs will be local time, and simpler for everyone.

Some have even suggested that UTC should be moved to the middle of the Atlantic so no country is UTC+0 and hence has any advantage. Let's give the UK that advantage.

ESP-IDF re-partitioning the flash on the fly

The ESP32-WROOM-32 has a 4M flash, which is usually carved up for boot loader, some small areas (NVS, etc), and then three code image spaces: 1 for "factory", and two "OTA" areas which alternate. Well, that is one of the two defaults you can set.

Sadly, even the most basic of code, with even log levels of logging enabled can easily hit a 1M code space limit. So I decided to make a new partition table.

Just two OTA spaces

The first thing I confirmed is that if I just have two OTA areas, and no "factory" area, it works fine. It can alternate the OTA flashing, and make flash just flashes the first OTA area.

Oddly, it looks to me like you can have two OTA areas of 0x1F8000 but the make system said it would not fit. I ended up going for 0x1F0000 which is close to 2M.

On the fly update

The real trick was changing the partition table on the fly. The key issue here was ensuring the original factory build was in the same place as the first new OTA block. I changed from factory+ota_0+ota_1 to just ota_0+ota_1 at 0x10000. This meant, whichever image I was running, by changing the partition table a reboot would find the factory release (in ota_0) and run it.

I simply had to erase and re-flash the partition block. I obtained the signed binary partition from the build directory and embedded. Then I just reboot, and sorted. Obviously you then need an OTA upgrade to move off the original factory code.

The simplest check was to compare the current partition table with the one I want, and if different, re-flash and reboot.

You do need to tell the SPI flash component to allow you to erase/write to dangerous areas of flash, but otherwise the code was pretty simple.

End result, I have nearly twice the code space.

GPS made easy

I am surprised how simple GPS is these days. For only £12 or so (from RS of all places, part 908-4085) I can get a small module with built in antenna that runs of 3.3V and provides serial GPS data and a very precise PPS output, is only 15mm square, and easily soldered to a PCB. It even has built in RTC that can run off a small button cell battery for fast fix and retaining time of day.

I have yet to decide on the best applications of this, apart from simple things like a speedo for a bicycle. There are applications with location logging (that dump to WiFi when back in range), or even if you just need a reliable time source for something like a door entry system (or a Brexit countdown clock, LOL).

A clock showing how long until sunset, wherever you are, would be cool :-)

I have made a module with GPS (the Adafruit module, as I found that before the RS part), and a display, so I can have a play around with it.

P.S. I have it set up for my bike, even with amount of daylight left...

Dissolvable PVA support

The TAZ Pro has two extruders which allows me to try and use dissolvable PVA support. Support is simple enough but hard to remove from the print cleanly, so using PVA allows extra options - just dissolve it!

I have had to play with the Simpify3D settings a bit as the settings for the TAZ 6 did not quite work. I am not sure if the bed is different, the nozzle, or what but my prints were almost welded to the bed. I have tweak the settings for nice clean prints with nGen now. The next challenge was settings for PVA supports.

I ordered some from RS (yes, the price was not silly, strangely), part 174-0082. Well, actually I ordered from someone else, and realised wrong diameter, after opening it, D'Oh, but now I have the right stuff I googled a bit to find temperature.

It is funny stuff, and I ended up printing at 205C which is apparently on the high side. I could also see from simply feeding the filament that it came out thick and slow (around 1mm).

The key setting needed to stop it just curling up was speed - it needs to be very slow. In the end I ran with 2x multiplier, 1mm wide, 20% print speed, and that actually worked. Well, mostly (the eyes went a tad wonky, but worked).

For a start, the PVA comes away from the model really easily, so that is a good start.

Then, put in warm water for a while to remove the last bits, and yay, it worked.

Now to try something more complex with enclosed parts that simply could not have been printed before.

LulzBot TAZ Pro (dual extruder)

I have been 3D printing for a long time - and even coded a slicer once.

I have tried many printers. My current favourite for cost and performance is the LulzBot TAZ. We have a TAZ 6 at the office and (even though not on the A&A web site) we do 3D printing if you want.

It is a nice heated bed, which works well, especially with my filament of choice ColorFabb nGen.

I have been using to make cases for various electronics / R&D.

Sadly I slightly fried my TAZ 6 units. I have a new controller board on the way, but decided to check what they have now, and saw the TAZ Pro. So I ordered one and to my surprise it arrived in a couple of days (in spite of warnings of back orders and long lead times). Nice.

It is nice...

But is it worth the price (around £5k)?

Well, possible. It has some nice features and works very well. I have only been playing with it for a day now, but I can make some comments. The bed levelling is faster and seems more consistent. The Z axis is belt driven meaning it can easily use Z moves when moving from one part to another (far too slow when Z axis is screw). The big thing is the dual extruder.

Cura?

LulzBot recommend Cura, a good, free, 3D slicer. It works well, and makes some impressive prints, but...

• It seems way slower than I am used to (I used Simplify 3D before)
• The prints, whilst nice, and precise, seems brittle. I expect minor tweaks to settings would help.
• It expects nGen to be glue sticked to the bed - and I know it can be printed without, but needs hotter bed for first layer. I tried that and it worked.
• Somehow the prints were distorted. I am not sure if cooling is right. I am sure it could be tweaked.

However, it did work, as I say. And the reason I tried it is that Simplify3D, which I was using, had no profile for the new printer.

But, Simplify3D support emailed me one, within a few hours of asking. It works!

Simplify3D

Whilst I do not usually end up recommending paid-for software, this is good. It works well. It is way faster. It is not perfect, but nothing it, but it is nice to use. So I recommend it.

Dual Extruder

This is the first time I have used a dual extruder. But the TAZ Pro does it really well. The heads are motorised to retract completely out of the way. The full bed size is still available to both extruders. It really works well.

There are several reasons to use a dual extruder.

• Two colour prints - obviously. To be honest this is a pretty minor use case. But it works well.
• Mixed materials - combine fixed brittle material and flexible rubber to make complex designs - not tried it yet, but fun possibility.
• Using dissolvable supports - this is likely to be the main use.

The dissolvable supports will be subject of another blog when the PVA reel arrives. Basically, 3D printing like this cannot always print what you want as some things are "in mid air". Using support material works, but has to be cut/broken off, and is impossible to do "inside" some designs. Using dissolvable supports allows the impossible to be printed, and then put in warm water for a while. I look forward to it.

The dual head working is impressive...

In the mean time - two colour test print...

Any downsides

The only one so far is the heads are covered by a safety guard with warning about being hot - which means you cannot see what is printed as it prints.  Ironically it means you cannot see the print head, and can end up burning your finger when trying to adjust something because you cannot see where your finger is. Elf 'n' safety gone mad, IMHO. If it always "just works" then I guess it is not a problem. TBH they should have designed with angled plastic to make a guard that allowed viewing of the actual print head. This may seem trivial but I do feel is more important than they realise. Even ignoring the practical aspects of knowing a print is not working right ASAP, seeing what is being printed it important for selling the whole idea of 3D printing...

USB-C not so easy

Using a USB-C connector for power only is, sadly, not as simple as I thought.

Yes, the pins are tiny, butt the end pins are double pins on the connector and just about solderable (0.6mm wide for two pins together).

However, whilst this works when you use a legacy cable from a USB-A, giving 5V, this will not work to a true USB-C power supply or device.

Why?

Well, resistance is not futile, it seems. There are two connectors (CC1 and CC2) on the USB-C which need resistors to GND to tell the other end you want power.

Whilst the end power connections are two pins at a time (0.6mm), the CC1/2 pins are not, and are 0.3mm wide at 0.5mm spacing. That is pretty much impossible to mill, as the milled track is anything from 0.3mm to 0.6mm wide. The picture above shows around 0.5mm milled track, which is deliberately spaced to allow soldering to the pin 6th from the right. So double pin on right (GND), next double pin in (VBUS) and then another pin we don't care for and then the CC pin. Same on the other side of the connector, except the CC pin is next to the VBUS on that side.

Yes, it is possible to solder!

Then to add the 5.1kΩ resistors to GND.

The end result is power from USB-C...

P.S. Thanks to John for pointing this out to me.
P.P.S. John also pointed out that you do need the two resistors - and cannot simply common up CC1/CC2 with one resistor (as RasPi4 did, by mistake).

Powering widgets

Making widgets for the alarm system means that power is quite simple - the alarm wiring has 12V DC, and a simple regulator means I can connect to that - typically with screw terminals.

However, making other widgets with an ESP32, like the environmental monitor, or the Brexit clock, means finding a way to power them. I am not yet playing with battery powered stuff (will do, eventually), so need a power lead.

So what to use?

Initially I tried a micro-USB. Leads for this are very common, and USB-A sockets providing 5V are so common they are fitted to standard power sockets even, so seems ideal.

The issue is the connectors. I have struggled to find what I need. I did see some from China (which have not yet arrived) which have only the power tabs on them making soldering easier, and have clips to hold to the PCB. Whilst waiting for those, I tried some simple surface mount connectors.

The problem is they are not very robust, even superglued to the board, they can become detached very quickly.

I have actually made a set of tools under OpenSCAD two make tight fitting 3D cases to help hold the connector in place, which has helped, but that is still a challenge, and I am not happy about it.

The other catch is the tiny tiny surface mount connectors... Thankfully I only need the ones at the ends. For reference, the three pins at the bottom are 0.1" spaced.

So, I wanted to find a solution, and turned to USB-C. These are a nightmare, not only tiny tiny tiny surface mount tabs, but tabs under the connector so you cannot get to them with a soldering iron. Finally I found a part that will do, and from a UK supplier.

It did mean some fine milling, and even finer soldering, but again, only needing the double connectors at the ends, so just about possible. Those pads are 0.5mm spaced!

The result is a nice solid, soldered to the board, connector.

IoT

I have not done much on the IoT for over a week, but being rather ill for a few days has not helped 🤮🤮🤮🤮🤮.

The Brexit clock is getting scary!

My grandson wants to do something with LEDs some time!

The environmental sensors are working well, and I have nice graphs from it... This will probably lead to some fans in the office now.



I am thinking the next IoT thing to look at may be Bluetooth. Any suggestions for how that could be fun?

But yeh, the LEDs for my grandson are going to be a thing soon. Here he is abusing his first LEDs.