Royal Mail API (RMAPI v4)

We have used Royal Mail for postage for some time. They had an XML API which we have used, and it works.

What is interesting is how all of these APIs are moving from XML to JSON now. One of the big benefits of JSON is none of the FUCKING XML NAMESPACES (sorry, did I say that out loud?).

Even so JSON has its issues, but nothing like XML with namespaces.

Manuals

They have a web site with a login and loads of interactive stuff, and that includes the API and API manuals. They also have a "V4 Shipping API Consolidated Guide" PDF they sent. So how hard could it be. I had two manuals!

To put this in context, I am someone that has used (and written) many APIs that are JSON/http based over the years. I practically invented the concept (I even have a patent on form posting from over 30 years ago). I have some clue, honest.

I actually quite like the latest "web" based stuff with JSON sent and received. Even OAUTH crap. I have some experience with this, honest.

Update: Before logging in there are some extra links, and another set of manuals, which is yet again different and contradictory - arrrg!

Authentication

First challenge is authentication, and to some extent the manuals were not that bad. The one critical thing they could have done is include, in any of the manuals, one simple example.

They use OAUTH2 but in a way I have not seen anyone else use. I needed a POST to a URL with a URL encoding post of grant_type=client_credentials (and it had to be a POST not a GET) using Basic authentication of the client ID and key they allocated. This is an unusual way to OAUTH2, but valid, to my huge delay of some hours getting the hang of it was really down to me. I get back a Bearer key with expiry (1 hour).

API URL

One issue is the URL, well, the hostname. The documents state that the hostname to use is for their TEST system. This is good, they have a test system, but the other manual suggests it is the hostname for the LIVE system. Why?

I eventually worked it out, there is one hostname for authentication, and whether it is test or live depends on your account. Your account can be in "sandbox". I mean that is great, a good system, but why the fuck not state that in either of the manuals?

What adds to the fun is that hostname is actually only to authenticate. The actual hostname for the service is different, and searching both manuals I do not see the actual hostname for actual API calls listed anywhere.

Why? WTAF?

JSON blob

The big clue was some swagger.json file you can download. Loads of JSON, some "OpenAPI" thing. That has been way more helpful than either of the manuals!

It explains the actual hostname to use for the API, and each of the schemas for each API.

The manual, well one of them, lists fields you need, but omits that these may or may not be in sub objects. The JSON blob explains in way more detail.

And, of course, this JSON file, whilst really good, is missing a load of things, like creating manifests!

Errors

For example, the error "Could not find member 'ContactName' on object of type 'Destination'. Path 'Destination.ContactName'" fooled me for a while as I was supplying Destination.ContactName. What it meant is I should not be supplying that. I read "Count not find" as I was missing something, but no, it meant I had supplied something it "could not find" a use for. That is shit UI. (FYI I should have provided Destination.Address.ContactName)

Tracking

There is, of course, tracking. There is an API to get tracking and an option to have tracking posted to a web hook - yay!

But the documentation for the tracking API just says the response is a tracking number? Which, err, makes no sense, what of the status of the shipment. Hmmm.

And apparently the tracking web hook is not supported, so why the hell is it in the API documentation? Arrrg!

Working

I have it working now - but this is literally one of the worst documented APIs ever.

The raw JSON file was more documentation than either if the manuals, but even that is missing bits.

Icing on the cake

Royal Mail have What3Words embedded. I kid you not.

This is so mental! RM have PAF which has detailed (like, to 10cm) delivery point details on all UK addresses. They are the one organisation that has ZERO use for W3W.

I don't know what they do with it - I will ask?

• If a W3W supplied is way off, like hundreds of miles, do you use it?
• If a W3W is next door to postal address provided, do you use it?
• If a W3W is within target address, e.g. a shed in garden, do you use it?

If all are "no", I have to ask WHY THE FUCK it exists in their API!

[Update: Answer is they don't use the W3W filed.... So WTF have it?!?!]

WIFi: Aruba vs FS

I have used Aruba WiFi for some time - they are nice, but expensive. The latest Aruba Instant series seem more sensibly priced though.

When picking WiFi there are a number of factors, but if you are going to get more than just the WiFi that comes with your broadband router you almost certainly want something that can allow multiple access points and roaming between them.

There is also a matter of the number of antenna and radios, and the protocol - summarised as catchy names like WiFi5 and WiFi6, these days. There are also access points for outdoor use, which can be useful.

The Aruba has one cool feature, and the main reason I like it, that it does not need a controller. A controller is what directs devices to move from one access point to another as you move around, and what manages the overall configuration. With the Aruba, the controller is one of the access points, and can even fall back to anther one if the main one is off.

So what changed

I have been pretty happy with the Aruba for many years. It has a lot of features and settings (perhaps too many), and as I say, needs no separate controller.

But recently some of my IoT kit (ESP32), which is all 2.4GHz, has been playing up, sometimes sulking not reconnecting to WiFi, and so I felt it was worth changing the WiFI to see if it helped.

Long story short, the WiFi choice was not the cause of that specific problem, and I think I have fixed it. But the change of WiFi had some surprising side effects. For reasons I won't go in to, I have all my 2.4GHz on the same channel. And so that is a pretty full channel. As such I was never surprised to find s/w upgrades on a set of IoT devices was always a bit slow - I set up to do one at a time. To my surprise, with the new WiFi, it is way quicker!

What is the new WiFi?

I have got some kit from FS. It is cheaper than the Aruba, which is a start, but it needs a controller. The controller (AC-1004) is also a 4 port PoE switch which makes it actually useful for a small home installation as you almost certainly want a PoE switch for your access points anyway. So again, not that much extra cost. It can handle 64 APs. I may be a convert to the idea of a controller.

The (web based) controls are a but more clunky than the Aruba, but to be honest, pretty similar. It also has a huge command line interface!

The APs actually come with mounting brackets, unlike the Aruba! I have AP-N505 indoors and AP-T565 outdoors.

So far, so good. I'll see if I am cursing the change in a few weeks. If not, I'll take down the Aruba APs.

BMA423

The BMA423 is a small accelerometer by Bosch. The data sheet is here. It is 129 pages. It is really crap if you want to just use this chip. It repeats itself quite a bit as well. The Quick Start Guide has a nice vague bit about writing the init file as a burst write to address 0x5E.

There is reference code, and the help forums for this just have Bosch saying to use the reference code. Some of us would rather not. A good data sheet should not expect you to just use the reference code. It is also a bugger to follow.

One of the main issues is the the reference code works with registers 0x5B and 0x5D which the data sheet lists as reserved. This is silly, the data sheet could define these. The reference code also has 6kiB of binary code that seems to need to be sent.

So, here is what I worked out, and it is now working.

Initialisation

Reading and writing registers is standard I2C register read write. I.e. write the register address and then data, or write the register address and then read data, allowing multiple byte writes or reads as normal (with on exception, see below).

Checking

• You can check all is well, and init done, by reading 0x2A (INTERNAL_STATUS), and confirming it is 0x01

Soft reboot

• You can do a full soft boot by writing 0xB6 to 0x7E (CMD).
• Wait 1s after soft boot

Loading ASIC code

• Write 0x00 to 0x7C (PWR_CONF). This disables aps (advanced power save).
• Wait 500ms
• Write 0x00 to 0x59 (INIT_CTRL)
• Send the 6k block to the ASIC address 0 (see below) - see here for the data.
• Write 0x01 to 0x59 (INIT_CTRL)
• Wait 150ms
• Check status by reading 0x2A (INTERNAL_STATUS), it should now be 0x01
• Read the ASIC address at 0x5B/0x5C, this is where you load FEATURES_IN (see below)
• Write 0x01 to 0x7C (PWR_CONF). This re-enables aps

Block read/write to ASIC

It seems there is a separate ASIC block of memory, and this is accessed by three registers.

• 0x5B is low byte of address to next read/write - BUT NOT AS SIMPLS AS THAT!
• 0x5C is high byte of address to next read/write
• 0x5E is the gateway to reading and writing

Normally, with I2C it would auto increment the register on read/write when transferring multiple bytes. This does not happen with 0x5E. You can write to 0x5B/0x5C as a 2 byte transfer, as per normal I2C.

The ASIC address is stored oddly. It has to be even (bit 0=0) and you have to read/write even number of bytes at a time via 0x5E. The address is stored such that 0x5B is (address>>1)&0xF, i.e. bits 1-4 of address. 0x5C is address>>5 (i.e. bits 5-12). So you need to allow for this when reading or writing the ASIC address.

It seems from reference code you can transfer up to a maximum of 8 bytes at a time, so you have to repeatedly write the ASIC address to 0x5B/0x5C then write 8 bytes to 0x5E in one block. However, my testing suggests you can transfer the whole 6kiB init in one go, so totally pointless reference code.

Once the init file is written, and the status is 0x01, you have to read the address in 0x5B/0x5C which tells you where the FEATURES_IN block is located in ASIC memory... Shift it appropriately to get the byte ASIC address.

FEATURES_IN

FEATURES_IN is block of 64 bytes in ASIC memory. You can read/write even number of bytes, setting 0x5B/0x5C to the base address (as read above) plus the offset in to the block. The data sheet does list these additional registers reasonably well. It just lacks any explanation of how you access these bits!

You can read all 64 bytes in one go, and write all 64 in one go, or read/write even numbers of bytes if you only need to change one register.

Steps

To enable step counting, I have done the following :-

• Write 0x00 to 0x41 (ACC_RANGE) to set ±2G
• Write 0x02 to 0x56 (INT1_MAP) - only needed if you want interrupts
• Write 0x04 to 0x7D (PWR_CTRL) to enable accelerometer
• Set FEATURES_IN(0x37) OR'd 0x30 to enable activity and counter
• Set FEATURES_IN(0x3B) OR'd 0x10 - not idea what this does as reserved in data sheet! 

Reading steps is then 4 bytes from 0x1E onwards, low to high bytes for 32 bits.

Resetting the step counter is setting FEATURES_IN(0x47) OR'd 0x04 (self clearing).

NON-STERLING TRANS FEE

One thing pisses me off is cards charging a Non sterling transfer fee. I'd have expected them to make money on the exchange rate, or be big enough that they can handle currencies without such silly fees to be honest. Even more so when it is someone like American Express, that, well are American, and I am paying something in American dollars. You would think they could handle dollars.

But I have recently discovered something slightly odd.

VISA

Using my VISA debit I was charged...

AMOUNT IN USD 206.87 ON 28 JUN VISA 1.269999 FINAL GBP AMOUNT INCLUDES NON-STERLING TRANS FEE £4.48 = £167.37

But when I was then refunded, it looks like...

AMOUNT IN USD 206.87 ON 30 JUN VISA 1.2669 FINAL GBP AMOUNT INCLUDES NON-STERLING TRANS FEE £4.49 = £158.80

This means I was charged for the refund, and oddly charged at a higher rate (by 1p).

Amex

Using my Amex I was charged... (different day, I have not compared exchange rates)

71.15 UNITED STATES DOLLAR, Non-sterling transaction fee: £1.68, Exchange rate: 1.2687 = £57.76

But when I was then refunded... (next day)

71.15 UNITED STATES DOLLAR = £57.82

Comparison

VISA charge the non sterling fee on charge and refund, but Amex do not. Indeed it seems they effectively refund the non sterling admin fee.

VISAs fee appears to be around 2.7%, Amex fee appears to be around 2.9%, very similar.

Conclusion

It is actually in my companies best interests if I pay for these on my personal Amex and claim on expenses, if there is any chance of a refund (which happens quite a lot if I have arguments with JLCPCB, for example).

toot

Twice I setup Mastodon.

Once many years ago - I followed the step but step guide and got it working. It worked for quite a while and then broke - out of disk. I gave up.

Then a second time we easier - this time my ops team set me up a VM, and Mastodon had improved a bit, so set up again, following the instructions. Again it worked. It caused problems as it was a "new" instance on an existing domain, and that caused things that had cached stuff to get confused. It got sorted, people had follow requests they had to cancel and try again, that sort of thing.

These days it seems to have a lot more "cleanup" logic, and a lot more obvious on disk usage, but still far from perfect from what I can see.

WTF is it not simple?

My big issue really is that Mastodon should be like apache, or many other quite complex systems, which are basically apt install mastodon. That really is what it should be. Simple as that. The web interface or command line could take you through the initial steps for admin user password and domain name and so on, but that should be it.

There are docker instances. I know nothing about docker, but I am told they sort of just work. Maybe that is the answer?

But why is it not as simple any anything else on linux, really. It seems Ruby is the main problem, as is python it seems. A pain in the arse.

But all is well, so not problem.

Yes, all is well, and even upgrading to next mastodon version as needed was not that hard. Again, following a load of step by step instructions not a simple apt upgrade mastodon or some such. But not rocket science.

But err, then, it broke. Basically the ops team wanted to upgrade Debian to bookworm, which is fair enough, but doing so cleaned out almost all of the stuff Mastodon needs.

Now I am not just upgrading Mastodon, and not installing Mastodon (as I don't want to lose/break what I have), I am sort of trying to make it work from instructions that are one or the other, and constantly hitting stupidities like OpenSSL is not available (how the fuck is that not standard in these tools?!) and not clue how to fix that.

I gave up

I must be getting old, because, yes, I gave up, it is too much of a pain in the arse, and I have better things to do. I tried, honest, but life is too short.

Bring in the experts!

The simple answer is to call in some experts. In this case Mythic Beasts. They actually run the A&A instance because my ops team were smart enough to not go near this with a barge pole.

They are not only running my instance, but in very short order managed to copy all the stuff from the old, and now broken, instance over, and make it continue seamlessly for me, well mostly - some avatar issues they are fixing. But followers, and so on, all working. I have not missed anything and was off for a day at most.

It is not cheap, it is several hundred quid a year. But what do I value my time at?

There is an irony here. I am paying that for my social media. If the likes of Twitter had people paying that for a really good, clean, not sales/advertising/pushy/confrontational/stressy service, they would make a fortune. That is not their business model.

The only mitigation is it may end up being more than just me - some family members, when they finally catch this boat. Of course, subject to whatever stupid rules the Online Safety Bill may create, LOL!

Pipelining

I am reminded of years gone by when "compiling" would take a long time, even hours in some cases.

This lead to a "pipelined" debugging and development cycle. By the time what you did was compiled you already had fixed a load of bugs and added a load of new features that needed testing. So you had to test things from the compilation as they were when you started. Sometimes you just knew the compilation would create something broken, but some other aspects would be things you need to test, so you let it run anyway to test them. You had to make code that would include things you need to test in the best way possible.

I am sort of doing that with PCBs over the last week or so - getting a new PCB takes slightly over one week from China.

I have been working on a load of small changes to various designs.

• New 5 pin pad for programming - I made a larger set of holes as the ones on my last batch were annoyingly small and have to be held in place while programming. This is one I finally have back prototypes and my new design is too big and loose, FFS.
• New New 5 pin pad for programming, staggered more than last time to allow pins and hold them whilst programming - waiting - works perfectly
• New voltage regulator - waiting - and I really hope it is right as I have several in the pipeline with no fallback - one version works nicely (the cheaper one), the other (tantalum cap) does not work.
• New USB UART - got a board, discovered I cocked up - but was able to rework some and test the correct design - some boards were able to be used anyway
• New New USB UART - fixed layout/design issue - waiting - working
• New RGB LED - waiting, but already spotted that I have cocked up and got G and B swapped, FFS. This means the resistors will be wrong for the colours as well. May be able to rework when this arrives.
• New New RGB LED - waiting on revised, but still not been able to check if resistors right - waiting - working
• New panel design - i.e. making part in a 70x70 panel instead of JLC doing it, and if a barcode is clean in silkscreen - waiting - looks nice but JLC being a pain
• New Tx on Faikin, open drain output - waiting - working
• New Rx pull to 3V3 on Faikin, detect loopback - waiting

The whole thing is frustrating. I have had to scrap one set of boards sadly because of the CH340X issue. Others I was able to rework, or remove and use a 5 pin programming header instead. One trick is making a new board that you can "fall back" if the new bit does not work.

The RGB LED is a pain - I used one (side mounted) that worked nicely because they did not have the simple one I got from Mouser and used for my hand soldered boards. But JLC places it badly (over 10% of boards needed rework). And I found they had an equivalent simple front facing RGB LED with low enough voltage on the blue for 3.3V working. The issue is the resistors - you can work out values - as I had originally but then find green is way too bright. For my manual working I had changed resistors and so am trying those values to start with. But until I get them, and get the G and B right, I will not be sure I have the right resistors even if I have the right colours. I may have to rework some 0402 resistors to test when they arrive.

On top of it, DHL, who have managed around 2 days from China with no VAT issues, have suddenly made it 3 or 4 days and not doing proper VAT with postponed VAT accounting - that will be a fun argument as I am planning to refuse to pay and insist they re-do as PVA. We will see.

Also JLC being odd on rules, insisting multiple designs on a board even if only in solder resist. Still not resolved. Even if I end up making panels with no extra silkscreen, at least making as panels myself, instead of leaving JLC to do it, lets me make cleaner slots and edges on the designs.

I have to test if the barcode in silkscreen works, and is acceptable (inside a nearly clear, slightly pink, anti static bag) with Amazon - so avoiding having stickers.

So a very frustrating week, and another such week to come.

CH340X DTR/TNOW with ESP8266/ESP32 BOOT and RESET

The CH340X is a small chip that does USB UART, and is part of a family of CH340 chips in various sizes. A UART can need few or many pins, and they do a really small one that is just Rx, Tx and RTS pins, which is the minimum you would want. But they also do a version specially aimed at the needs of small processors like the ESP to handle boot mode.

BOOT and RESET

Both ESP8266 and ESP32 (and many other processors) have a BOOT and RESET input, well, actually GPIO0 and EN.

You need these to program and debug. To program you have to set BOOT MODE, (which is GPIO0 pulled low) and pulse RESET (EN low and then high). For debug, you want no BOOT or RESET, or maybe RESET to restart the processor, but not BOOT mode (GPIO0 staying high).

What is weird is the various ESP based modules out there. I have seen complicated FET based links of RTS and DTR from a UART linked to GPIO0 and EN, and some with R/C combinations linked to RST and GPIO0. I think the logic here is for more traditional use of DTR and RTS, i.e. usually one has RTS active the whole time, so an R/C allows RTS to be activated (low) but EN to pulse low and back high. Similarly linking the use of DTR and RTS to allow a normal UART operation not to be BOOT MODE.

However, the esptool and idf.py as part of the ESP IDF, and even the tasmotizer tool and web page all treat DTR as BOOT MODE and RTS as RESET directly.

You can connect DTR directly to GPIO0 and RTS directly to EN, and it works perfectly!

What is even more weird is the number of modules you see with actual push buttons for BOOT and RESET on the board, which costs money to fit, even though they have a USB UART.

I can only assume some legacy designs and copying reference designs and so on.

All my designs connect DTR to GPIO0 and RST to EN, and work perfectly. Yay.

CH340X DTR

The CH340X has Tx and Rx and also CTS (input) and DTR/TNOW (output).

The TNOW is multi function though, the datasheet says :-

5.3. DTR and multi-mode MCU download

For CH340X, 6# Pin defaults to TNOW, weak pull-up during power-on or reset, output TNOW during normal operation used for half-duplex transceiver switchover. By connecting an external resistor with 6# pin, TNOW can be switch to DTR#, the two options are as follows:

1 If the 4.7KΩ pull-down resistor is connected to GND with the 6# pin, it will enter the open source DTR enhanced mode, and the 6# pin will automatically switch to the open source driven DTR# pin , which is used to connect the boot mode pin of MCU. The default DTR # is not output and is kept low by the external resistor, but the DTR# pin can be set to output high level or not output by the application program, which is used for multi-mode MCU download with DTR# default low level.

2 If a 4.7KΩ resistor is connected between the 6# pin and the 5# pin, it will enter the push-pull DTR enhanced mode, and the 6# pin will automatically switch to the push-pull driven DTR# for connecting to

the control pin of the MCU. The application program sets the DTR# pin to output high or low level for multi-mode MCU download with DTR# default high level.

This is excellent, and makes it clear they have considered the use of DTR as a BOOT MODE for processors. I have not tested the DTR to GND via 4k7 - I think they are saying it inverts, i.e. DTR drives high. I may be wrong.

But the second mode, DTR to CTS via 4k7, I have tested. Well, actually I misread / assumed 4k7 to 3.3V, and that does not work, but when 4k7 to CTS it works, and drives the pin as DTR allowing direct connect to GPIO0 yay

Why a 4k7?

But I am puzzled. I understand the 4k7 to GND, as it pulls down, and allows active drive high as needed. But why 4k7 to CTS.

CTS is an input, so the only use of the link is to allow the CH340 to confirm the connection from DTR/TNOW (output) to CTS (input). Indeed, it pulses that output low a few milliseconds after startup, presumably to test it is connected to CTS.

The thing that puzzles me is why the 4k7. With CTS as an input it is quite safe and valid to connect DTR directly to CTS, output to input. And, as you would expect, doing that causes the CH340X to switch in to DTR output mode.

I suppose one could argue that you may want to make use of CTS as well, and the 4k7 allows that even if DTR is outputting something different. Except that does not work. If you are using CTS, that means you have an active signal feeding CTS from somewhere. If that is the case, the test of DTR to CTS does not work (I tried it!) as CTS will not see the signal from DTR/TNOW via the 4k7. So the 4k7 does not allow that. Indeed, the only use may be if you have something driving CTS which does not drive it at power up - that would be a rather special arrangement and somewhat unlikely. That said, I suppose if CTS was a general purpose GPIO from a CPU which starts as floating/input, it could make sense and allow CTS to be signalled from the CPU.

So I'd expect the data sheet to say just link DTR to CTS and not use CTS, or use 4k7 if CTS is floating at power up and needs to be used.

Anyway I have done that (just linked DTR to CTS), and it works nicely.

USB-A tracks to CH340X

There is a snag though - using DTR to drive GPIO0 works, but on my Shelly board it is drive GPIO0 that is connected internally in the board, with some pull something, and an LED, and so on. So the CH340 might "see" the pull down on it when it starts perhaps. So I have to drive through a buffer in that case. Thankfully that is basically the only case, as normally it just goes directly to GPIO0 on the connected ESP module.