Debugging

There are lots of ways to debug stuff, but at the end of the day it is all a bit of a detective story.

Looking for clues, testing an hypothesis, narrowing down the possible causes step by step.

It is even more, shall we say, "fun", when it is not definitely a software or definitely a hardware issue. Well, to be honest, we know it is hardware related, but it could be hardware because the software has set something up wrong, or is doing something wrong, maybe. Really a processor hang should not be something software can ever do no matter how hard it tries, in my opinion, but in a complicated system with complicated memory management hardware, it is possible that a hang can be the side effect of something wrong in software.

I was going to say that "when I was a kid, software could never cause a hardware hang", but I am reminded not only of the notorious "Halt and Catch Fire" accidental processor operation, but that one could walk in to a Tandy store and type the right POKE command on one of the earliest Apple machines and turn it in to toast, apparently. So maybe there has always been this risk.

The latest step in the "watching paint dry" process of trying to diagnose the small issue we have with the new FireBricks is underway now. It has been a long journey, and it is too soon to say it is over. It will be an awesome blog when it is over, honest.

One of the dangers with software is the classic Heisenbug: a bug that moves or goes away when you change something. We are chasing something which, by our best guess, is related to some aspect of memory access. This means that even the smallest change to software can have an impact. Make the code one byte shorter and you move all the interactions with cache lines when running code, and change the timing of everything as a result. When chasing a big like this, you cannot rule out those being an issue. So a change of one thing may result is a change in behaviour somewhere else. We have seem a lot of red herrings like this already.

The latest test is unusual for us. It is a change to an auxiliary processor that controls a specific clock signal to the processor before the code even starts to run. One we don't currently need. And we are removing anything we don't need, no matter how unlikely it is to be the cause.

What is fun is that this means we have not changed a single byte of the main code we are running.

If this works, and only time will tell, we can be really quite sure it is not some side effect of simply recompiling the code. It pretty much has to be the one thing we really did change.

Being able to test something so specific by a software change is quite unusual.

Big number maths

One of my first C programmes ever, at uni, was a calculator. I had done a lot of code in BASIC, Z80, 6502 and other languages before, but C I learned at uni.

It was a strange exercise for me as I was teamed with someone else, and basically gave up on him. To me a calculator, i.e. something that could parse 1+2*3 and get 7, and understand brackets, was inherently simple. The code was a couple of pages at most and I had added all sorts of extras over and above the basic +, -, *, / binary operators. It handle brackets and operator precedence, and even had a table of operators to allow easy expansion. My "partner" made code that was dozens of pages and every test that involved extra brackets or some unexpected sequence broke it and he had two add extra code for edge cases. Oddly I once encountered a compiler that must have been written in such a way as adding extra brackets, e.g. 1+(2*3), actually broke it and made wrong code. Scary that stuff like exists.

My code has a simple loop to process: prefixes or "("s, operand, postfixes or ")"s, operator, in a loop, ending after last operand. It stacked operands, and operators (after processing the stack for any higher operators before adding), and that was it. It allowed unary pre and post operators, and binary operators. Simple and easy to understand, IMHO.

Of course, it used the normal C code to parse and output numbers, storing internally as floats (I think).

However, having played with mechanical calculators, one of the things I have meant to code one day and had not got around to for over 30 years, is a decimal calculator library... That is until this weekend.

Basically, one tool I have written, and my friends use a lot, includes an "eval" function to evaluate a simple sum for them. It is used in loads of places (as it embeds maths in a back end for a web page). It used integer (long long) types (64 bit) and shifted by number of decimal places it saw, giving it around 18 significant digits. Sadly it broke if you went over that (my bad) and they had some silly case of some numbers plus a fraction which had rather too many decimal places. So they asked me to fix.

A simple fix was limit the size, and then limit size and apply bankers rounding at the limit.

But I decided this may be the time to finally do that decimal maths code. I googled, and there are a few decimal libraries. There are also some arbitrary precision binary libraries. To my surprise the latest GCC has decimal types, where data is stored and processed in decimal. I can only assume we have decimal maths in some processors even, these days. This is good for accounting where the rounding errors you can get are bad (essentially binary does not have a way to represent 0.1 or 0.01 without recurring digits). Sadly GCC may have it, but clib does not yet (scanf and printf), but will some time, and I can see that being useful. Even so, _Decimal128 does have limits on number of digits. There are libraries in other languages, but I wanted something for C.

So I made a C string decimal library, and put it here on GitHub for all to use.

It is a library and command line, and includes functions to add, subtract, multiple, divide, and compare, as well as a simple evaluation function that parses basic sums, just like my uni project calculator.

The key thing is that it works on C text strings for numbers. A simple sum (I learned at school, and now out of date) was 366.2564*86164.091=31558149.7789324. It was numbers from an encyclopaedia in the library. I remember because I had to do the maths manually as no calculator I had would do it to full precision. Even google calculator truncates it a bit. That was my first test, and it worked. Yes, I also learned π to 50 places (and now only remember 25).

So, for add, subtract, and multiple, it is simple to ensure you have as many digits as needed. Sadly division can go on for ever, so I had to include a limit of decimal places, and a rounding rule, and remainder option.

I included rounding on limit on division, and also a separate simple rounding function - obviously actually applied at whatever digit you set. Default is banker's rounding :-

• Round towards 0 (i.e. -3.9 rounds to -3)
• Round away from 0 (i.e. -3.9 rounds to -4)
• Round towards -ve (i.e. -3.9 rounds to -4, 4.2 rounds to 4)
• Round towards +ve (i.e -3.9 rounds to -3, 4.2 rounds to 5)
• Simple rounding (i.e 2.4 rounds to 2, 2.5 rounds to 3)
• Banker's rounding (i.e. 2.4 rounds to 2, 2.5 rounds to 2, 2.6 rounds to 3, 3.5 rounds to 4)

I went further though, and made the calculator work on rational numbers, that means all of the maths is done with numerator and denominator, and only finally at the end does the division get done and rounding applied. This means that 1000/7*7 is 1000, unlike bc which makes it 994. minor optimisations for adding/subtracting with same denominator, and anything with denominator of 1 or a power of 10, making it simpler. For anything without division the maths does not even need a final divide.

I had a few silly bugs, one that fooled me is that zero is a special case, which I store as a magnitude of 0 and no digits, which is fine, but comparing numbers I started by comparing magnitude before comparing digits, so 0 looked like a bigger number than 0.1 as 0 was 0*10^0 and 0.1 was 1^10-1 and magnitude 0 is greater than magnitude -1. I had to add checks for comparison with 0.

However, overall, I am quite chuffed. Heck, I even asked it to work out 1e1000000000+1 and it just worked, that is a billion significant digits!

This is pure coding fun though. But may be useful - feel free to use it.

P.S. Someone asked about Banker's rounding, and I was sure I had blogged once, but cannot find it. Unlike that which I was taught at school, and every Casio (and other) calculator I had, rounding is not as simple as you think. I used to assume that a residual of exactly 0.5, or above, rounded up, and below 0.5 was down. That is what my calculator did. However, this creates a bias. Ideally you want to round an exact 0.5 residual down half the time and up half the time to remove bias. This could be random, but that is bad as you get results which are not reproducible. One simple way, bankers rounding, is to round 0.5 residual to nearest even number. Hence 0.5 rounds to 0, 1.5 rounds to 2, 2.5 rounds to 2, 3.5 rounds to 4, and so on.

Soft RS485 UART using interrupts on ESP8266

My non technical readers can skip this :-)

Background

As you will have noticed, I have made a complete access control and alarm system based on a Raspberry Pi but working over RS485 bus connections to standard Honeywell Galaxy parts like RIO, Max Readers, and Keypads.

As you will have noticed, I have now started making WiFi connected parts for the alarm system based on ESP8266 parts (ESP-01, ESP-12F, ESP-12S). These include a small ESP-01 based board with RS485 driver that fits inside a Galaxy keypad talking RS485 to the keypad.

As you may have noticed, I have NFC readers sorted and working with secure NXP MIFARE DESFire EV1 cards. These are way more secure than poxy proxy cards used by Max Readers.

Challenge

The concern some people have raised is the reliance on WiFi. It will do the job, but it is not that hard to disrupt externally. Now, much like using a screwdriver on a Max Reader, it will set off the alarm to disrupt it, but it is perhaps a tad easier. One has to consider if it is easier than simply forcing the door or breaking a window, which is always the benchmark for any security :-) But it is a concern.

So can I make the door controller, which does the secure cards, look like a Max Reader and do RS485 to a conventional Galaxy system? I think I can...

Approach

Initially, for the keypad system, I used the built in serial drivers on the ESP8266. These are good, interrupt driven, well supported, except...

The code on the ESP8266 using Arduino is the NONOS SDK, and means one thread for all. It can be delayed a bit for all sorts of reasons, even many milliseconds. For serial that is all handled behind the scenes on interrupts this is no problem, until is it RS485!

The problem is that for RS485 you have to enable and disable the line driver around your transmission. Initially I simply set a time to disable after poking the message in the Tx queue. But that could be delayed and mean after transmitting we held the line driven for too long.

The solution was to wait in the main loop task until sent, and then disable the driver. This is not nice, as any other operations of the device are delayed, but for the keypad driver application there is nothing else. So it works well.

This will not work for the card reader as it has to, well, talk to the card reader, and inputs, and outputs for the door control, and so on. Also, if I am a slave device (unlike the keypad where I am master) I am expected to answer polls promptly and not hang around for 10ms randomly.

My plan - interrupts! Make a soft serial RS485 UART on an ESP8266 to do the grunt in the background.

UARTs

A UART (Universal Asynchronous Receive/Transmit) is a device to send and receive old fashioned serial byte / character data. The concept dates back to what? the 50s or 60s. It is simple...

Each byte, or character, to be sent is coded as a series of 1 or 0 on a bus. What 1 and 0 mean in terms of current or voltage depends on the flavour of system (RS232, RS485, RS422, etc). Each bit time (1/Baud rate) the line has a 0 or a 1. The line is idle (1) and then the byte/character starts with a start bit (0), bits in the byte (low bit first), then any parity bit, and then stop bit (or bits) which are 1.

Obviously both ends have to agree the rate (Baud rate) for bits, and byte size, and parity present and if even or odd, and even number of stop bits.

The Galaxy uses 9600 Baud (so a bit every 104⅙µs), 8 bit data, no parity and one stop bit. Simples!

Sending is easy, you need some sort of timer, and you set 1 or 0 on the output to do the start bit, data bits, and stop bit. The Arduino ESP8266 SoftwareSerial sends in real time from the command using careful timers, it seems. I plan to use interrupts.

Receiving is a tad harder. You have to find the start bit for each byte, and then sample each bit, ideally in the middle. But once you have the bits you shift in the data bits and check the stop bit is 1 (else an error). Not that hard?

Edge interrupt on rx?

Interestingly the SoftwareSerial code simply interrupts on each edge and looks at time from last edge to work out the received data (I think). That is one approach but I decided to clock in bits on interrupt - I just need to sample mid bit somehow.

My initial approach was to set an interrupt on the falling edge of the data line to set the recurring timer half a bit later. However, this has all sorts of challenges, no least of which is race conditions on a regular per-bit interrupt and the edge interrupt. I did work, but not 100%. I am sure I could have solved the race conditions, but actually there is a good reason to stick to simple regular interrupts.

So the approach I took was just a regular interrupt, and this was 3 times per bit. This means ⅔ of the interrupts just count a sub bit and return in most cases, so very light weight in terms of time. But what it does mean is that I could in fact have multiple instances of RS485 busses running on this one interrupt if all the same Baud rate. I have not yet coded that though. It is, however, nice and simple and working for one instance well.

Why 3?

The 3 times per bit is simple. For Tx it makes no odds, skip 2 out of 3 interrupts and set the line to 1 or 0 for the next bit to send. For Rx, it is the same for sampling, except when waiting for start bit. If waiting for the start bit you check for the line going 0 every interrupt, and if it does set the sub bit counter to sample one interrupt later and then every 3 interrupts.

This is not as clean as an edge interrupt adjusting the timing to the microsecond, but it does mean that we can have sampling of the centre of the bit, ±⅙ of a bit which is more than adequate. Doing 2 times bit rate is no good as you could be sampling in the middle or right at the end. Hardware UARTs usually do 16x bit rates, but 3x is fine in my opinion.

This is how I have coded soft UARTs for over 20 years dating back to PIC16C84s, realy!

Messages

The way the RS485 bus works on Galaxy systems is that messages are sent as a sequence of bytes with no gaps, and then a gap - no length indicator in the message (but there is a checksum). So I coded the system to handle this, and even, if we are a slave, auto respond with current status response.

This allows the application to work on a message level, and if it is slow, we answer polls promptly anyway with current status. The driver also does the checksum.

Interrupts on ESP8266

Now the challenge is making it work on an ESP8266. They are rather special. They are a weird mix of old and new. The ESP-12S has 4M of flash! As an embedded controller that is huge! But it has around 80k of RAM (yes, k) and I had a BBC micro with more in the 80s (32k main and 64k second processor). The fact it does TCP at all is mental, but it does, and it does WiFi too.

The usual pressures on coding are complex - embedded systems have pressures on code space, RAM, execution time, power, all sorts.

In this case interrupts are special. The processor has a cache for the code, but this cannot be used by an ISR (presumably some interrupt process handles cache fails). This means the ISR has to be in RAM.

The trick is marking the code you use for the ISR with ICACHE_RAM_ATTR attribute. The RAM used is small, so not only does the ISR have to be fast it has to be small in code space too.

Thankfully a UART is simple code in terms of space and time. Small TARDIS code, even?

This worked, well, sort of, it worked mostly if started after about a second after boot. It turns out that using digitalRead, digitalWrite and pinMode is bad as these Arduino functions are not marked ICACHE_RAM_ATTR. Instead you need to find the underlying GPIO_REG_WRITE and GPIO_REG_READ functions. That then works reliably. The GPIO access is also faster, which is good, but very processor specific.

The result is working RS485, on interrupt, on an ESP8266!

I even added a clock track (blue on that trace) for debug - toggling an output for tx and rx clocking per bit.

It works!

This was deployed as new code to talk to the keypad, but means I can also act as a slave and pretend to be a Max Reader and provide a fob ID derived from the NFC card ID when checked using AES handshake.

It is also quite important as I am using the hardware serial at 115200 Baud to talk to the PN532 NFC card reader!

P.S. I have tested now as both master talking to Keypad, and as slave, pretending to be a Max Reader.

SVG, font metrics, am I being thick?

I really hope someone tells me I have missed the bleeding obvious for a change.

I think SVG lacks two key features.

1. The ability to set a maximin size on a line of text, i.e. so that if it works out longer it will be squashed in some way. There is a way to set a calculated size and have either the spacing or the spacing and glyphs adjusted to match that, but I cannot find a way to say "squash, but only if it would be too big".

2. The ability to take a block of text, give it a width (and height, why not) and line spacing and have it fill the box, breaking on spaces between words.

If there is svg magic (and I don't mean javascript to run on a web site that will do the business later), just plain svg that will easily convert to png or pdf, and still work, that is what I want.

The alternative, for which I am still massively failing to find a solution, is a simple Debian installed C code library (not C++ please) that allows me to ask "how wide is this string in this font". I found qfontmetrics which looks spot on, but is C++. Obviously, including emojis!

With this I can make the tools I current have which produce svg output (to later be converted to PDF or PNG, etc) work out when to set a line width, or break blocks of text, and work with normal svg.

All of this is a doddle in postscript, but unless someone has a patch for gs to handle utf-8 strings natively and handle a fuller character set with the likes of emojis, I need to move away from postscript, hence using svg. Just hitting this key stumbling block for now.

Update: Thanks for all the comments. I am looking at freetype at the moment. It looks like it has exactly what I need, but slightly struggling with Emojis and the like at the moment :-)

Update: Yes, FreeType has simple C functions to load a font/face and find the size of glyphs based on unicode characters, and simple to tell it does not have the character and use a noto/emoji font instead, etc. This should be simple to then work out line wrapping and sizes for SVG with tspan for emojis and the like. Now I know it is all possible and how to do it, I have to think about the project for which I was asking this - a new system for designing printed plastic cards...