This has been a hell of a week, and far more actual hard work than I am used too, and after it all (averaging over 15k steps a day) I have not even lost any weight!
The plan was simple, replace existing 7 door system, and PIRs and exit contacts, and so on. 28 modules in total. I have carefully made and tested all the modules and spares. All ready to go.
But I ended up making work for myself by tidying the wiring. The new system just needs power, so ended up with 6 battery boxes, and removing the old power and bus wiring. Much neater. I also needed to change the exits from rangers (that were unreliable) to new exit buttons, and hence also needed to change the break-glass boxes as they were integral. And I wanted to move the break-glasses further from the letter box (for obvious reasons). This meant running wires down the aluminium door frames on the front doors (to be tidy), which is easier said than door. I probably wasted a day on that (and I had some help). I can recommend NYLON CABLE & FLEX DRAW TAPE 30M (98FT) (screwfix 75807) for the job :-)
Finding the cable runs to bell boxes was also fun, and I gave up chasing wires to original PIR and door sensors and just ran new cables. Ceiling tiles for the win.
We actually have a mystery - the original alarm when we moved in was removed something like a decade ago, but the PIRs were still in place. When the new galaxy alarm was installed new PIRs were put in. I have actually put in new PIRs this time with integral ESP32 WiFi modules in the PIR. Part of the work was removing the galaxy system which replaced have already replaced what was there before, and removing all the old battery boxes. The original PIRs still have power and blink when people move? But there are *NO* battery boxes nor alarm panels in the building at all. I have literally removed them all, and the new battery boxes only go to my new kit. So WTAF. If I had time I'd have investigated further, but it is a mystery!
The result is it took a whole week, and I have to admit I was losing my enthusiasm somewhat during the week.
What did not help is finding some bugs and issues. I expected some, and a couple of small snags were simple, but the biggest issue was the wifi mesh being unstable. It was fine at the start but got worse during the week. I found issues with multiple wifi channels (2.4GHz) which I fixed with config changes on the APs. But I would find the mesh unstable. Reseting the wifi could fix, but not always, and changes to wifi config helped, but not in a concrete way. I wasted a lot of time on that. But ploughed on with the installed.
But Friday it was clear the problem was now intractable - the modules were all resetting every 30 seconds or so. The nice thing is the system was still usable. People in the office could come and go, as it resets very quickly and works autonomously. But it cannot work as an alarm if the mesh is not working.
It seems (and I have to do a lot more bench test and investigation to be sure) that the modules may be running out of memory when the mesh gets beyond around 12 units. This is very disappointing as I I can't see why the mesh would need much memory at all - each routing entry is a 6 byte MAC and very little else. It will be interesting to investigate.
Thankfully the site is two distinct adjacent units, so the fix was to make it two sites. That is a pain, as it means provisioning fobs for two sites, and if someone had not mistakenly ordered only 2k DESFire fobs, that would be easy, so new fobs were programmed, again! Just means slightly more admin. I cut down the sites slightly and ended with both sides with 12 units. When one had 13 it was not quite stable, but 12 seems fine. So we have a work around.
It may be that in longer term we need some modules using ESPs with extra RAM to act as root nodes, something to consider if I cannot track down the memory footprint problem.
Even so, the end result seems really good. Time will tell.
P.S. I have improved stability a lot with some tweaks to the way we work the mesh, but not solved the underlying issue. I suspect big step will be waiting for mesh code in ESP IDF to mature a little. Like cheese.