The issues I found in my RADIUS code were subtle. And were down to state machines and some assumptions in original design (valid at the time). The error needed a complex combination of enough customers on line and the right chances that one of them is a wholesale (L2TP relay) customer at the right point in time. It involved timing of the RADIUS and L2TP and thwarted all my off line testing.
I plugged several loopholes, but this is a fun bit of software design logic. I have made a number of changes that are "defensive coding"...
My older code made assumptions and was thwarted by changes due to the way the new RADIUS code worked that undermined those.
The new changes now are to make the code safe for those assumptions. Make the code not make them and work regardless of state change logic that I may dream up in the future.
It is like assuming that there are other developers on the project that are "out to get you" and making sure that nothing they can do will break things. The irony is where that other developer is another instance of yourself, a few years later, making new code for good reasons and forgetting the constraints on the old design.
The end result is code that is safer for future developments, which is good. It is technically less efficient, but I still find it hard to get core CPU usage over 1% on the most heavily loaded LNSs, so I really did not need to bother.
Anyway, end of the day, I have been snowed under. I am no longer writing more broken lines of code than working lines of code, and I have a bottle of SoCo that is a lot less full than it was...
Now I have to catch up! We are rolling out the updates. I have a huge list of stuff to do that I have been putting off. Some odd RADIUS handling on the servers (linux) I need to re-check. Summer time upsetting that! Much still to do.
When I am finally caught up I can progress the use of this new, well tested and robust, RADIUS code for VoIP server use, at last. I am several weeks behind my plan on this, but it is all progress.
The new VoIP system will, when finished, be ready to replace out core VoIP services. That will be another challenging time and lots of advance testing. It will create a new platform that can scale massively.
On top of all of that we finally have a new logo :-)
It would help if the cat did not leap over my desk, take out a mug of soup, and make my computer all sticky!