Friday, 13 January 2012

Bad luck to be superstitious

Well, it is, in that if you are not superstitious then the crap that happens is just crap that happens and not bad luck as such...

So, Friday 13th, and clearly the network switches in THN know they are about to be replaced next week as one of them starts going "iffy". Well that is the best explanation we have at present for flaky traffic between two boxes on the same switch this morning. As ever, a total failure would have everything falling back in seconds to the secondary systems, but "iffy" is the system engineers worst nightmare.

Of course it happens when we have a completely separate minor issue affecting incoming connection RADIUS on 20CN lines due to our favourite telco jumping the gun on a config change. Well, an issue that would be minor. Basically, it only affected new connections, a couple of lines, and would take us minutes to solve. It took longer because all hell broke loose when the switch went iffy, and as we dropped all sessions it meant nobody on 20CN could reconnect! Anyway, all sorted now.

The good news is the new rack is being kitted out - we have several gigabit fibres in already and a load more links to set up and move over, new switches, new routers and LNSs (FireBrick, of course). All moving to multiple gigabit operations hopefully by the end of the month.

Of course that is not the only "crap that happens", but that is a different story.

Happy Friday 13th.

No comments:

Post a Comment