Fun day

Well, first off we decided to do a long standing change to swap aaisp.net.uk and aa.net.uk. That did not go well. More at http://aa.net.uk/news-2012-02-dns.html.

But other than that thing went well.

We worked out that normal BGP logic was not helping us on the route reflectors. If we had more than one external BGP box on the same subnet the next hop is on that subnet. If one box loses the link it still announces the subnet. That means a link failure could make a black hole. The answer is that all the edge routers need next-hop-self set to ensure we only route traffic to routers that have the working external BGP links. Fun!

That change worked like a charm - no issues - no dropped packets.

We also moved over one of our special wholesale customers which involved complex routing table logic and RADIUS changes, and, well, just worked.

We now have all transit and all peering and at least one wholesale handover on the new rack! Yay!

Next are the remaining special interconnects and moving the old BT host link over. They have their ISDN pair in now, so should be this week.

Basically going well.

And I do apologise for the DNS issues today, but aa.net.uk rules now.

1 comment:

  1. The bane of our sector :)

    No-one notices when migration plans go off without a hitch and are silky smooth.
    No-one notices when things "just work"
    No-one notices when nicely designed backup systems kick in when things do go wrong.

    We only notice when there's a problem :)

    Anyway, thanks for being upfront about the issues faced, caused and dealt with - as per usual :)


Comments are moderated purely to filter out obvious spam, but it means they may not show immediately.

ISO8601 is wasted

Why did we even bother? Why create ISO8601? A new API, new this year, as an industry standard, has JSON fields like this "nextAccessTim...