Monday, 13 February 2012

Fun day

Well, first off we decided to do a long standing change to swap and That did not go well. More at

But other than that thing went well.

We worked out that normal BGP logic was not helping us on the route reflectors. If we had more than one external BGP box on the same subnet the next hop is on that subnet. If one box loses the link it still announces the subnet. That means a link failure could make a black hole. The answer is that all the edge routers need next-hop-self set to ensure we only route traffic to routers that have the working external BGP links. Fun!

That change worked like a charm - no issues - no dropped packets.

We also moved over one of our special wholesale customers which involved complex routing table logic and RADIUS changes, and, well, just worked.

We now have all transit and all peering and at least one wholesale handover on the new rack! Yay!

Next are the remaining special interconnects and moving the old BT host link over. They have their ISDN pair in now, so should be this week.

Basically going well.

And I do apologise for the DNS issues today, but rules now.

1 comment:

  1. The bane of our sector :)

    No-one notices when migration plans go off without a hitch and are silky smooth.
    No-one notices when things "just work"
    No-one notices when nicely designed backup systems kick in when things do go wrong.

    We only notice when there's a problem :)

    Anyway, thanks for being upfront about the issues faced, caused and dealt with - as per usual :)