Unintentional DoS attacks?

We have had some issues today, both last night, and for a few of hours during the day today on and off. It looked a bit like a denial of service (DoS) attack via LINX, but it seems was not actually intentional!

Basically, somehow, a major content provider with which we peer suddenly thought we were a transit route for a chunk of their traffic and flooded our peering and hammering some of our transit to get the traffic to its actual destination (another country even!).

We've been working with them to try and understand and fix this. We are very sure we are not announcing someone else's blocks by mistake to them. They confirmed they could not see any layer 3 route to us for the traffic. So nothing looks wrong! But they are sending the traffic and it was enough to cause packet loss on LINX for us. Best guess is a hardware issue on their router.

I'll update more on here when we get to the bottom of it - we have had to shut down the peering, meaning no diagnostics can really be done to find the underlying cause, which is a nuisance. At some point we have to re-establish peering and do testing to confirm if fixed.

Even with peering down, we are still seeing bursts of traffic, but dropping the peering has helped, oddly. With peering down, this means it is some sort of layer 2 (Ethernet) issue. At least, when there is traffic, we stand a chance of diagnosing the problem.

Whilst packet loss on a major external link is a problem, it does not usually have that much of an issue on normal access to web pages, email, etc. For a start, it is only the one link, and we have many. But it does have quite an impact on VoIP services which mostly go over peering links. We immediately redirected some of our VoIP routing to try and avoid this, but some calls were still via LINX and suffering break up in audio.

Obviously we need to have a serious look at ways we can cater for this sort of issue in future - ultimately we have very little control of things going wrong "in the Internet", and it is almost impossible to pre-plan every possible contingency. None the less, we will try and learn from this.

I picked a good day to be off sick!

Update: I would like to thank my engineering team (Paul, Andrew, Jimi) for working on this all day and on in to the evening in their own time, and the guys form LINX as well. It seems the exchange itself (LINX) is not at fault in any way, which is good news. Some additional steps with the route server do seem to have stopped the bursts of traffic as of around 5pm Friday, and we intend to leave things like this until the peer can investigate further.

Update: When another LINX peer suffering the same issue contacted the offending peer this morning (Saturday), they immediately reset the card facing LINX and fixed it. One wonders why they would not do that when we reported it yesterday, shame. It does however confirm this was not "just A&A" being affected.

Knowing your customer

Obviously, as a business, we have to know our customer. To be more specific we have to know some particular details about our customer.

This is not always true - a shop selling a mars bar does not have to know anything of their customer, and neither does someone running a vending machine.

However, we have to know some details, though in theory we could provide some services where we do not need to.

For a start we have to know what that customer's legal entity is, e.g. a limited company, or a sole trader, or so on. We also have to know a service address. The reason for these, apart from anything else, is so that we are able to enforce the contract (to take someone to court if necessary) for things like non-payment of bills. This is the same for  pretty much any businesses offering any sort of ongoing service.

But we also have to know a few other things, and we do ask some of these when people order. Here are just a few examples of what and why...

Under 18

If someone is under 18 they are a minor, and there are a whole load of things that impact the contract we can make with a minor and the extent that we can enforce such a contract. We have actually made a decision for the services we offer not to deal with minors (sorry). We do ask though, as lying about that would be fraud. If we did not ask, it would be tough luck on us.

Consumer

The rules on contracts, in terms of what is allowed, and some of the steps we have take in giving notice of contract terms and getting agreement, are impacted by whether someone is a consumer or a business. Consumers have extra rights for cancelling contracts that have not yet been provided. There are also differences in terms of late payment penalties that can be applied. Of course it is possible for someone to be an individual, but buying as a sole trader in a business and so not be a consumer!

Small business (10 or fewer individuals work for the business, paid or volunteer)

This is a complicated one, as any small business can change to have greater or fewer staff over time. This one impacts a couple of things - one is the ability to make use of ADR (Alternative Dispute Resolution). Someone with more than 10 people doing work for them cannot use ADR.

This is also part of the new GC22 migration. Anyone with more that 10 people cannot expect to be able to migrate broadband or fixed line telephone services (though in practice they may be able to).

Communications provider

This is complicated!

For Digital Economy Act, being a Communications Provider means you are exempt from copyright notices. It is possible to be a Communications Provider and not be an Internet Service Provider, even, so avoiding other obligations of the Digital Economy Act. However, for this, the definition of Communications Provider comes from the Communications Act, and seems to me to cover anyone with a network switch or router at home would meet the definition.

For ADR, a Communications Provider cannot use ADR, and that definition is from the Communications Act too.

For GC22 migrations of broadband or fixed line, the definition is different! It is someone providing DSL (including FTTC) or fixed line telephone services, so someone with a switch at home is not a Communications Provider under that definition.

This makes "Knowing your customer" slightly more complex as someone could (a) be a Communications Provider under the Communications Act (and so for DEA and ADR) reasons, but could be (b) a Communications Provider under OFCOM GC22, quite independently (one, or the other or both). You then have that for DEA one has to be "buying service as a Communications Provider" so it is not simply whether you are one, but if you are buying as one!

At the moment we ask if someone is a Communications Provider under the Communications Act, but we may have to refine that question slightly!

OFCOM confirm...

GC22 - the new migration rules that replace MAC codes for broadband, simply do not apply to lines for customers with more than 10 people working for them or which are communications providers.

They suggest such migrations are best managed by a cease of old services and re-provide of new services with some overlap (and notable cost).

In some ways this is sensible, in some it is not.

They do not confirm who is responsible for determining that an end user has more than 10 people working for it, and who is liable if a telco acts on statements made by the customer on such things or what happens if number of staff changes over time.

They do not confirm the reason code to give when a losing ISP refuses a migration because its customer is more than 10 staff and hence "GC22 does not apply".

I have to feel that OFCOM have not thought this through.

P.S. - this is not just broadband, but fixed line telephone services as well!

Training is hard work

I run a couple of regular courses, one of them is the FireBrick course (two day), and one is a crash course on IP (one day).

Both are hard work - they mean a lot of standing up in front of a room of people with a projector and a whiteboard marker and a lot of talking.

I hope I do a good job - I get a lot of good feedback on this - but it really is a hard day's work from which I always feel knackered. The course this week was even more so as I am still recovering from being ill, and I appreciate the help from Alex on several parts of the course.

The IP course is perhaps the easier one. It is a crash course on IP done over one day - from the basics of ethernet packets, MAC addresses, ARP and ND, and building up from there all the way to how TCP works and email and web browsers. It is basically working through in a logical order layer by layer. It makes sense as each layer builds on the knowledge we have built up the layer before. It is a lot for one day, but the material has not had to change for a long time.

The FireBrick course is a lot more of a challenge! For a start it is always changing - every time we have new features and the course grows.

But even then, every time we run it we find ways to make things better. One of the issues is there is no logical order for the material - it is not like IP which is layer on layer - it has so many features, and whilst some are more fundamental (profiles, logging), so many others are independent for each other and so have no meaningful order. Every time we add a new feature we have to work out where to slot that in to the course.

I think we have a pretty good formulae now, but it is tricky to fit in two days and I can see this becoming a three day course soon. Right now we have to be selective about which areas we do with a practical and more detail and which we explain and continue on to next topic.

This week was harder as we had a mix of ISPs and end-user or IT company types and had to try and cover both areas. I think we managed a good mix.

Even so, we are considering whether some sort of SME IPv6 course is worth running - but I suspect we have to cover a lot more "windows" than I would wish to go near.

In short - I have even more respect for professional teachers in schools - this really is hard work!

Building a new community

I ran a FireBrick course today and met some very nice chaps from a small ISP who deal with the likes of BT and Talk Talk just like we do. (Freudian slip, I initially typed "lies" not "likes" in that!).

They were very interested in our efforts to start a Wholesale Broadband Buyers Forum  (wbbf.uk). They have even considered doing the same themselves. I was really pleased at how much common ground we had.

There are a lot of small (and even larger) ISPs that face issues with carriers on a daily basis, and we all want to work together and with carriers to solve these problems so that we can provide the best possible service to our customers. We really do not want to be fighting BT or anyone - but we need a way of working together that means we do not have to fight, and that is not how it is right now.

One of the biggest issues which has plagued all ISPs for over a decade now is BT SFI (Special Faults Investigation). It is a problem in the first place, as it was created on broken foundations so solve a problem that should not have existed. BT, in classic big company style, do not fix the underlying issues but pile on top new layers of bureaucracy and incompetence. The latest is that SFI disputes are such an issue with ISPs that they have a new process for management of disputes (rather than fix the reason for the disputes in the first place).

I had a meeting many years ago face to face with people that dreamed up SFIs, and in that meeting they said something along the lines that an engineer visit that did not find a fault was chargeable anyway so making an SFI added extra value by being able to check end user wiring and stuff as well for that fee. I pointed out that this was in fact untrue - there was never any charge for such a failed engineer visit - I even challenged him to show me where in the contract and price list (and indeed any bill we had ever had) where there was such a charge. He failed to do so - so the fundamental premise of SFI as adding benefit to customers who already paid for the failed visit was actually flawed. Even so, they did not change their ways. This is typical and key here - they created SFI because not checking user equipment was a waste of the customers money on a pointless engineer visit, when in fact the customer was not charged for such visits - the whole logic for creating the concept of an SFI visit was totally flawed and they refused to admit it.

Over the years they have changed the "service" of SFI one step at a time as we find ways to counter it. First it was "charge for work beyond NTE" and we said "do no work beyond NTE". Then it was "work beyond NTE includes visual inspection of end user equipment" so we had people hide their kit when engineers visited - they still tried to charge for a visual inspection, but of what?! Then it was "Fault not in BT so has to be end user equipment at fault" so we sent end users BT branded modem/routers purchased from BT and within warranty as the only equipment on site, making them liable for the line being faulty or liable for saying their supplied equipment was faulty.

The battle goes on to this day with each new stage being a new battle.

The latest two crazy steps are: (1) That a series of engineer visits where the last one finds and fixes a fault in BT requires payment for the previous (presumably incompetent) engineer visits that did not find or fix the fault, and (2) that all visits where the line tests to an analogue phone spec on arrival shall be chargeable even where they expect us to book an engineer for a PPP fault, or a BRAS fault or a BT modem profile fault all of which happen on a line that is good at a copper test level.

With Talk Talk they create another level of indirection and similar stupidity. At least TT are only at the stage of "SFI" being a product and us saying no thanks, we do not wish to buy that product, now fix the broadband we have already bought. This is a stage we had with BT many years ago.

Somehow we need to find a way to solve this - to get BT and carriers to actually work with us to fix faults, and not create a machine for creating and disputing charges. We also need to work on other things like the cost of backhaul bandwidth, the stupidity of wires only FTTC (leading to yet more SFI issues), issues over backhaul congestion within the network, and stupidities of ordering and faults systems not working properly. We even have to consider OFCOM related stupidities like the latest broadband line migration systems.

So the idea of WBBF is to allow this - to bring together ISPs using BT and TT, and solve these issues. We are gathering momentum now.

The next step may have to be a physical event - in London (pretty much has to be, sorry), and with as much chance to talk as possible (so no pub/restaurant with music or entertainers). Something with food and drink - beers with peers sort of thing - and a couple of presentations and question sessions. Mostly about drinking, eating, and talking.

The trick is very much not to be another LINX, LONAP, UKNOF type of thing (all of which are good, but not what we are aiming for) - but a place to allow as much networking and talking as possible. I did wonder if there is any venue with a big anechoic chamber (with booze and food) - I'd pay to use that!

The IT crowd: The next big challenge for IPv6

IPv6 (Internet Protocol version 6) is the current version of Internet Protocol - whoopty doo! What does that mean?

It means that the way computers talk to each other is changing slightly. From the point of view of people using computers it makes no difference. Most people have no interest in how computers communicate.

The problem is that some times, some people, need to do something about it, and those people do not always realise this.

Basically, what this is about, is upgrading the Internet to a new version, and that means all of the bits of the Internet changing as a result. People are used to upgrades on their computers and laptops and phones and even TV sets these days, it is pretty routine and seems simple enough - but sadly it is not quite that simple...

In order for the Internet to be properly upgraded it means everything on the Internet using the new system. But while we are getting there -  everything has to use both the new and old system at the same time. Only once everything is caught up can we stop using the old system (called IPv4).

The issue is that, even now, the old system (IPv4) pretty much works well enough. But we can't wait for it to stop working properly before we act. Long before that point we have to have everything using the new version as well, and it takes a lot of time to change things. So lots of people, just like me, are pushing and nagging for this to happen before it is too late.

So where are we at now?

• Pretty much all of the home computer systems that matter are already upgraded or can be with an automatic upgrade some time. There will be a few things that don't get upgraded but we can generally work around them. The main things are devices that you use to communicate with the Internet, so computer/PC, laptop, iPad, phone, that sort of thing. They all work with the new version, which is good news.
• Some Internet providers (like A&A) are already making sure that your Internet connection is upgraded already, and have done for many years, but the big players like Sky, BT, Virgin, and so on will start upgrading people's Internet connection soon (probably this year). That will mean that most homes will be using the new version where they can, which is good news.
• The big companies that use the Internet to provide services, like FaceBook, Google, Netflix, and so on, have already upgraded - this means that people at home, with an upgraded Internet provider, accessing these services, are using the new version of Internet.

This is all very good news, and it is finally happening. Finally the old version can be seen as the poor man's, old fashioned, Internet and pressure applied for the last remnants to finally start to die out, one hopes.

But there is one stumbling block - SME (Small//Medium businesses).

The larger businesses have generally had a plan for many years and been upgrading their systems, but smaller businesses have not. It is not surprising in many ways - shit still works - if it aint broke, don't fix it - and this costs money to even think about or plan let alone make happen.

The problem is that small businesses are not using Internet like home users. For a home user, typically, you have some magic box from your Internet provider, and you have stuff like an iPad or PC, and they just work somehow - you do not have to think about it. That is good news, and a real endorsement of the hard work put in by a lot of people to make this work so well.

With a smaller business you actually think about how Internet addresses work in your company. Which devices have what addresses, and how multiple sites link together. How computers log access. How visitors get restricted access. Things like firewalls and address allocation policies. All of this works because people in a small business - the IT crowd - understand how the old version of Internet works, and do not necessarily understand how the new system works.

Without some mandate from above they have no interest in making their life difficult by taking on such a complex project - not while things still work properly the old way.

So here lies the challenge - how do we get SMEs, and their IT departments, to embrace the new system - to realise the power of IPv6, or at the very least realise the limited lifetime of IPv4 and everything still working properly the old way?

Talk Talk at it now

I am really annoyed at BT trying to charge for SFI (Special Fault Investigation) visits where the engineer has either been incompetent and not found the fault (shown to exist by a subsequent engineer), or has found and fixed a fault within the BT network (shown by engineers notes saying he did work, and the fault going away as a result).

We are pretty much at the stage of going to court over it - we have solicitors involved, and the next step if they do not back down is we take them to court.

But now Talk Talk at at it - they simply send the SFI disputes to BT who send them back saying "The initial test passed and the engineer carried out all the required checks for a base module, therefore the charge is valid. Thank you for your enquiry."

Talk Talk even list SFI as a "product" they sell. Obviously we have no interest in buying such a product from them. We buy broadband, and if that is not working we require it to be fixed at no extra cost. We have made this clear to them, so we await details of how they plan to fix faults without using this "SFI product".

We'll see how it goes, but it may end up in court with Talk Talk at this rate.

Why is it so damn hard for BT and TT to understand that they have to fix faults in the services they sell.