Calling all small ISPs: Fixing broadband faults

There is a big problem in the industry with broadband faults and getting them fixed. Some of this has leaked out to end users who get charged to fix broadband faults, but mostly it is the smaller ISPs buying from wholesale back-haul providers that are suffering and have been for years.

This video below explains it in terms of BT Wholesale, as of today. But this is just the current ploy with BTW. The issues exists with other carriers, and we even see moves to actually try and formally charge to fix faults for which the carrier is responsible (is that even legal?).

It has been going on for years, and has changed over time - with some crazy inventive logic like "engineer charge applies if he visually inspects the end user equipment" (i.e. looks at the router) and charge applies even if router is actually hidden away in a cupboard - that is how crazy it has been. You'll see from the video how bad it is getting now. Next year it will be some new stupidity, I am sure.

This needs sorting, and it may have to be sorted in a court to finally fix it. Like many smaller ISPs we challenge these invalid charges, and we always win. But if it went to court properly then every ISP ever charged incorrectly to fix faults going back at least 6 years (maybe more) could reclaim what they paid. I'm not after back dating the solution (as we have always been successful in every dispute of charges to fix faults, eventually) but it would be a blow if any case went to court. What I would like is this whole issue knocked on the head and sorted once and for all - each party in the the whole broadband Internet Access supply chain taking responsibility (including financial responsibility) for the part of the service they provide, simple as that. Not unfair or unreasonable - simple.

We have created a Facebook group called "Wholesale Broadband Buyers Forum". If you are an ISP buying from a wholesale provider, please apply to join. We want to encourage an open and frank discussion with you [Chatham House rule]. We want to solve the issue of paying to fix faults. We are happy to share arguments and techniques to get disputes resolved. We want to make working with back-haul carriers work for us, you, and them. We want to save everyone the hassle of these disputes. Some of this will mean wholesale carriers changing their ways and taking proper responsibility. Some will mean Openreach taking some responsibility. Some may even mean OFCOM and political public policy issues if that is what it takes. But it needs fixing or else broadband in the UK is doomed.

We are after people happy to engage in debate, and I cannot rule out meetings in pubs even...

Watch the video, and if you are a UK small ISP using carriers like BT Wholesale, Talk Talk Business, Vodafone, or others, please do apply to join the Facebook group.



P.S. we did try this before, with an actual (mutual) company and web site, but to be honest Facebook works well as a forum and means to discuss such things, so we are trying again. Thank you for your support.

Teaching us to suck eggs? BT?

We have a customer on a fibre to the cabinet (FTTC) service which has packet loss.

The red is loss, as measured by one second LCP echoes over the PPP link, and is often over 5%. Levels of random packet like this are severely impacting his ability to make use of the service.

The loss started at the exact moment BT did work on the circuit due to a major outage, so is clearly related.

This is the line before the outage, and you do not get much clearer than that - no loss. Same as every day before :-

This is the day the line came back after their major outage (which lasted two days) :-

And this is the next day, which looks much like every day since :-

It does not take a rocket scientist to see there is a problem there - periods of around 5% loss, sometimes more, most of the day, every day, since the outage.

And yes, that is start of OCTOBER 2016! BT have failed to fix the fault for that long!

Today we got this, and I am almost at a loss for words! Talk about teaching us to suck eggs!

Is the customer using a VPN? Data is transmitted in discrete units known as packets. When a network server is overloaded, these can get discarded. 
This is known as packet loss and results in slow loading game dynamics and graphics, or the unsatisfactory performance of a VPN connection.
In such circumstances, there isn't much which can be done to improve matters, as the cause is not associated with your PC or broadband service.

I'm really not happy about this, but the "there isn't much which can be done to improve matters" is just shocking. We have asked BT to confirm if they are stating, officially, that 5% loss on an idle line is considered "acceptable" for a GEA/FTTC service - we await the response.

They even go on to say :-

In the meantime can you ask the customer to run some traceroute and provide and this hopefully will aid us in seeing where in the network  the packet loss is occurring.
SPs engineers can use a "wire shark" which can detect packet loss at points in network.

This is after explaining that we can see the loss at the LCP level on the PPP link and providing access to and copies of graphs showing the loss over and over again! It is like dealing with Dory to find a fault called Nemo. We keep having to repeat ourselves.

There is one other small snag.

We are all used to the notion of "fibre" broadband not actually being "fibre" which is why this is "Fibre to the cabinet". BT sell this to us as "Fibre to the cabinet" and call it FTTC. It turns out this line is in fact "Microwave to the cabinet". A good idea, normally, but not as described, and clearly beyond BT to actually understand and fix.

This just highlights the problem with a clear definition of the service: We need a clear specification of levels of idle/random packet loss, idle latency and jitter, reliability/resyncs, min sync speeds up and down, and even throughout before loss/latency starts. Without these you can literally spend months bashing your head against a brick wall and having engineer after engineer sent (each potentially costing around £200).

P.S. Today we get "Openreach have completed the engineering checks on the Radio back-haul and discovered some issues with the Link.". Well, yes, we have said this from day one (113 days ago). How bloody annoying. Maybe we finally get it fixed now.

P.P.S. After 120 days they finally fix the microwave link and the line is clean. So annoying.

Brick wall BT

Hurry up and build that wall, Trump, I want to bang my head against it as it would be easier than BT...

So, simple BT fault, suspected issue on fibre back-haul from an FTTC cab, started after MSO, so probably damaged to dirty fibre. Not a complicated fault at all.

Symptoms are idle levels of packet loss all day, sometimes peaking to 20%, that is shitty!

The dialogue sort of goes back and forth, but we finally go somewhere, after escalation, with a clear statement from BT:

'We have run diagnostics and can see numerous code violations which will require an engineer to liaise with 2nd line DCoE if the fibre escalation team are unable to assist.'

That is pretty clear, so we ask them to fix it.

• Us: Please fix the fault.
• BT: You need to book an SFI
• Us: An SFI is an optional service to check the metallic path to SIN349, nothing wrong with the metallic path here, so silly, just fix it
• BT: No, you have to book an SFI, that is the process
• Us: OK, what is the process for fixing this fault without ordering an optional service.
• BT: There is no other process
• Us: Are you really saying BT have no process to fix this fault without ordering an optional extra service - if so that is breach of contract as contract says you will investigate and fix faults
• BT: We don't see a BT fault
• Us: OK, so you are saying a line with constant loss peaking at 20% is acceptable, is that the formal standard BT work to?
• Us: What actual investigation did you do, and to what standard or reference did you test the line so as to decide there was no fault?
• BT: If you want us to investigate the fault report you raised further then please book an SFI
• Us: No, the contract says you will investigate and fix faults, are you refusing to?
• Us: Repeat of questions about this being "not faulty" at 20% loss?
• …

I'll add some more as it goes. But it goes round in circles.

Either BT

(a) consider this to be acceptable level of service, in which case, especially with ideas of universal broadband service obligations and automatic compensation, we need to take this up with OFCOM, BIS, and the Digital Economy Bill parliamentary committee as a matter of urgency.

(b) accept that this is a fault and BT are breaking contract by refusing to investigate and fix it.

BT just need to pick one and stick with that story to the end. Either that or just damn well fix the fault!

Why OFCOM's ruling on BT charging Talk Talk SFI and TRC charges does not help most ISPs

As reported by ispreview, Talk Talk made a complaint about the amount BT plc t/a Openreach charged for Special Faults Investigation (SFI) and Time Related Charges (TRC). They argued that the charges were not cost based and much of their complaints are upheld by OFCOM. There is now much ranting over back-dating and refunding, and so on.

As you know SFI charges are a major problems for most ISPs. The costs, of there order of £160+VAT per visit, are massively more than the monthly, or even annual, profit from selling a broadband line in most cases. Either ISPs get stung or end users get stung, even when the work done was fixing a broadband fault, something that we should not pay extra for!

To explain the problem you have to understand the layers of services provided. I'll explain for ADSL as this is where it is most relevant.

1. BT plc t/a Openreach sell wires in the ground, a metallic path service. They are not selling broadband.

2. BT plc t/a BT Wholesale buy the metallic path from BT plc t/a Openreach that connects a home or office to the exchange; their own equipment in the exchange (DSLAM); their own back-haul across the country; their own BRAS equipment; and routers and links to ISPs. They sell broadband. Talk Talk Business do the same as BT plc t/a BT Wholesale in this respect.

3. Companies like A&A buy broadband from BT plc t/a BT Wholesaled Talk Talk Business; have our own routers; DNS servers; and make use of transit and peering and equipment in data centres. We sell an Internet Access Service.

The complaint Talk Talk made is against BT plc t/a Openreach, and is only about the price. It is sensible for BT plc t/a Openreach to sell a service (SFI) that finds and fixes broadband faults as only their engineers can work on the network. It is sensible for BT plc t/a Openreach to charge for that, because it is over and above the metallic path they sell.

The issue we have, repeatedly, with BT plc t/a BT Wholesale and Talk Talk Business is that we have no interest in buying an SFI service. We already buy, and pay for, working, broadband and any work to fix that broadband is the responsibility of BT plc t/a BT Wholesale or Talk Talk Business. The fact that BT plc t/a BT Wholesale or Talk Talk Business have to pay BT plc t/a Openreach to find and fix broadband faults is not our problem, any more than the fact they may have to pay CISCO engineers to fix a BGP router in their network.

So our gripe has never been with BT plc t/a Openreach.

This ruling may make some SFI visits cheaper but as we should never pay for an SFI visit ever, and it is not a service we want to buy, all this ruling will do is make the amount we dispute every month slightly smaller. It won't solve anything useful for us. Sorry.

Even so, well done to Talk Talk on this - it will reduce their costs which is good news.

BT refusing to fix faults

Once again we are at the point of BT refusing to fix a fault unless we book a Special Faults Investigation (SFI) engineer.

This happens every few years - BT keep juggling their definition of fault repair in various ways, trying to somehow make it a chargeable service even though it is clearly part of the broadband services we buy that they investigate and fix faults.

The latest fiasco, as I have blogged before, is BT plc t/a BT Wholesale have stated that SFI is an optional service provided by BT plc t/a Openreach which they make available to us. They have gone on to say that an SFI engineers sole job (though he may choose to do more) is to test the line to SIN349, and charge us if the line meets SIN349. He does have the job to fix the line if it does not meet SIN 349 though.

SIN 349 is the technical spec for the copper pair for telephone use - it is not even a spec for broadband, so such a service makes no sense as something we would every want to buy. Also, we can test a line to SIN 349 for free from the exchange end tests and it would be unusual for these to disagree with a test on site by an engineer.

The problem is that BT have no process for actually fixing broadband faults - it is either a phone line fault and fixed by the line provider part of BT, or it is a broadband fault at which point they offer the option of a pointless SFI service that will only test/fix a phone line fault rather than any means to fix a broadband fault.

So we have the argument over and over again.

What would be especially amusing this time, if it was not causing delay and inconvenience for us and our customer, is that BT are insisting we book an SFI engineer for a PPP fault on an FTTP service.

FTTP is fibre to the premises. I don't mean like "Virgin fibre optic cable", I mean actual real fibre optic cable that actually goes to the house. Not a copper pair. So insisting we send someone to test the  "copper pair" to SIN349 is just farcical.

I do despair at BT some times.

P.S. 24 hours of nothing happening, and finally some progress.

P.P.S. They finally got an engineer out, changed the ONT and upgraded its software and then cleared the fault as no BT fault - I bet we get charged! Good news is customer is on line.

Talk Talk (wholesale) once again do not understand what they sell

They sell us broadband, not a copper pair.

They sell a service with a demarcation point of a master socket. That means they are responsible for everything that side of the demarcation point.

Sometimes there will be a fault in some part of what they sell us that means the broadband service is not working. They may have to pay someone to fix that fault.

Yet again, like BT, they seem to think that just because they have to pay a subcontractor to fix some aspect of the service they are providing, then they can expect us to reimburse them for that subcontractor's charges.

They even sent some e-learning Q&A that tries to ram this home, down right condescending and offensive if you ask me.

I sent them some new Q&A to try and explain :-

Q1. The fault turned out to be in the BRAS. Talk Talk had to engage a subcontractor to replace a card in the BRAS. Is this chargeable.

A. No - because the work was Talk Talk's side of the demarcation point for the broadband service they provide, it is their responsibility to fix the service when it breaks at their cost.

Q2. The fault turned out to be in the DSLAM. Talk Talk had to engage a subcontractor to replace a card in the DSLAM. Is this chargeable.

A. No - because the work was Talk Talk's side of the demarcation point for the broadband service they provide, it is their responsibility to fix the service when it breaks at their cost.

Q3. The fault turned out to be in the exchange jumpering to the DSLAM. Talk Talk had to engage a subcontractor to replace the faulty exchange jumpering. Is this chargeable.

A. No - because the work was Talk Talk's side of the demarcation point for the broadband service they provide, it is their responsibility to fix the service when it breaks at their cost.

Q4. The fault turned out to be in the copper pair between the exchange and end-user premises, it also affected telephone service (i.e. did not meet SIN349). Talk Talk had to engage a subcontractor to fix the copper pair. Is this chargeable.

A. No - because the work was Talk Talk's side of the demarcation point for the broadband service they provide, it is their responsibility to fix the service when it breaks at their cost.

Q5. The fault turned out to be in the copper pair between the exchange and end-user premises, but it did not affect telephone service (i.e. did meet SIN349). Talk Talk had to engage a subcontractor to fix the copper pair. Is this chargeable.

A. No - because the work was Talk Talk's side of the demarcation point for the broadband service they provide, it is their responsibility to fix the service when it breaks at their cost.

Q6. The fault turned out to be in the end-user router/modem. It seems that a replacement router/modem had not been tried. Talk Talk had to engage a subcontractor to investigate. Is this chargeable.

A. Yes - because the work was not on Talk Talk's side of the demarcation point for the broadband service they provide. It shows that the ISP did not carry our the agreed steps to eliminate the end-user router/modem, so they have to pay, as a penalty, the cost of engaging the subcontractor.

We fixed your FTTC - that'll be £25,200.00 please

I go on about SFI charges, but the latest scam is BT charging "Time Related Charges" for work (mainly on FTTC line faults).

We had a customer with a fault on an FTTC line, constantly dropping, many times a day. We had, of course, replaced all of the equipment and had it connected to the master socket test port (so no extension wiring), so proving the issue was with BT.

BT's tests said: "Impairment in copper joint detected most likely in local network. Please continue to submit a trouble report" so we reported a fault. BT then asked us to "arrange an appointment" (not, as is more usual, a case of saying there is nothing wrong and "offering" an SFI service).

So, at BT's request, we arranged an appointment, and the engineer went out. The line was off for around an hour while he worked on it. Now it is all working - no more drops. Thank you BT.

To our surprise we now have a bill from BT for "Time Related Charges" for this fault report, as BT had closed it as "Customers Equipment, Error or Misoperation;Fault found on customer sited non BT maintained equipment". Somewhat odd as BT's own tests said to report a fault, and we had not approved any "Time Related Charges", and we had already eliminated all customer equipment (the same equipment still in use and working). In fact it was BT who asked us to report a fault and BT who asked us for an appointment. They took this upon themselves all the way.

This is, in itself, typical of the hassle we have every day from BT.

Except this time BT have charged for 350 hours of time related charges on the fault. The entire fault report end to end was only 7 days (168 hours), most of which was waiting for BT to actually go out and do something. From point of no return at start of day, to closed job, was under 7 hours. They would have had to have had 50 engineers working for that whole 7 hour period to justify this charge!

The charge for this one fault repair?

£25,200.00

And BT are actually meant to be double checking our bills before we get them because of the number of incorrect charges. Odd for that one to have slipped through. Well done BT.

Needless to say that this will be added to the disputes!

P.S. It is probably worth explaining one of the underlying issues here. TRCs are for engineering work. Now, either the engineer did work on BT's side of the NTE (to fix a fault), in which case we do not have to pay, or they did work on the customers side of the NTE, in which case we did not request or agree the work, so again, we do not have to pay. Unless we specifically order work to be done, and pre-agree a TRC budget, and it is not work to fix a BT fault, we should never get a TRC charge whether it is £252 or £25,200.

P.P.S. Three FTTC faults on the bill with TRCs total £39,744.00

Right When Tested

We had an interesting fault on a line in Nottingham last week, and for a change I have something nice to say about BT!

The first engineers decided it was a REIN issue (Random Electrical Impulse Noise), which is basically some sort of interference. So an engineer was dispatched to investigate the REIN issue. Hw spent some time testing at the exchange and cabinet and DP and tracked down the cause, which was, it turns out, the lift in the flats.

This is not the first time we have seen a fault like this, and the engineer said he had seen cases like this himself. He then called us to confirm in detail what the issue was and that he had found it. The end user is contacting the building maintenance people now to sort it.

But this means it was correctly diagnosed as a REIN issue, and BT engineers actually tracked it down to a cause which outside our customer's premises, and BT's side of the master socket, but not their fault as such.

So well done to the competent engineers on the ground - many of them are good and technically able.

Of course, the rest of BT lets that down, sending the fault back as "Right When Tested; IP/SP Network Connection;Line tested ok by BT engineer at customer premises".

Well, actually he demonstrated it failing at the end user premises, and explained it to the end user in detail as well as calling us (their customer). It was wrong when tested, ie. breaks every time someone uses the lift. Also, it was definitely not the "IP/SP Network Connection", we did not cause this.

So once again it looks like we will have to spend time disputing a charge for an engineer.

What is a broadband fault?

Exterminate the orc?

Continuing a theme here, but my mate Mike came over. One of the games he is quite good at is playing devil's advocate, rather well, and he actually made some really good arguments on the issue of SFI charges and faults. I'd like to thank him for that, oh, and for the dalek he brought over to surprise us!

The crux of his point is that the very definition of a broadband service is that it is rate adaptive and so it is a service that will try and get the best performance on the voice grade copper pair to which it is attached. It is possible for the characteristics of a voice grade copper pair to be so bad (very long line) that it cannot even do broadband, and that would be tough. That basically the performance of the broadband line is kind of whatever it is. Indeed, a comment on my previous post about TalkTalk was raising this very point.

He has a very good point, and it has allowed me to hone the arguments I can use against such points with the likes of BT and TalkTalk.

Yes, the broadband is what it is, "as well as it can do" on the voice grade copper pair to which it is attached. We do accept that to some extent. There are caveats - in that the copper pair will have a forecast speed, and if the line is way out of that, then that probably means something is wrong. But in general, you get what you get. However, this relates to the rate of the broadband, as it is rate adaptive.

I would argue that there is a huge difference between a line that is a low rate on a line because it is all the line can do, even if that means the rate has reduced a bit over time due to increases in cross talk from other lines and so on, and an actual fault.

The issue comes down to how you define a fault in the broadband. This is different to how you define a fault in the voice grade copper pair.

One of the key aspects in defining a fault - perhaps the key aspect - is that a fault can be fixed. It is a condition that is the result of some aspect of the copper pair which is not as good as it could be giving the length and routing of the pair. Faults could be down to poor joints, corrosion, water ingress, poor insulation, electrical contacts, degraded materials. It could be fixed by using a different pair, or repairing joints or equipment.

So how do you spot a fault? Well, BT plc t/a BT Wholesale actually defined one way of doing this. They monitor the line in the first 10 days and find the maximum stable rate for the line, and set a fault threshold rate (FTR) for the line based on that (with a percentage taken off). This defines a sync speed below which the line is considered to have a broadband fault. That is excellent as it gives a crystal clear metric which we can all agree defines a fault. TalkTalk don't do this, but it would not be hard for us to do the same based on sync speed history and agree a metric with them.

Another way to measure a fault would be to consider that the line loses sync lots of times. To be honest, once a day is too often. Normal lines stay in sync indefinitely and would only lose sync do to some external interference, or a power blip or some such. So frequent loss of sync should be a fault. Bear in mind, even a long and highly lossy line should stay in sync, albeit at a slower speed.

We also have metrics in terms of packet loss. Whilst we measure latency, that is not normally a clue to any sort of line/sync issue and more likely to be backhaul congestion (or a router moving enough traffic to build up a queue). I think packet loss (when no traffic) is a really good indicator of a fault. Also, if the line has any internal metrics (OAM frames, Header errors, FEC errors) that is an indication of a fault.

Indeed, the DSLAM can almost certainly report if there are lots of bit swaps, problem frequency bins, FEC errors, HEC or other error stats that indicate a fault, as opposed to simply being a poor/long line.

Unfortunately, neither BT nor TalkTalk define metrics for packet loss, or line errors or re-syncs as a fault metric. I think they should.

Fortunately, when there is a fault, the levels are pretty clear cut. It is rare for a line to have a "tiny bit of loss", though not impossible.

In practice, the fact that the broadband has a fault of some sort, based on our monitoring, or the DSLAM stats, is not itself usually the issue. Usually we can agree that there is an issue with the broadband. We can also usually agree that the copper pair meets the voice spec, when it does.

One of the key things that tells us we have a fault is when a line that used to just work fine (at whatever speed it had) now has problems which it did not have before by any of those metrics. That suggests it can be fixed and made to be back the way it used to be by some means. We know what is possible at that point.

I think we just need to pin down ground rules with providers for what is and is not a broadband fault, and pin down that they are responsible for broadband faults, not us.

So let's start the discussions with them, and try to make things better. Well done TalkTalk for being the ones wanting to talk at this stage. Well, either that or we send round the exterminator!

Did not fit through the door

Race to the bottom

Update: TalkTalk really are trying - and this looks like a case of a discussion point on this one line rather than a more general policy, so stand down the panic for now... Let's talk to them!

Latest from TalkTalk - if the broadband service they sell us gets a fault, their side of the agreed demarcation point (e.g. in the line itself), they will not even try and fix it, but will decide that they can no longer provide the service.

So get a fault - fix by ceasing the service.

I guess that is one way to stitch up your customer!

Well done TalkTalk - you are sinking lower than BT Wholesale now.

Update: We may be making some progress on this, so no need for everyone to ask us to move them back to BT just yet, but thanks to those that did ask. I do hope that we will soon have more sensible ways of working with both providers.

The plot thickens

To SFI or not to SFI?

BT plc t/a BT Wholesale have stated that "SFI2 is an Openreach service which is made available to BT Wholesale customers & charged for on a modular basis." and "The SFI2 visit simply checks whether a line is working within the specification of SIN 349." This is, of course, the basis of my various rants on the matter, and that by that definition there is no way to get a broadband fault fixed.

However, BT plc t/a Openreach, who (from the above) actually provide the SFI2 service state that "SFI2 is a chargeable investigation product that attempts to identify and resolve Digital Subscriber Line (DSL) Service affecting problems." and it goes on to explain that this service is used when the line "is apparently working within the LLU contractual specification of SIN349".

It goes on to explain the modules available, checking at the exchange, and checking the network, etc. It goes on to list all of the steps done by the engineer in the initial (base) module to identify the cause of the problem, such as checking the modem is connected and shows sync. It does say that the engineer does a pair quality test, and if that fails, he will work on basis of a line fault first, and then go back to trying to resolve any remaining broadband issues.

This is, basically, the way SFI2 engineers used to be defined by BT plc t/a BT Wholesale. So why have BT plc t/a BT Wholesale changed their definition of this service, whilst now claiming it is simply a service provided by BT plc t/a Openreach and offered to us. And why are they trying to charge us for it? It is defined as service they can buy to fix the service they sell us (broadband) and even defined in such a way that it should only by used where the line meets SIN349 already, which means BT plc t/a BT Wholesales charging (when it meets SIN349) would always charge us. It would be a mistake for an SFI2 visit to not be chargeable as it should not have been requested in the first place if the line does not meet SIN349.

One of them, either BT plc t/a BT Wholesale, or BT plc t/a Openreach, must surely be lying to us? And the only possible reason we can think of for doing so is to charge us money, making such a lie in to criminal fraud, in my opinion.

We've asked questions about the apparent difference between the two descriptions of the service, and await answers. However, the previous email on this has been weeks with no reply.

For now, we have to work on BT plc t/a BT Wholesale's statements as to this optional service which we would never want or need, and ask them to actually fix the broadband.

It is worth mentioning that this may impact how we deal with TalkTalk slightly, as they are not making the same claims about what the SFI2 service provides. They are, also, responding to us and asking to discuss how we can work together to come up with something better. Well done Talk Talk.

Being fair to A&A customers

Some people may be concerned over some of my fights with BT and what it all means.

• A&A offer, generally, an "Internet service" to our customers.
• We buy a "Broadband service" linking us to our customers from BT plc t/a BT Wholesale
• BT plc t/a BT Wholesale "buy"* a metallic path from BT plc t/a Openreach

At each stage there are gaps, and each entity is responsible for those gaps.

E.g. if the broadband link from us to you is fine, but the "Internet connection" is broken, it is up to us to fix that. There are some things beyond our control, sadly, but we are selling an Internet service and it is up to us to try and make that work.

When the problem is with something we buy, like BT plc t/a BT Wholesale's broadband link and backhaul, they are responsible for fixing that, and we are responsible for making sure they do that.

When BT plc t/a Openreach are failing to provide a working metallic path, then they are responsible for fixing that, and BT plc t/a BT Wholesale are responsible for chasing BT plc t/a Openreach, and we are responsible for chasing BT plc t/a BT Wholesale.

At each stage there is added value. BT plc t/a BT Wholesale do not just buy a metallic path, they also have modems and BRASs and backhaul links and all sorts to make it all work. They expect the metallic path to provide working broadband which is more than BT plc t/a Openreach sell them, so BT plc t/a BT Wholesale have to find a way of making that work, just as they have to make a BRAS work.

Similarly, we add value, and if our routers are not working, or our LNS's are not working or our transit providers are not providing the service we contract them to provide, we have the job of getting that working and chasing people that need to do that.

It is all layers, with each layer agreeing to provide a specific layer of service.

All we are doing is making sure each responsibility fits in the right place and has the right party paying - we would never expect our customer to pay for repairing a metallic path or a broadband service or an Internet service.

We might expect someone to pay if sent on a wild goose chase and incurring costs when it was in fact their own modem or PC or wifi that was actually at fault, which is why we work with end users to check these things first. But if everything checked an eliminated, there is no risk of costs, and the same should apply at each layer in this. That is all we ask.

At the moment, one of these "layers", the BT plc t/a BT Wholesale and the Talk Talk part are trying to absolve some responsibility for the "broadband" aspect of what they sell, and pass on BT plc t/a Openreach charges even when they are for fixing broadband issues and so not our responsibility. That is all that needs fixing here.

So, just like Virgin(!) we will not charge for fixing a fault in what we provide.

We just need our suppliers to not charge us for a fault in what they provide to us.

* I say "buy" as the idea of one legal entity buying from itself is strange, at best.

What comes after SFI2?

This is what we have asked BT plc t/a BT Wholesale (and similar to Talk Talk). Will be interesting to see what they say. Other ISPs may like to ask them the same question.

I have been going over this SFI2 issue somewhat and I think I see the flaw in the process. I hope this helps you, and I would be interested in your comments.

When we report a broadband issue, having eliminated end user wiring and equipment and checked dial tone, BT do various tests, including checking the metallic path is OK to SIN349 from the exchange end. I understand you have some pretty comprehensive testing systems for this.

If the metallic path is not OK, we can arrange an engineer and there is generally no issue with charges. This is not a problem.

However, if you do not find any issue with the metallic path, the next step is that you offer us an SFI2 visit. You do not let us take any other action at that point in the fault process.

This is the step that is broken!

An SFI2 visit is simply to test/ensure a line meets SIN349 - something the exchange tests have already done pretty reliably. It is totally pointless, and severely uneconomical, for us to repeat the tests already done for this from the exchange.

So, from now on, we'll be happy to accept BT's diagnostics and agree that the metallic path meets SIN349. What we need to know is what happens next? An SFI2 is clearly pointless as we both agree the SIN349 test would pass, so what is the next step in resolving a broadband fault within the SLG when it is not caused by a failure of the metallic path to meet SIN349?

And, specifically, how do we get your fault reporting past the insistence on booking a pointless SFI2 visit at that point?

We have a policy now of never disagreeing your SIN349 exchange test and so not booking any SFI2 engineers. So I need to know the next process step please, as a matter of urgency. After all, BT have put a time limit on fixing faults (the SLG) and we want to ensure we do our part to ensuring we follow the process to get the broadband service fixed.

So, what comes after/instead of an SFI2 visit when a line meets SIN349 please?

I look forward to your prompt reply.

I have a cunning plan

As usual, when BT make a change, we have to adapt.

At one point SFI was charged for "work done on end user equipment beyond the NTE", so we made sure no work was done.

Then they decided "a visual inspection of end user equipment" counted as work, making visits chargeable. That was some genius weaseling on their part! So we had end users actually hide their routers. They still had the cheek to charge!

SFI was an "optional extra service" which we refused to order, so they changed it to be "part of the fault repair process" hence us arguing that we should therefore only pay if we actually breached some term such as checking the equipment before reporting.

Now we have the latest that SFI2 is a product or service they offer and which checks the line to SIN349, and fixes (free) to that if it was not to SIN349, or maybe does other work if it does but is chargeable. So once again SFI2 is an optional service.

So the cunning plan...

• We are have got a JDSU, which is the test kit BT engineers use to test a line to SIN349. We hope we will not have to use it that often to prove the point here.
• When we get a fault near us, we will send someone (our staff) to test the line.
• If the line fails SIN349, we report a PSTN fault and get fixed
• If/when the line passes SIN349, we report the broadband fault

Now, at this point BT will, as always, offer an SFI2 engineer. However, we will have the justifiable reason to not take that offer as we will have tested the line to SIN349 already so do not need to pay for a service described as "The SFI2 visit simply checks whether a line is working within the specification of SIN 349". Job done, move on to next step in fixing fault!

However, we will have evidence in our line testing of a broadband fault and be well within our rights to insist BT fix the broadband fault, within the 40 hour SLG, and without using an SFI2 engineer.

If they refuse to fix the fault, county court claim.

We'll see how it goes, and what next crazy step BT come up with to stitch up ISPs after this one.

P.S. Same will apply to TalkTalk.

Update: The JDSU has a lot of useful tests which should help where we can use it, including ADSL and VDSL tests. Looking in detail at SIN349, which is literally a "Metallic path facility interface description" the tests are purely electrical characteristics, and most are tested by BT from their end for us. Indeed, we are not really able to test these without isolation and/or loop at the exchange end. We'll ask BT and TT how we can arrange open/loop at exchange end for independent testing and see what they say. Even so, the specification of SFI2 makes it very clear that it is simply not the appropriate "product" to consider as a means to address broadband faults so we will be back to "Not buying SFI2, are you refusing to fix the fault?" as before. Oh well.

Update: It seems that a SIN349 test needs an earth. If the end user does not allow use of power socket or other earth in premises a full SIN349 test cannot be done. If doing a SIN349 test is what we are paying for and the engineer cannot do that, well, then we won't have to pay, will we?! That could be another angle (assuming engineer does, as they usually do, go on to fix the problem).

I wonder if we need to get OFCOM involved

There is a serious issue with SFI charges, as I keep ranting on. This is because BT have pushed and pushed and pushed to stitch up ISPs, even when there is a genuine fault in the service they sell us.

The issue is not unique to BT "Wholesale" but also carriers like Talk Talk.

Both buy copper pairs from BT "Openreach", who have a monopoly on doing any work on that pair. They cannot send their own engineers to fix the copper pair to resolve broadband related issues, they have to ask BT "Openreach".

The problem is that BT "Openreach" offer "SFI2" as a service, and that is clearly defined as an engineer who will test a line to SIN349 (and fix if it does not meet it). The engineer might do more work to try and fix broadband issues, but might not, it seems. All he has to do is prove to SIN349.

This means that neither BT "Wholesale" nor Talk Talk have any way to get an "Openreach" engineer to fix broadband specific issues on the line. But they cannot send their own engineers either.

It seems that, from what I can see, there is no official way for a carrier offering broadband to get broadband issues fixed. All they offer us is BT "Openreach's" SFI2 service, for which they charge.

Surely Openreach need to offer this type of engineer, and surely the carriers need to absorb any cost for such engineering work on the basis that they sell a broadband service, and not a phone line, to ISPs like us.

The current situation means that nobody can properly offer a working broadband service using BT copper pairs - a situation that surely needs OFCOM intervention.

Or have I missed the bleeding obvious?

SIN349

I had not realised this was quite the official stance before, but it was confirmed today.

If a BT SFI (Special Faults Investigation) engineer goes out for a broadband fault, and tests the line to SIN349 (which is a standard for the quality of the copper pair for voice), then even if he goes on and fixes a reported broadband issue, the visit will be charged by BT (Wholesale).

So that means the engineer found and fixed a broadband issue in the broadband service we buy and yet they charge for fixing it, and that is, it seems, official BT policy to do so.

They have pushed SFI step by step over the years getting more and more outrageous, but this is really pushing too far. They offer no other choice to fix a fault in the service we buy apart from booking an SFI engineer visit. It used to be that if they found and fixed the issue they did not charge, but now they will actually charge for that engineer to fix a broadband fault, even though we have paid for a working broadband service already.

Just to be crystal clear here - if there is an issue with the broadband, as opposed to the phone line itself (i.e. line meets SIN349), BT have no way for us to get that fixed other than an SFI engineer, for which they will charge in that case. They have no option to actually fix the broadband fault otherwise.

I don't think that would get past a judge somehow!

Hopefully we can work with BT to resolve this broken policy - we are working on that, and it may be a chance for our Wholesale Broadband Buyers Forum to get started.

What are other ISPs doing about being charged to fix a fault?

Industry wide process (SFI)

So, once again, the same old issues all over again but this time with Talk Talk.

We had managed to previously get them to agree that they would not charge for an engineer unless it is proved that the cause of a fault is the end user equipment. We are generally happy with such a system because where there is proof given to us we can use that to justifiably charge our customer who will have claimed to checked/replaced his equipment as part of the process. (Obviously if we screwed up in the process, that is our cost).

It seems, once again, that things are not working well, possibly because we have a long list of outstanding disputes over SFI charges, and oddly TRC (Time Related Charges) too.

Once again it is "won't send an engineer unless you agree possible charges", and once again it is "charges is engineer fails to find fault in BT network". This last point is subtly (but importantly) different from "proving it is end user equipment at fault" as they are often reasons to "not find a fault in BT network" even when one exists (incompetence or bad luck or intermittent fault).

Just like we had with BT all those years ago, with Talk Talk, "SFI" (Special Faults Investigation) is a product they sell us to investigate a fault. As such we can choose not to buy such a product - why would we - we have investigated the fault ourselves already and found it to be in the network which we pay Talk Talk for already. All we want is the fault fixed, not some private investigator (SFI) to investigate it for us.

Of course this means essentially Talk Talk refuse to fix the fault, and we get deadlock again.

Their latest is that this is the industry wide processes. Of course I don't give a shit about that! For a start we have a contract with Talk Talk for a broadband service and we expect it to be fixed if there is a fault. It is no concern of ours if TT have engaged the services of some company that seems hell bent on ripping off the whole industry (just my opinion). But importantly the industry wide process is that every ISP disputes these crazy charges so much that BT have to create special processes on top of normal billing systems to manage the number of disputes (another industry wide process).

This really has to be sorted somehow.

Just to be clear (and this is where the lawyers reading may like to comment, unofficially). We consider it that the supplier has to provide a working service and repair faults if not working. We do accept that we have some role to play, checking the equipment, and eliminating it as the cause of the apparent fault. We recognise that if we fail to do what is agreed, and do not check the equipment, we should be liable for the wasted engineer visit that may ensue, as a pre-agreed penalty for a breach of contract. What we do not wish to do is purchase a new investigation service or pay for fault repair, or more importantly - pay if an engineer is unable to find a fault that exists.

It is quite unacceptable for the whole SFI system to encourage poor performance of engineers; to encourage the failure to find a fault.

So, once again, we are at a deadlock, until someone blinks. Only we have done this before and won the argument many times before. Let's see how it goes.

The real challenge is picking the fights that will not impact end users - they should not suffer from this, but if we do not fight it simply means us paying more and charging end users more. It is better to have a system that actually fixes faults and to work with BT and TT to do so.

P.S. We have offered to help Talk Talk break up this industry wide extortion racket process and try and fix things.

Banging head against wall (BT)

We pretty much have to employ someone full time to handle disputes now, it is crazy.

Today Alex, myself and Stuart were reviewing the disputes on one of the latest bills. There are half a dozen still left after we have disputed charges twice. But I really do despair over this.

These are disputes that have supposedly been looked at TWICE by a real person who has responded saying that we are wrong and the dispute is still valid.

One of them has some really detailed engineering notes:-

Engineer notes: On arrival I was met by Adam who represented the service provider as this issue has been on going. I was told we have dial tone. I couldnt perform a Pq as there was no earth. I also had sync on my tester, but Adam could nt see the sync on his laptop. I had called wholesale and spoke to mark, and they couldnt see my tester in sync either. He confirmed that the mapping is incorrect. The llux and lluy on out records show its on ports 24 when infact it should be on 26. Mark said he will update the routing as I couldnt. I then went to the exchange and had a c0-op to get it changed to pair 26 which we did. Went back to ur and demoed it to Adam who was delighted as he was able to access the router. They test head states fault on ex nudge when infact its ok as its junpered to the correct ports now.

Now, this really is detailed (ignoring typos, which I am happy to do, and even though Adam did not represent us), and I have to congratulate the engineer on such good notes and for getting to the bottom of the problem and fixing it. It explains that BT Wholesale had the records wrong and that was the problem - the line was on the wrong ports at the exchange. There is no doubt whatsoever that this was a fault in BT and that the customer equipment was not in any way at fault. Obviously we disputed the charge for this...

The first and second response from BT to this dispute is :-

"Engineer notes clearly states as issue with the customer equipment. Hence, charges are valid"

So we are basically banging our head against a brick wall here - how on earth do these people get dressed in the morning?

Also, several of the disputes are TRC (Time Related Charges) for an engineer visit (about £90 rather than the usual £144 for an SFI). Now, to try and explain this you have to understand the usual logic :-

1. Customer reports an issue to us, and we can see it as well on our monitoring.
2. We get customer to go through checks, including changing all his equipment.
3. We run BT diagnostic tests which come back saying that they cannot see a fault, but if we would like to book an SFI (Special Faults Investigation) we can.
4. SFI engineer goes out, typically does not find or fix the fault.
5. BT charge us.

OK some times engineer finds and fixes the fault, but the principle here is that we say there is a fault, and BT say there is not one, so they offer an investigation service to resolve this disagreement. If that finds it was a BT fault we do not pay, if not then we have to pay. That is the logic.

For a start this logic is very flawed :-

1. BT's tests are not testing the service as provided at the delivery point (master socket) so they are inherently unable to determine accurately whether there is a fault or not.
2. It is in BT's interests to find no fault and offer an SFI as an SFI makes them money.
3. It is in BT's interests for the engineer to fail to find or fix a fault as then they can then charge (even if a later engineer does find and fix the fault!)
4. BT have a monopoly on working their side of the NTE so there can be no competition to address the huge commercial interest bias in steps 2 and 3.

Anyway, putting that aside for a second, just occasionally, and in the case of these specific disputes, we have a slightly different sequence of events:-

1. Customer reports an issue to us, and we can see it as well on our monitoring.
2. We get customer to go through checks, including changing all his equipment.
3. We run BT diagnostic tests which come back saying "report fault to BTW to investigate" (i.e. they are agreeing that something is hinky).
4. We report the fault, and BT come back and say "Please book an engineer appointment with your End User" (no mention of SFI)
5. We arrange an appointment
6. Engineer arrives and fails to find or fix fault
7. BT charge us "Time Related Charges" for his time.

This is odd on two fronts. For a start, Time Related Charges (TRCs) always have to be pre-approved in all work with BT, and we don't approve such. But also, in this case, BT agreed there was a problem, and BT asked us to book the engineer so there is no way in hell we are going to pay BT for that!

Sadly, BT have returned the disputes twice now saying TRC charges apply, and the next step is solicitors and court!

This is, of course, one reason we have started wbbf.uk.

Update: Whilst we are chasing the rest of the issues - the really stupid one at the start of this post has now been "waived" (not a good word, as it implies they are voluntarily giving up some right rather than correcting a mistake). They have not really explained why we had to dispute it so many times, but suggest it was because they cannot see the notes after 28 days. Odd really, as the notes still show on eco+ even a year afterward a fault, and they are included in the dispute we sent, but what is really odd with this excuse is that their reply to the dispute references the engineer notes - or are they admitting that they just make this shit up I wonder? We've asked for more details of exactly what they person who sent the response was referencing. Should be fun.

Building a new community

I ran a FireBrick course today and met some very nice chaps from a small ISP who deal with the likes of BT and Talk Talk just like we do. (Freudian slip, I initially typed "lies" not "likes" in that!).

They were very interested in our efforts to start a Wholesale Broadband Buyers Forum  (wbbf.uk). They have even considered doing the same themselves. I was really pleased at how much common ground we had.

There are a lot of small (and even larger) ISPs that face issues with carriers on a daily basis, and we all want to work together and with carriers to solve these problems so that we can provide the best possible service to our customers. We really do not want to be fighting BT or anyone - but we need a way of working together that means we do not have to fight, and that is not how it is right now.

One of the biggest issues which has plagued all ISPs for over a decade now is BT SFI (Special Faults Investigation). It is a problem in the first place, as it was created on broken foundations so solve a problem that should not have existed. BT, in classic big company style, do not fix the underlying issues but pile on top new layers of bureaucracy and incompetence. The latest is that SFI disputes are such an issue with ISPs that they have a new process for management of disputes (rather than fix the reason for the disputes in the first place).

I had a meeting many years ago face to face with people that dreamed up SFIs, and in that meeting they said something along the lines that an engineer visit that did not find a fault was chargeable anyway so making an SFI added extra value by being able to check end user wiring and stuff as well for that fee. I pointed out that this was in fact untrue - there was never any charge for such a failed engineer visit - I even challenged him to show me where in the contract and price list (and indeed any bill we had ever had) where there was such a charge. He failed to do so - so the fundamental premise of SFI as adding benefit to customers who already paid for the failed visit was actually flawed. Even so, they did not change their ways. This is typical and key here - they created SFI because not checking user equipment was a waste of the customers money on a pointless engineer visit, when in fact the customer was not charged for such visits - the whole logic for creating the concept of an SFI visit was totally flawed and they refused to admit it.

Over the years they have changed the "service" of SFI one step at a time as we find ways to counter it. First it was "charge for work beyond NTE" and we said "do no work beyond NTE". Then it was "work beyond NTE includes visual inspection of end user equipment" so we had people hide their kit when engineers visited - they still tried to charge for a visual inspection, but of what?! Then it was "Fault not in BT so has to be end user equipment at fault" so we sent end users BT branded modem/routers purchased from BT and within warranty as the only equipment on site, making them liable for the line being faulty or liable for saying their supplied equipment was faulty.

The battle goes on to this day with each new stage being a new battle.

The latest two crazy steps are: (1) That a series of engineer visits where the last one finds and fixes a fault in BT requires payment for the previous (presumably incompetent) engineer visits that did not find or fix the fault, and (2) that all visits where the line tests to an analogue phone spec on arrival shall be chargeable even where they expect us to book an engineer for a PPP fault, or a BRAS fault or a BT modem profile fault all of which happen on a line that is good at a copper test level.

With Talk Talk they create another level of indirection and similar stupidity. At least TT are only at the stage of "SFI" being a product and us saying no thanks, we do not wish to buy that product, now fix the broadband we have already bought. This is a stage we had with BT many years ago.

Somehow we need to find a way to solve this - to get BT and carriers to actually work with us to fix faults, and not create a machine for creating and disputing charges. We also need to work on other things like the cost of backhaul bandwidth, the stupidity of wires only FTTC (leading to yet more SFI issues), issues over backhaul congestion within the network, and stupidities of ordering and faults systems not working properly. We even have to consider OFCOM related stupidities like the latest broadband line migration systems.

So the idea of WBBF is to allow this - to bring together ISPs using BT and TT, and solve these issues. We are gathering momentum now.

The next step may have to be a physical event - in London (pretty much has to be, sorry), and with as much chance to talk as possible (so no pub/restaurant with music or entertainers). Something with food and drink - beers with peers sort of thing - and a couple of presentations and question sessions. Mostly about drinking, eating, and talking.

The trick is very much not to be another LINX, LONAP, UKNOF type of thing (all of which are good, but not what we are aiming for) - but a place to allow as much networking and talking as possible. I did wonder if there is any venue with a big anechoic chamber (with booze and food) - I'd pay to use that!

Interesting approach to SFI charges

XLN have taken an interesting approach to SFI charges, offering a sort of insurance.

Obviously, I can see why they are doing this. SFI charges are a lot of money and not something the ISP can really cover within the low profit margins, but also not something customers will want to stomach.

But the underlying issue is that BT are blaming customer equipment when they simply have failed to find the cause of a fault.

In principle, I am more than happy if BT were to prove the fault was end user equipment to then charge for wasting their time, but in our experience that is rarely the case.

So the idea of offering an insurance against the risk of an SFI charge is a novel approach. I hope XLN have the appropriate FSA stuff to offer insurance and are charging insurance premium tax correctly and so on. Or is it not insurance? I wonder.

The only thing I would disagree with is that they seem to be forcing this on their customers rather than simply offering it as an option.

I suppose I should work out if we should offer something like this at A&A. Comments welcome.