Saturday, 18 August 2012

I don't know how they cope

I do get involved in broadband faults at the weekend - ironically this weekend I was going to be coding (in an air conditioned office) but feeling grotty I stayed at home and so tried to help a customer on irc.

But really, I don't know how my tech staff cope. I dread to think what tablets I'd be on after a week of this.

Got a simple fault case. New fibre to the cab line, which is normally simple as BT provide everything up to and including the modem. So faults tend to get sorted quickly. Line has been in a month.

With normal ADSL the line speed estimate is a range as speed depends on so many factors which are not all within BT control (splitter, internal wiring, router model, etc). However, for FTTC BT come up with a simple estimate of speed, and in this case the customer had, and still has, a speed estimate of 31.4Mb/s. Yes, that is an estimate to that precision. It is not a hard target, it is an estimate, and quite often lines exceed the estimate. It is not an "up to" figure.

Now, the line did not get 31.4Mb/s, or even close. Which is odd. It was pretty stable at 21Mb/s to 24Mb/s for the first 20 days. Now BT do have a notion of the first 10 days being for the line to settle, i.e. basically for the characteristics for the modem to be established to get a stable speed. But the line was over 21Mb/s after 20 days. Then, suddenly, it dropped to 16Mb/s and was unstable, and finally now it is 14Mb/s.

So, we get an engineer out to fix it. This was not too much hassle as we ran a few tests until the BT test system said there is a fault and to contact BT to book an engineer. This should be plane sailing at that point. Engineer for today.

Engineer goes out, checks the line, says it will never get more than 16Mb/s and leaves. No attempt to fix it. The line was in fact synced at 14.3Mb/s, a far cry from the 31.4Mb/s estimate or previous 21Mb/s speeds.

So, simple, I get on echat to BT, as you do. This is where is gets silly. It took me over an hour of arguing to get anywhere. BT saying the engineer has fixed the fault? BT saying that the engineer tested the line and it will never get more than 16Mb/s. BT even saying it never has. They started with comments about why I feel it should be faster and I had to explain that BT are the ones that feel it should go faster as even now, today, the speed estimate is 31.4Mb/s. Eventually, after 70 minutes, and quoting sync speed and date/time for the history of the line to demonstrate that this line had more than 21Mb/s for 20 days, suddenly they changed their tune.

That was actually quite a shock. After loads of no can do attitude, and won't ever get faster, and basically tough! the attitude suddenly changed. I am not quote sure what I said, but suddenly it was I'll book and engineer to fix it and will today PM do? Gobsmacked! Today? I hear from my customer that the engineer had just turned up even. I am pretty sure I did not say Shibboleet but I may have to read through the transcript for the magic words...

So, the long and short of it was that first engineer did not fix it, so second engineer same day, and he is supposedly fixing it. That should have been it. That should have taken 5 minutes of echat at most. But they had to drag it out and make it 75 minutes of patronising hell. Why?

I remember when we started with broadband 12 years ago, and we would call BT, tell them a line was faulty, and they went and fixed it. None of this crap. No arguing the toss. I don't think I could work in support fully time with this hassle these days. Well done guys!

Update: Poor customer not much better off, 16M and packet loss. So BT not meeting their 40 hour SLA. So we escalate it, which is the process for when a fault is not meeting the SLA. Only the escalation department won't look at it until Monday. So what? They have a team to handle delayed fault fixed that themselves don't work the hours necessary to get faults fixed within the SLA. Arrrrg.

On top of that it seems, according to the person on echat, that BT plc have no threshold at which they consider the estimate to be wrong or the installation failed. Even estimating 31.4M and getting half that is, as far as they are concerned, acceptable! We are trying to get a straight answer, and if they don't come up with them we'll tell them what we consider "acceptable" on future orders (which will piss them off - they hate when we do that sort of thing).

Update: We have been reading the handbook in more detail - mainly to make sure I am up to speed if I do faults at the weekend. Seems lines getting below 2M are considered faulty. Lines getting below 15M if they initially got over 15M are faulty. And lines starting over 15M and losing 25% in under 14 days are considered faulty. Complicated set of rules none of which seem to relate to their forecasts (so their forecasts can be as wrong as they like). Even though echat said there was no fault threshold and argued about the speed being acceptable, we now know that under BT plc's own rules that this line is faulty so needs fixing. Arrrg!

Update: Got BT to change to new pair and customer is happily on 25M now. This meant a lot more time arguing yesterday and today to get it sorted. Why they have to spend so much of our staff time, and their staff time arguing rather than doing, I do not know. It would be cheaper for all concerned if they did this right from the start!


  1. The chat went over an hour and it was now causing a detriment to his KPIs and he wanted to get rid of you?

  2. Interesting - our FTTC install has had a similar degradation in performance - estimate was 41mbit down, 7 up.

    We got 27 down for the first 2 weeks, which I was fairly happy with at the time. Since then it's been mostly hovering between 19-22. Most annoying is the latency which has gone from 12 ms to 36ms. BT's view seems to be that there's nothing wrong. Do you think it's worth getting your support guys to give them another kick?

    1. We need to establish some hard rules with them I expect. This case was one where the line suddenly dropped speed which is normally a fault. With ADSL they set a 'fault threshold" based on initial speed less a percentage, so we have a hard limit below which was considered a fault. Though they then changed "fixing" to mean "resetting the fault threshold to lower value". I will have to check the rules on FTTC though.

  3. the photo shows the problem, a dog has pissed on it...

  4. the photo shows the problem, a dog has pissed on the cab...

  5. Doesnt FTTC also have a fault threshold rate? Not got the docs infront of me at the weekend but I thought one was set based on the initial stable rate.

    1. Indeed, I hope so - but as I am not doing faults every day I am not fully up to speed. The BT rep said categorically not. So I will be checking handbooks for both FTR and for failed install low speeds compared to estimate.

  6. Since you are often held up by BT, I though you might be amused to note that they are sometimes hoist by their own petard. Today I got a bit of junk mail proclaiming:

    "London 2012 is here! What do you want to see?

    BT, official communications services provider of the Olympic and Paralympic games"

  7. Ha! I'll take your "arrived Aug 20" and raise you "arrived Aug 13" (less than 12 hours after the Olympic Closing ceremony).

  8. In my case (not an AAISP customer) my line is sync'd at 79Mb/s but twice now throughput has dropped to 30Mb/s peak (averaging a bit lower) 24/7. First time it stayed like that for two weeks. Now it's done it again. Looking at the BTor docs it sounds suspiciously like congestion control. One thing bothers me about that though: Their docs say speed is /momentary/ reduced to 30Mb/s - this seems 24/7. It also seems odd to have congestion only two months after the cabinet goes live then again two months after that. I'd have a thought a newly enabled exchange would start off with a load of backhaul capacity.

    Unfortunately BT do not regard it to be a fault for a line syncing at 79Mb/s to only achieve 25Mb/s from their speed test. Apparently anything over 12Mb/s is acceptable. My ISP isn't interested in even trying to raise an issue with BT. They say there's no point.

    I just have to hope it gets fixed after a couple of weeks like it did last time.

  9. Not involved in VDSL being a WISP operator but it would be worthwhile finding out if the end user (or less easily their neighbours) are using Powerline networking devices as these occupy the same spectrum as VDSL
    BT's 'comtrend' devices are particularly severe offenders.

    Another possibility is other electronic devices that are creating very high levels of Radio frequency interference on the mains. Low voltage lighting, LED lighting and plasma TV are all potential candidates

  10. I have a line which estimates over 71 Meg, initially sync'd at 45 Meg and has dropped to 39 Meg. KBDs reckon all is well :(

    Have you ever been on echat when the lovely offshore chappies keep you holding whilst they "check with their suppliers" so you go off and do something else, and completely forget about your echat window?

    Oh, and over the past few weeks we've started seeing a *lot* of occasions where FTTC installation appointments are simply missed by BT or the 'engineer' (I'm not sure anybody at Kelly Communications is classed as an 'engineer') turns up and is ill prepared and so cannot install it and it gets delayed to the next available appointment which just happens to be 2-3 weeks down the line. We've had them refuse to leave the VDSL modem on site. Gone very much downhill, and they're blaming the weather!