Monday, 16 January 2017

Teaching us to suck eggs? BT?

We have a customer on a fibre to the cabinet (FTTC) service which has packet loss.


The red is loss, as measured by one second LCP echoes over the PPP link, and is often over 5%. Levels of random packet like this are severely impacting his ability to make use of the service.

The loss started at the exact moment BT did work on the circuit due to a major outage, so is clearly related.

This is the line before the outage, and you do not get much clearer than that - no loss. Same as every day before :-


This is the day the line came back after their major outage (which lasted two days) :-


And this is the next day, which looks much like every day since :-


It does not take a rocket scientist to see there is a problem there - periods of around 5% loss, sometimes more, most of the day, every day, since the outage.

And yes, that is start of OCTOBER 2016! BT have failed to fix the fault for that long!

Today we got this, and I am almost at a loss for words! Talk about teaching us to suck eggs!

Is the customer using a VPN? Data is transmitted in discrete units known as packets. When a network server is overloaded, these can get discarded. 
This is known as packet loss and results in slow loading game dynamics and graphics, or the unsatisfactory performance of a VPN connection.
In such circumstances, there isn't much which can be done to improve matters, as the cause is not associated with your PC or broadband service.

I'm really not happy about this, but the "there isn't much which can be done to improve matters" is just shocking. We have asked BT to confirm if they are stating, officially, that 5% loss on an idle line is considered "acceptable" for a GEA/FTTC service - we await the response.

They even go on to say :-

In the meantime can you ask the customer to run some traceroute and provide and this hopefully will aid us in seeing where in the network  the packet loss is occurring.
SPs engineers can use a "wire shark" which can detect packet loss at points in network.

This is after explaining that we can see the loss at the LCP level on the PPP link and providing access to and copies of graphs showing the loss over and over again! It is like dealing with Dory to find a fault called Nemo. We keep having to repeat ourselves.

There is one other small snag.

We are all used to the notion of "fibre" broadband not actually being "fibre" which is why this is "Fibre to the cabinet". BT sell this to us as "Fibre to the cabinet" and call it FTTC. It turns out this line is in fact "Microwave to the cabinet". A good idea, normally, but not as described, and clearly beyond BT to actually understand and fix.

This just highlights the problem with a clear definition of the service: We need a clear specification of levels of idle/random packet loss, idle latency and jitter, reliability/resyncs, min sync speeds up and down, and even throughout before loss/latency starts. Without these you can literally spend months bashing your head against a brick wall and having engineer after engineer sent (each potentially costing around £200).

P.S. Today we get "Openreach have completed the engineering checks on the Radio back-haul and discovered some issues with the Link.". Well, yes, we have said this from day one (113 days ago). How bloody annoying. Maybe we finally get it fixed now.

P.P.S. After 120 days they finally fix the microwave link and the line is clean. So annoying.


13 comments:

  1. "Microwave to the cabinet" sold as "Fibre to the cabinet" ?

    How would you stand legally if your customer (who you are contracted to provide FTTC to) chooses to cease their service because the provisioned service is *not* what they ordered from A&A and *not* what A&A ordered from BT ?

    ReplyDelete
    Replies
    1. Indeed, though we sell as VDSL (which it is) rather than FTTC, generally, for this very reason.

      Delete
    2. It'll get fun when G.fast is also GEA-FTTC...

      Delete
  2. What the...!?! I would have never believed BT's level of technical competence and customer service on a wholesale level between isp's could even come close to when they're dealing with residential customers, until I saw this. Just shocking.

    It would be quite interesting to see what happened if anyone took BT to court over FTTC being used to describe the speed as.opposed to how the service was actually delivered (or not as the case may be)...

    ReplyDelete
    Replies
    1. It has been apparent in Adrian's previous posts that quite often AAISP are talking to support personnel who are working an "end user" level not "ISP" level and who have real difficulty with the fact that LCP echo request logs predominantly test the link between the customer CPE and the ISP and are not affected by packet loss deeper in the Internet.

      OK, strictly it checks to the PPPOE endpoint so includes a bit of the customer network.

      I am not sure whether this is because BT is cheapskate or incompetent but suspect both are part of the answer.

      Delete
    2. My experience when working for ISPs around the turn of the century was that BT would fob you off to the lowest level they thought they could get away with. Usually that was a little higher than this incident, but hey, they've had a few years to improve. One of their phone bods insisted he was getting a ping return until I told him I was holding the other end of the cable in my other hand.

      The more of your time they waste with cheap phone droids, the higher the chance you'll go away and stop bothering them.

      Delete
  3. Strictly speaking, is it not microwave to the exchange, and fibre to the cabinet?

    Or is there an actual microwave transceiver stuck on the side of their PCP?

    ReplyDelete
    Replies
    1. Nope, it really is to the PCP, like this:

      http://www.ispreview.co.uk/index.php/2015/02/bt-trials-wireless-cabinet-vdsl-broadband-village-westow.html

      That's quite innovative in a way!

      Delete
    2. Already have to do this in reverse on the other side for a few customers, who despite their close proximity to FTTC cabinets, are not connected to them and BT refuse to route new cables despite clients offering to pay the cost. We then install Wireless gear on the property of friendly neighbours connected to the cabinet and send the Internet merrily on it's way over to our clients. There are countless areas around here where OpenReach could do this themselves and provide a Wireless Broadband solution with small masts erected in key locations saving the cost of FTTP deployments or getting properties either to far to benefit or on poor lines. Many community projects and some commercial offering are trying to pick up the slack and as good as they are, even they struggle to cover all the areas needed.

      Delete
  4. I'm guessing TalkTalk backhaul is not an option for this customer given the microwave link to the cabinet.

    ReplyDelete
    Replies
    1. I think TT would work the same as I expect it is the microwave link that is the issue. TT would see as just FTTC.

      Delete
    2. Depends where the microwave is. If it's part of the Openreach jurisdiction, then all CPs who consume GEA will see the same issue. I believe this is the case here, so both BT Wholesale and TalkTalk interconnecting at the GEA serving exchange (not necessarily the actual local serving exchange, but probably a bigger one further upstream) will have identical transmission path to that specific cab.

      Delete
  5. Oddly enough I had a similar issue myself which turned out to be an issue with my VDSL modem. However AAISP support suggested a router reboot long before things had escalated to this level which got my line back to its normal speedy self.

    Trying to use this line would be an absolute nightmare based on my experience. my 80/20 FTTC line was getting transfer rates that made ADSL1 look good. ;)

    ReplyDelete