Thursday, 18 April 2013

"if you paid more this would not happen"

One of the problems we face on a day to day basis is customers who's broadband service has a fault.

An individual broadband line can work for many years with no fault, but with thousands of customers our support department deal with faults every day. The most common types of faults are physical faults on lines or equipment. This is no different to telephone lines that can fail due to wear, corrosion, even copper theft. In some ways broadband is more susceptible to faults because it uses more of the phone line in a way - i.e. you can have working phone but non working broadband.

One of the challenges the support staff have is handling a customer for whom their Internet connection is critical for their business. It is increasingly common that a business will grind to a halt with no Internet access.

It is a fact of life that putting copper wires in the ground, you will get faults. Some of the wiring is decades old. It has many joints that are in various types of cabinet and distribution point. Things can flood and break. Thankfully some newer fibre technologies increase reliability, but for normal broadband this is the state of things. We try very carefully to explain this when people take a service - that it can fail and can take days to fix in some cases.

So when someone has Internet that is critical for business, it makes sense to ensure you are not relying on a single broadband line which can fail. You need some sort of backup. That could be as simple as an iPad with a data SIM or a mobile you can WiFi tether - if you are a small business. For larger businesses you need to consider multiple lines, 3G backup and so on. As an ISP we offer a range of higher availability services from multiple lines, our Office::1 service with 3G backup, and even direct fibre connectivity. There is a choice of price and level of service and reliability. A business can make a simple commercial decision as to what level they need and can afford.

But at the end of the day, if you pay more for a higher availability service, the chances of a fault stopping your Internet completely is greatly reduced.

I say reduced, we had a case last week where a copper theft took out hundreds of lines, including both lines for a customer that had two lines. 3G backup is the next step, but there are, of course, even less likely scenarios where all links can fail, and even simple things like power cuts can happen. It is all about managing risk.

What is strange is that, when you explain this simple fact of life to a customer who only has one line for their business: that "if you paid more this would not happen" (as a customer put it), they get all indignant and cross. What is that about?

I think the best we can do is try and ensure staff are as tactful as possible.

But if you have a business that relies on Internet, please do think about what backup you have for that.


  1. Business Continuity Planning - it's not just for fires

  2. Yep I can't imagine why any customer would get annoyed if, when their most critical connection is down, you then tell them that paying more would avoid that problem :). It might be true but that’s not how you approach it.

    Politely suggesting solutions like a 3G or Satellite backup is one thing but I'd avoid tripping into situations where you run the risk of sounding like you're lecturing somebody, whom is no doubt already in a frustrated mood.

    Mark -

    1. We're taking the complaint seriously and reviewing the call. We should, indeed, be tactful. I think, in this case, the customer kept saying how critical it was and getting cross, which makes it a bit difficult not to say something a tad less tactful. I don't think we did say anything wrong, to be honest, but as ever, we try and improve...

  3. The problem is, I think, greater than simply "pay more for better service". The problem is that there is a vast, vast increase in cost for a service with some kind of SLA and which uses alternate paths/routes to the cheap (xDSL) solutions.

    As you have already said, you have one customer with two lines that were both taken out of action, so saying "have another line" isn't necessarily an answer. A business doesn't have to be that large before a GSM based backup solution becomes woefully inadequate.

    The other problem is also that as a society, we pretty much cannot function without an Internet connection and yet the laws of the land don't recognise this. At least in terms of ensuring that everybody has a right to a working Internet connection.

    So, I'll set you a challenge. Consider the following hypothetical situation:

    Company X has an office at postcode SN9 6BQ (I picked somewhere in the sticks at random). GSM service is ropey (they can get an OK 2G signal outdoors). They have 1M on ADSL Max and currently have two ADSL circuits bonded. BT require £5,000 in works charges to install any more lines at the property. The company employs 10 people from the local area and just about breaks even every year. All ten staff are telephone based using a hosted VoIP service and make extensive use of the Internet to receive orders from their customers and place orders with their suppliers. The nature of their business requires them to be located in that area.

    What is the minimum amount more the company would have to pay to protect themselves from, say, copper theft between them and the exchange?

    OK, so all the above is worst case scenario, but it's an interesting problem to discuss, no?

    1. That is an extreme case, obviously. Ultimately where you choose to set up business is a factor - if you rely on Internet connectivity then do not set up on a remote Scottish island, for example. In most cases a second line will massively reduce the chances of a total failure. Nothing can completely eliminate all risk, but a second line is a simple, and relatively cheap option for most people. It is rare that you cannot get a second line installed, even out in the sticks. There may be options for satellite links, for example (not something we offer yet) which have high latency but consistent so should even be usable for VoIP as a last resort. The main gripe I have and the reason for the post is where people could have a second line for very little more money and choose not to - yet get upset when the one line breaks and takes a day to fix.

  4. @Nick - Although in the same business, I raise issue with "At least in terms of ensuring that everybody has a right to a working Internet connection"

    No one has a right to an Internet Connection. There is no USO to provide an internet connection and while I fully understand the companys' issues, surely in this day and age if you were a manufacturing company that absolutely had to have mains gas to operate, you would choose a location that had mains gas (and enough of it to service your needs). If you need Internet, just like a rural location without mains gas (or small supplies inadequate to meet your needs) you can pay for it and pull some fibre in - then of course they can have as much as they like.

    To my mind businesses / individuals who are 100% reliant on the Internet/mains gas/whatever else need to wake up and ensure the location provides the services they need, or be prepared to pay heavily to get the infrastructure to them..


    1. I sort of agree. But...

      (Almost) every VAT registered business is 100% reliant on an Internet connection (to file their return). More and more businesses are becoming reliant on e-mail at the very least.

      Speaking personally, I do pretty much everything I can on the Internet - shopping, car tax, insurance, telephony etc. It won't be long (I reckon ten years) before this is the norm and some services become Internet only - like VAT returns are now. At the point when you start disenfranchising people who do not have an Internet connection, the right to an Internet connection should be a given.

      However, like most things to do with government, it won't happen until it's too late.

    2. Relying on Internet for your business or just using to file VAT return is far from the same - that you can just go to the library and do your VAT return, and you have a whole month to do it, so a day or two with no Internet at office is not relevant.

      You can get an Internet connection good enough for a VAT return and HMRC RTI and other statutory things EVERYWHERE, apart from dial-up you can get satellite lines. A "right" to internet access is not the same as having 2Mb/s+ and low latency. But even if you had such a right, it would be allowed to break, just like power can now.

    3. I think you're missing my point. I was using VAT returns as an example of something which requires Internet access and posited that other services will follow. At some point (and it may be ten years down the line), we will require an Internet connection to function in society.

      I have never said that we have a right to Internet access. What I said is that at some point (when everybody needs a net connection), it should become a right.

    4. Not really - my point is that internet access for something like VAT returns or any statutory function, is very very easy. It does not have to be 24/7 or fast. Dialup covers that, as does satellite. Saying you have a "right" to Internet for such things is a tad pointless as you already have access, anywhere you can see the sky, to slow and high latency and expensive Internet access via various satellite providers, right now.

      As someone else says, it is a bit like a right to mains power. You can get mains power pretty much anywhere, but it is not going to be on a scale to allow you to run an iron foundry - you would have to pay for special power feeds. The same is true for Internet - basic connectivity can be done anywhere, but 100% uptime high speed low latency "industrial" Internet is very different.

    5. I didn't say I had a right to such things.

      I made a point of clearly stating that that was exactly what I was not saying.

  5. I always have the up front conversation with new VoIP clients about the various internet connectivity options, and explain the guarantees (or lack of) and risks etc for all the various options.

    I think 90% of them choose the cheapest option (which is to have at least 2 ADSL lines, with one normally dedicated to VoIP but failover to the other - we used to offer a sharing one line option, but that just lead to far too many quality issues).

    What I find annoying is that if a fault does occur that takes out both, they complain that they need it fixed NOW and how they're losing money etc, when it's out of our hands as the fault is with the carrier. We tend to avoid reminding them that we pointed out and they accepted this risk when choosing this option, as it typically just leads to more aggro. What's even more frustrating is that once the problem is resolved, we will ask if they want to investigate a more reliable / better guaranteed service - unfortunately once they see the price tag they normally say no way (even when reminded they claimed they were losing £x an hour when it was down previously!)...

    Oh well, I guess human nature is to go for the cheapest option, and there's not much we can do to avoid that...

    1. Sounds like we're in exactly the same boat. When the inevitable happens, I tend to respond "OK, I know it's a pain in the backside, but sadly those sort of response times are what you get with xDSL service. Would you like me to go through the options for service with an SLA?" It seems to work - calms them down and makes them realise that reliability comes at a (significant) cost.

  6. if you rely on Internet connectivity then do not set up on a remote Scottish island, for example

    only until the end of 2016 - "400km of subsea cable being deployed as part of 19 crossings"

  7. Your Business::1 product has multi-line resilience, and optional 3G fallback in case of multiple failures. Well done! Few providers will do anything like that on anything short of dedicated fibre runs.

    I guess that in the event that both DSL lines go down, and it's running on just 3G, that inbound connections to the address ranges (v4 or v6) will be failing, because of (a) evil carrier-grade NAT, and (b) you state that the 3G will protect in case of an outage at the ISP end, so unless the firebrick attempts to hook up a tunnel over 3G to the ISP, and if that fails then and only then fails back to outbound IPv4 only, then the 3G is the last ditch attempt to have some sort of connectivity for the office.

    1. We called it Office::1 and the 3G can be either using a SIM from us, so same IPv4 and IPv6 via protocol 41 tunnels, or can be NAT via a separate operator - customer's choice. You could use a USB hub and a second dongle and have two layers of 3G backup even. But staying connecting in some way at all, even NAT over IPv4 over 3G is better than nothing, and people have happily run a small office with VoIP phones in such cases!