Our favourite telco have very much stolen Christmas this year, not just for me, but for an elderly couple in Malvern with no Internet or VoIP phone over Christmas.
The executive summary is as follows... Our favourite telco messed up and broke the configuration on a line, and it stopped working on Friday night before Christmas. The MD of ISP (me) spent all day Christmas Eve, Christmas Day, and Boxing Day, chasing the telco to get it fixed - dispelling and disputing every excuse and escalating all the way to Director level within the telco. This meant pretty much every half an hour checking the status, updating the customer, chasing the telco on email, echat or call, and in-between reading terms and conditions to quote at them. I did have time to open a few presents with my family in the gaps.
But lines break, and take time to fix, surely. This is not a special case is it?
Well yes, but... For a start the fault was caused by them, not some tree falling on a line, but the config on their kit. Then they take ages to make any progress, but do find the cause (well done Archana). This is to no avail as they then say that people who can fix it only work normal working days. But hang on - they agreed to fix faults within 40 clock hours - this is not even trying to fix within that time, and offering something you won't even try to achieve is unethical, dishonest, or even fraudulent... Naughty telco.
Late Christmas Eve I get someone to change their mind and pass to a 24/7 team that should be able to help. Told I have to wait until morning. Hmmm
At 5am Christmas morning I find they have chosen to order a tie pair modification (TPM). This is wrong! The customer equipment, modem, line, MSAN and even BRAS are all working - just that the session is being killed by them after a second with message "Subscriber provisioning failed". It is not a faulty port. What really winds me up is that I took the time to make very very clear to the person on the phone on Christmas Eve that two of their tests were misleading. The RADIUS test incorrectly assumes all short sessions are a "reject" by the ISP and in fact we are accepting the session, and also their TAM system that tries a login will say PPP failed which is to be expected as the link is closed within a second and does not mean the port is faulty. Even so, at 5am they decide (as TAM test PPP is failing) that it is a port fault and needs a TPM - arrrrrrrrrrrg!
To add to the fun, the first thing a TPM does is shut down the existing port (it is a process to move the service to a new port). This is mad when a line is partly working as it takes it out of service for up to 3 working days. In this case it means that at 6am the line stops trying to connect every 4 seconds and is no longer working. Our favourite telco tries to justify the TPM on the basis that the port is not working - but the TPM was 5am and the port stopped working at 6am. Time travel has been discovered it seems.
What is then strange, after a lot of shouting and even involving some direct telco staff member contacts at home at 07:22 on Christmas day morning (you are a star, Ian) and a duty manager who was working (well done John), we get the Director's Service Office involved. They work out the quickest thing now is to try and get a tie pair mod finished quickly. They are pulling strings, trying to get engineers on call and all sorts. Seems it is hard to get an engineer out on what is by then Boxing Day.
I am glad they are trying at last, but wait, this makes no sense, surely the have engineers anyway. After all they sell an "enhanced" level of support which means they work 7 days a week (even Christmas). So where are those engineers. And hang on... They have a service that means you can (for a fee) have an engineer working on this, aiming for a 7 hour fix. This means getting engineers to fix things at short notice is a standard service they offer (for a fee), so why is it a problem when they look to be in breach of contract for them to invoke those normal processes? No answer yet on that one!
Of course, one small gem in this is that at one point they said they cannot do anything Sunday as they do not have people working Bank holidays. They do have people working Bank holidays for the enhanced care. Just to add to the fun we pointed out that 25th Dec is not a Bank Holiday this year, 26th and 27th are. No reply on that one for some reason. It raises the point though - their definition of working hours (where it matters) is Monday to Saturday excluding bank holidays - so if the 25th Dec is Friday or Saturday that means the Saturday (not a bank holiday) is normal working hours. A fun one in future years I think.
The big thing here, and one of the main reasons I personally as MD of an ISP have been so tenacious about this, is that they are not even trying. To be quite honest I think I would have been just as tenacious whether it was my parents' line or not! Over and over again they will ignore the promise they made to fix a fault in 40 hours. They spend a huge amount of effort making up excuses rather than actually fixing the fault. It is especially wrong when the fault is theirs in the first place and is a "soft" fault (i.e. fixed by someone at a computer terminal).
The upshot is a lot of reading of terms and conditions which makes matters worse for them. They say they will fix within 40 hours. They say that if a visit to site is needed they will respond within working hours. The two statements are not incompatible in any way. We are quite happy for a fix in 40 hours but no "response" until working hours. The delay in "response" does not remove or change the obligation to fix in 40 hours, and the only way to achieve that is to have people working bank holidays. If they don't then they are not doing or even trying to do what they agreed to do (fix in 40 hours) and that is serious - not just breach of contract but if the initial offer was made knowing they would not try, then you have possible fraud issues. We will pin down what working hours are even when 25th Dec is a weekend and we will pin down what their contractual obligations are even if I have to pay a legal advisor. Plan is to put the exact outcome on a detailed web page so our customers know where they stand.
Whilst I am happy to say that I will always fight this cause, I am dismayed that it is a fight at all - it should not be - they should be prepared to do what they agreed (they make these terms and conditions) and work with us to make that as efficient as possible. We want that. We want lines fixed, not a huge machine in place to make excuses. They even have a process for being rude, called "hard turn back" where they will point blank refuse to help you - surely someone must have realised they were turning to the dark side when they invented that process!
Maybe worth saying what should have happened in this case, in my opinion... Their systems agreed there was a fault at the start. So someone should have looked at it, and perhaps even felt out of their depth. It is an odd one. They should have passed to someone that could understand it, and they should have passed to a team that can fix it. This could well have taken several hours, but the next morning we should have had a clear diagnosis and fix in place and been back on line. There should have been updates at each stage, even if hours apart. That is the ideal world. No need for me to bounce it back at every stage. No need to pass to departments that don't do 40 hour fixes. No need to try and make a Sunday into a Bank Holiday or invent any more excuses, and certainly no need to involve the Director's office. If they had done that then a handful of people would be involved for a matter of minutes each. Instead I dread to think how many people have spent how many hours on this. I alone have spend most of three days on this, days I should have been with my family, or at least in Azeroth...
As it stands they are really not working together on this, and are making it a battle. When they have an ISP like us that stands up for what is right and want to actually fix faults they will always have a battle unless they change that attitude. We can be a valuable asset or a major foe, and it is their choice. But maybe their heart is two sizes too small... Bah Humbug...
Happy ending? To be added (see below). As of time of posting the line is still not working - in fact after eventually getting a TPM and chasing why it did not work for some hours we have got back to the state the line was in at the start, on Friday night - connecting and being closed by their end with "Subscriber provisioning failed". I wonder what next.
Update: 18:32:47 boxing day - on-line. Finally. Simply needed a "rebuild" of radius and mux config - something they could have done Friday night.