The telco that stole Christmas

Our favourite telco have very much stolen Christmas this year, not just for me, but for an elderly couple in Malvern with no Internet or VoIP phone over Christmas.

The executive summary is as follows... Our favourite telco messed up and broke the configuration on a line, and it stopped working on Friday night before Christmas. The MD of ISP (me) spent all day Christmas Eve, Christmas Day, and Boxing Day, chasing the telco to get it fixed - dispelling and disputing every excuse and escalating all the way to Director level within the telco. This meant pretty much every half an hour checking the status, updating the customer, chasing the telco on email, echat or call, and in-between reading terms and conditions to quote at them. I did have time to open a few presents with my family in the gaps.

But lines break, and take time to fix, surely. This is not a special case is it?

Well yes, but... For a start the fault was caused by them, not some tree falling on a line, but the config on their kit. Then they take ages to make any progress, but do find the cause (well done Archana). This is to no avail as they then say that people who can fix it only work normal working days. But hang on - they agreed to fix faults within 40 clock hours - this is not even trying to fix within that time, and offering something you won't even try to achieve is unethical, dishonest, or even fraudulent... Naughty telco.

Late Christmas Eve I get someone to change their mind and pass to a 24/7 team that should be able to help. Told I have to wait until morning. Hmmm

At 5am Christmas morning I find they have chosen to order a tie pair modification (TPM). This is wrong! The customer equipment, modem, line, MSAN and even BRAS are all working - just that the session is being killed by them after a second with message "Subscriber provisioning failed". It is not a faulty port. What really winds me up is that I took the time to make very very clear to the person on the phone on Christmas Eve that two of their tests were misleading. The RADIUS test incorrectly assumes all short sessions are a "reject" by the ISP and in fact we are accepting the session, and also their TAM system that tries a login will say PPP failed which is to be expected as the link is closed within a second and does not mean the port is faulty. Even so, at 5am they decide (as TAM test PPP is failing) that it is a port fault and needs a TPM - arrrrrrrrrrrg!

To add to the fun, the first thing a TPM does is shut down the existing port (it is a process to move the service to a new port). This is mad when a line is partly working as it takes it out of service for up to 3 working days. In this case it means that at 6am the line stops trying to connect every 4 seconds and is no longer working. Our favourite telco tries to justify the TPM on the basis that the port is not working - but the TPM was 5am and the port stopped working at 6am. Time travel has been discovered it seems.

What is then strange, after a lot of shouting and even involving some direct telco staff member contacts at home at 07:22 on Christmas day morning (you are a star, Ian) and a duty manager who was working (well done John), we get the Director's Service Office involved. They work out the quickest thing now is to try and get a tie pair mod finished quickly. They are pulling strings, trying to get engineers on call and all sorts. Seems it is hard to get an engineer out on what is by then Boxing Day.

I am glad they are trying at last, but wait, this makes no sense, surely the have engineers anyway. After all they sell an "enhanced" level of support which means they work 7 days a week (even Christmas). So where are those engineers. And hang on... They have a service that means you can (for a fee) have an engineer working on this, aiming for a 7 hour fix. This means getting engineers to fix things at short notice is a standard service they offer (for a fee), so why is it a problem when they look to be in breach of contract for them to invoke those normal processes? No answer yet on that one!

Of course, one small gem in this is that at one point they said they cannot do anything Sunday as they do not have people working Bank holidays. They do have people working Bank holidays for the enhanced care. Just to add to the fun we pointed out that 25th Dec is not a Bank Holiday this year, 26th and 27th are. No reply on that one for some reason. It raises the point though - their definition of working hours (where it matters) is Monday to Saturday excluding bank holidays - so if the 25th Dec is Friday or Saturday that means the Saturday (not a bank holiday) is normal working hours. A fun one in future years I think.

The big thing here, and one of the main reasons I personally as MD of an ISP have been so tenacious about this, is that they are not even trying. To be quite honest I think I would have been just as tenacious whether it was my parents' line or not! Over and over again they will ignore the promise they made to fix a fault in 40 hours. They spend a huge amount of effort making up excuses rather than actually fixing the fault. It is especially wrong when the fault is theirs in the first place and is a "soft" fault (i.e. fixed by someone at a computer terminal).

The upshot is a lot of reading of terms and conditions which makes matters worse for them. They say they will fix within 40 hours. They say that if a visit to site is needed they will respond within working hours. The two statements are not incompatible in any way. We are quite happy for a fix in 40 hours but no "response" until working hours. The delay in "response" does not remove or change the obligation to fix in 40 hours, and the only way to achieve that is to have people working bank holidays. If they don't then they are not doing or even trying to do what they agreed to do (fix in 40 hours) and that is serious - not just breach of contract but if the initial offer was made knowing they would not try, then you have possible fraud issues. We will pin down what working hours are even when 25th Dec is a weekend and we will pin down what their contractual obligations are even if I have to pay a legal advisor. Plan is to put the exact outcome on a detailed web page so our customers know where they stand.

Whilst I am happy to say that I will always fight this cause, I am dismayed that it is a fight at all - it should not be - they should be prepared to do what they agreed (they make these terms and conditions) and work with us to make that as efficient as possible. We want that. We want lines fixed, not a huge machine in place to make excuses. They even have a process for being rude, called "hard turn back" where they will point blank refuse to help you - surely someone must have realised they were turning to the dark side when they invented that process!

Maybe worth saying what should have happened in this case, in my opinion... Their systems agreed there was a fault at the start. So someone should have looked at it, and perhaps even felt out of their depth. It is an odd one. They should have passed to someone that could understand it, and they should have passed to a team that can fix it. This could well have taken several hours, but the next morning we should have had a clear diagnosis and fix in place and been back on line. There should have been updates at each stage, even if hours apart. That is the ideal world. No need for me to bounce it back at every stage. No need to pass to departments that don't do 40 hour fixes. No need to try and make a Sunday into a Bank Holiday or invent any more excuses, and certainly no need to involve the Director's office. If they had done that then a handful of people would be involved for a matter of minutes each. Instead I dread to think how many people have spent how many hours on this. I alone have spend most of three days on this, days I should have been with my family, or at least in Azeroth...

As it stands they are really not working together on this, and are making it a battle. When they have an ISP like us that stands up for what is right and want to actually fix faults they will always have a battle unless they change that attitude. We can be a valuable asset or a major foe, and it is their choice. But maybe their heart is two sizes too small... Bah Humbug...

Happy ending? To be added (see below). As of time of posting the line is still not working - in fact after eventually getting a TPM and chasing why it did not work for some hours we have got back to the state the line was in at the start, on Friday night - connecting and being closed by their end with "Subscriber provisioning failed". I wonder what next.

Update: 18:32:47 boxing day - on-line. Finally. Simply needed a "rebuild" of radius and mux config - something they could have done Friday night.


BT invent time travel

In chasing BT since Friday on a fault I have had a couple of absolute gems I thought I would share...

Backwards time travel

They are saying they have to do a TPM (tie pair modification) because the port is down, but the logs confirm the TPM was ordered at 05:07am but the port did not stop working until 06:04am. So they must have invented time travel to use the port being down as the reason for the TPM being ordered.

Forwards time travel

They have decided that clocks stop for UK holidays. This is one we are going to have to research in the T&Cs somewhat. They say that the "40 clock hours" excludes UK public holidays. That seems crazy to me, but it might in fact be what the contract says. I'll report back on that. They have, of course, slightly shot themselves in the foot on this as today (25th) is not a UK public holiday (it is a religious festival, the public holidays are Monday and Tuesday), so that is not an excuse even if the T&Cs do exclude them.

Lets see how this one pans out. I have a more comprehensive and scathing blog post to follow on this when I get the wording right.


What happened to Christmas this year?

OK, apparently it is not just me.

For a start, how is this evening not Christmas Eve? It feels like it. It seems somehow a 24 hour lul in the middle between now and proper Christmas. We finished work - we had pizza at lunch time - we put presents under the tree - we got pissed - we played with light sabres in 3D... It must be Christmas Eve.

Even so, Christmas seems to be broken this year some how - not the usual enthusiasm or anything. I have got a few key pressies for a few people, but it sort feels not very Christmassy at all.

Just not quite there. I am sure a lack of snow is not helping, but that is not unusual - snow next month I expect...

Well, I guess Merry New Year and a Happy Chistmassy thing to all of you...


3D computer games

I have already mentioned a bit about 3D TV, and especially Sky 3D which sort of cheats by sending a normal HD picture which happens to be two side by side squashed images. The TV separates them, stretches them and presents one to each eye. It is a cunning way to use older technology to send something new. The storage, transmission, Sky box, recordings and everything sees it as just a normal HD image until your TV does the magic. In my case the TV is a large Sony which alternates the left/right image and uses active shutter glasses controlled by infra-red to block one eye at a time. Some TVs are now working in different ways and ultimately I hope they will use the cheap passive glasses like at the cinema (I think LG have one that does now).

My 3D TV has down sides - the glasses are expensive, big and uncomfortable and the batteries go flat. There is ghosting of one image on to the other. You also lose resolution by the side-by-side transmission used by Sky. That said, you can, with enough glasses, have the whole family sit and watch a 3D film and enjoy i. It works!

3D computer games are a tad different. I have a nice nvidia card with DVI and HDMI outputs (even comes with an HDMI cable). I have an Acer 24" monitor with 3D glasses. The monitor takes HDMI, DVI or VGA and actually comes with all of the cables (impressed!). The glasses are nicer, lighter, and recharge from a USB lead which you can have plugged in whilst using them. The monitor switches at 120Hz and has no bleed through between eyes at all. The glasses presumably have longer blackout periods including both eyes during the switch between images making everything else in the room rather dark - this is probably only possibly because you sit so much closer to a computer monitor, but the end result is excellent.

It is worth understanding a bit about how these games work though. Once upon a time a computer game designer worried about the pixels they produced on the screen. These days the graphics cards do all the work and the game is in fact creating animated 3 dimensional objects from triangles in space. Games like Wow are a lot more about story and artwork than graphics. Some games have a lot of back end physics engines, such as car games, but in WoW characters can just walk through each other. The graphics card does the projection in to a 2D image, and even does lighting, shadows, smoke, dust, flame, and all sorts of effects to make the image look real.

What this means is that to make a game 3D you just need the graphics drivers to plot from a slightly different viewpoint that the game says, one to the left and one to the right, and present the two views, one to each eye. The card sends full resolution 3D to the monitor, either via HDMI or DVI. The HMDI has some restrictions on frame rate and/or resolution which means it may even work with my Sony TV (to be tested later). But the Acer using DVI will do 1920x1080 60Hz and display in 3D.

Update: Tested with Sony 3D TV. Works for setup and tests, but seems to need a £25 download from nvidia to actually use it - what a con!

The game does not even know you have 3D.

In fact the game does not really care what frame rate, resolution, or capability your graphics card has. It is a smart way to do things and allows people to have cheap cards that provide a lower quality or expensive cards and large monitors for higher quality yet the underlying game stays the same. 3D just adds to this like any other clever thing the card with do. The Nvidia card has a control panel which even allows per game settings that override the defaults (such as turning on anti-aliasing - useful).

Whilst this is a really good idea - that any game using the graphics card for 3D rendering can be played in proper 3D using the glasses, there is a down side. It means the game designers do not think about 3D, or don't have to. But should this matter? Surely they are just making a 3D scene which must work else it won't look right with one eye (the normal 2D projection). Not quite.

I have played Star Wars - The Old Republic and World of Warcraft. SWTOR was clearly not designed for 3D and EA have not bothered to sit one of their designers in front a 3D screen with it. A simple and inexpensive move which would result in a handful of minor tweaks at almost no cost, but sadly not. WoW on the other hand clearly have.

So what are the issues?

The first issue is where you put the view point. When rendering in 2D it does not matter how far back you are or how close you are - that just changes sizes of things and some of the viewing angles. However, SWTOR places a lot of things in front of the monitor (close to you) - so much so that the default settings are unusable. Setting minimum depth on the nvidia controls you can finally play it, but your avatar and anything else close is in front of the monitor. That alone is not a big issue - it works, if a tad odd. Things at the edge of the screen are a tad odd if too close, but workable.

As you may expect, all of the the on-screen controls appear flat at the monitor - as a sort of heads up display - like a sheet of glass in front of you and you are looking at the scene through it with text and graphics on it. In WoW the scene is all behind the monitor so the on-screen graphics are fine as they are in front of everything. In SWTOR the on-screen graphics are visible in front of things but behind them in perspective so it breaks the 3D effect. Even the mouse pointer in front of your avatar is horrid, and I kept moving it out of the way during many of the graphics sequences as it appears in a sort of double vision effect and is just wrong. In WoW using the mouse to click on things in the distance takes getting used to as the mouse pointer is on the screen, but that is usable, and does not look wrong.

Update: top tip for WoW - turn off hardware mouse cursor. WoW will then place the pointer in 3D space and move depth depending on what is there. This allows you to more naturally point at things in the distance. Very cool.

SWTOR has a few other quirks - like the name banner over a character. It is in the 3D space, but is quite a way behind the player. It is like they wanted it a bit smaller and instead of making the font smaller they put them in the distance a bit. That is fine in 2D, but in 3D it makes it hard to work out who has what label!

So, last night, I sat next to my son playing SWTOR. It was funny as he has no idea how to play and I was telling him what to do an what keys did what even though I have never played (yes, it is just WoW with light sabres). I just got so used to the 3D, and it was excellent. Looking to one side and seeing his screen was just weird as it was, well, flat! It looked boring and strange compared to my 3D screen.

Overall - well impressed!


Quote of the day...

I almost feel like A&A have been 'technology magicians' in what has been done. 

I like that - we should use it in marketing :-)

Turning to the dark side

So, becoming a Sith is turning to the dark side - possibly. Having got my son Star Wars The Old Republic for his birthday he wants someone to play with (me!).

Unfortunately it is not working well with wine yet, so it seems. And not available for Mac. So that leaves wind owes as the only option.

So now I have a windoze machine - that really is turning to the dark side. I think it will be temporary until wine works well enough with SWTOR. But still, I have no idea how to use one of these things. It looks like I can turn it on and click on the WoW or SWTOR icon, and not have to touch anything else. I hear they get lots of viruses - though I have had my annual flu jab - I am not sure I want to get infected with microsfot.

Of course, if I have to get one, I may as well get the nvidia card that does 3D and an Acer monitor and glasses. It looks awesome for WoW in 3D - impressive.

Still a shame the game makers don't do a side-by-side output mode for a normal 3D TV, as that would allow playing on the 55" TV rather than this measly 24" screen. It seems, however, that the 3D is all in the graphics drivers and the game may not even be aware of it. After all, games are just feeding 3D textured triangles to the video card these days, so it can all be done behind the scenes, it seems. It helps if the game designers near it in mind in the design, though.

I'll try SWTOR on 3D as soon as my copy arrives - hopefully today.

Of course, we already have issues. The is operating system and WoW and SWTOR only, so we figured a small(ish) 60GB SSD would be fine. Nope! 60GB all used up. So getting a bigger drive this morning. Anyway, this is my Feast of Winter Veil present to me - thank you Grandfather Winter.


RIPE still squirming over PI

Well, we keep replying and getting no closer really.

They keep saying what current RIPE policy is, and that according to current RIPE policy the current policy applies to old assignments as well.

However they continue to refuse to explain any legal mechanism by which our customer is subject to RIPE policies in the first place (current or old).

Even if the customer agreed to RIPE policy back in 2003 it is not clear that the policy they agreed at the time bound them to newer policies. i.e. that the 2003 policy allowed for unilateral changes to policy later.

I suspect that at the end of the day it will just be simple blackmail - my customer having to sign under the duress of a threat to disconnect their Internet.

We'll keep trying to get a straight answer out of them. They could at least have the courtesy to admit that it is just blackmail and not pretend that their policies apply by magic.


I am not cool enough!

So, my 16 year old has a birthday party, and is using the conference room in the office for it. She also has the company bouncy castle but it started hailing (no, not raining, hailing FFS, so I imagine that is off)...

That means I am not cool enough to be invited, but the orc is! For The Horde!

Happy Birthday G

P.S. ChoccyWoccyDooDah did OK but not quite what we asked for. Shame about the giraffe.


How to complain!

If you want to complain the first thing you need to do is stop, think, take a deep breath, calm down and think "what do I want to achieve by this complaint". This is especially true for any sort of formal complaint.

We all feel a gut reaction to complain when something is wrong. Some times a complaint serves just to make us feel better, or maybe it goes as far as an apology. Some times it is just to register as a number - i.e. you don't expect your complaint to cause any effect but if you are one of a million people complaining maybe it will change things.

Sometimes there really is no point - like someone that wanted to complain to OFCOM because (apparently) one of my staff was rude to her - that really is getting silly.

The best sort of complaint is one where you actually have a clear objective. Where you clearly spell out what was done wrong, why it was wrong, and what needs to now be done to satisfy your complaint. If you can make that clear in a formal complaint you stand a chance of getting what you want, or at least an apology. You stand a good chance of getting a reply at least as you are being rational (rare these days I know).

But we (personally) just had an odd complaint from my daughter's school that she had been off sick for a whole week.

So I have to wonder - what do they actually aim to achieve by making a complaint to us? What action are they expecting us to take differently in future (the future being two more terms ever in school). i.e. what was the actual point of the complaint letter?

After all, she had flu shots as she does every year. In this case the school sent her home ill, and she ended up having to go to hospital (the school know this) and have antibiotics. She was instructed by a doctor to stay at home for a few days until she stopped coughing... You do not get much more clear cut proper ill than that!

So, clearly their complaint cannot be to stop her taking more time off school - we could not do that - after all they sent her home from school in the first place. So maybe they are concerned that she got ill in the first place and how we could reduce that?

If that is the case the only practical steps I can think off that would help reduce the time she is ill is to take her out of school. Clearly being with so many people all day massively increases the chance of infection.

I wonder if that is what they intend by the complaint? I will have to ask them...

I bet, however, it is someone that thinks they are in a position of power and feels good asserting their authority by sending pointless annoying stroppy letters to parents.

But my rant here is a sort of complaint... All I get from this is that I feel a bit better having posted my rant. I doubt the school will see it... To be honest, from a psychological point of view this works well. I expect half of the benefit of facebook and twitter is allowing people to vent. If I don't post something I get cross, with annoyance bottled up. By posting a rant I feel I have filed away that annoyance nicely on a web page where it can disappear in to obscurity without stressing me out.

Isn't psychology magic some times?!


Really strange key on my keyboard?

A double colon key!

OK this time it is in fact a very strange keyboard.

We think everyone assumed that the IPv6 Buddy was either a joke or a scam. The web site looks like a joke (and is not even on an IPv6 address).

On top of which it appears to be a UK company selling to the US (and UK) in dollars only using paypal, which also seemed iffy.

Whilst the guy that runs it clearly has no idea about raising invoices properly, and no idea on correct product labeling requirements, it seems the keyboards do in fact exist. I have one. It does what it says on the tin!

It is a USB keypad specifically designed to make entry of IPv6 addresses, with CIDR, easy. It has a full hex keyboard, dot, colon, double colon, slash, backspace, tab and enter keys.

Testing the operation it seems the colon and double colon are handled specially in that they send a left shift and colon/semi-colon key press and release sequence on key press (twice for double colon). The other keys work as on a normal keyboard (separate press and release as you work the key) and so auto-repeat as expected.

We may even put them on the A&A site :-)

Say what you mean!

There have been a number of interesting articles about the sign outside Aldwych tube station, e.g. amateur photographer article. The sign reads "Due to their combination of high quality sensor and high resolution, digital SLR cameras are unfortunately not permitted inside the station." which is a strange thing to say.

As many people have commented and tweeted, it is idiotic. It also inconsistent with other statements made that said the ban was professional camera equipment, tripods, and for safety reasons. What is even more daft is that they were allowing high resolution non-DSLR camers and SLR film cameras.

From a safety or slowing-down-the-tour viewpoint a modern DSLR will be able to take pictures much more quickly than most compacts or phone cameras, and cause much less disruption.

What gets me is that this is a typical trend for anyone doing any sort of administration and trying to get people to do what they want (or not do it). Rather than say "we do not want you taking good pictures that would undermine our sales of photographs" which is (what many people guess) the reason for it all - they make up some plausible excuse such as safety or slowing down the tour, etc.

The big problems with such made up excuses, rather than saying what you mean, is that the silly excuse can usually be challenged as stupid or bypassed (e.g. using a film SLRs), defeating the whole object. Indeed, there are a whole type of person (like me) that would go out of our way to technically comply with the silly excuse whilst deliberately defeating the presumed actual intention (by taking a film SLR, or a compact camera on a tripod, etc.).

One problem is that the real reason is usually one that people do not want to own up to. It is possibly not justifiable or objective. People like to invent an excuse that is somehow objective and has some justification instead.

I recall one occasion when I worked for STL and they said that they wanted clean/tidy desks. I am not sure why, or how they would justify it. I expect it was just to look tidy, but that was never clear as the excuse was for security reasons. Of course, as a made-up excuse it was hard for them to elaborate. They did not want to provide locked cabinets for everything (that would cost money). They said that they did not want anyone to be able to casually see anything confidential. They had said that security personnel could confiscate anything confidential that was left visible on desks, so clearly security staff were not the concern. The result was not tidier desks but a standard front page on all printouts saying "non confidential cover page" and the document name. This addressed the stated concern perfectly as nobody would casually see anything but non-confidential cover pages, and the desks could stay as a "I am in the middle of working on something and do not want to waste 10 minutes in the morning getting it all back where I want it", AKA untidy, state.

I suppose it is that people like a sense of self importance in jobs like this. They should take up herding cats rather than try and direct pedants with made up excuses.


Unhooking BRAS

That got you :-)

Looks like our favourite telco are no longer running the silly BRAS rate system on 21CN lines. Not seen a briefing yet, but it was mentioned in an ISP forum some time ago.

Previously the BRAS rate was set by some config which was based on a back end system that sent updates based on changes to sync rate. The system could take hours to apply changes, and the rates were from a set of fixed values (in some cases in 1M steps). This meant people had delays getting updates after a sync change and meant losing up to 1Mb/s because of the steps. There were also issues where the databases did not agree or the updates did not happen.

What seems to be happening now is that the BRAS rate is set based on the sync rate on every connection. We have updated our systems to match this.

So, they have finally unhooked the BRAS on 21CN.



Sick day

So, had some sort of cold or something for a few weeks - general feeling grotty effect and I get the feeling I am not alone. Loads of people off work (and off school) with something similar, but I seem to have it pretty mild. Heck, I felt so rough last night I hardly drank anything!

What the hell I figure I can take a day off work as not feeling all the ticket. The boss won't mind - wait, that's me - ooops.

Never works does it? I have done coding, sorting billing issues, looking at packet dumps, answering questions from accountant, sorting loads of email, facetime to a customer (well, mate) for an hour, as much work as a normal day. And now I have a list of things still to do this evening, as usual.

The only good point really was I managed to not get up until 09:30. Win!

There are times when I remember being employed fondly. Not often, but this may be one of those occasions...


PI space scam?

I am sure I have raised this before but we have recently had cause to discuss this with RIPE NCC, and their answers are rather telling.

Basically PI space is IP address space (in this case IPv4) which has been assigned by RIPE to end organisations or individuals for their use. It is normally assigned in order to let people multi-home their network, but the rules allow for it to be used in contexts that are not even connected to the Internet as such.

Traditionally this has been assigned to end users without imposing any contract terms on those end users, and understandably, especially with IPv4 running out, RIPE decided that there should be a clear contract in place for the PI assignments - either directly with RIPE or with an LIR that have a contract with RIPE.

So far, so good.

The bit i think that is wrong in my view is that they are trying to retrospectively apply this requirement to existing assignees. This is where you get a bit of a catch 22...

Because there is no existing contract, you have no legal basis to force the existing assignees to do anything (such as enter a new contract). It is because of this lack of control that RIPE want to have contracts in place. But if they had the control needed to force such contracts they would have a contract in place already.

I think that is the problem, and what is rather telling is that in discussions with RIPE NCC regarding one of our customers long established PI assignment, RIPE NCC have not been able to answer simple questions such as "what legal basis is there compelling our customer to enter in to a new contract". They just keep spouting that it is RIPE policy (now). They can't even tell me what RIPE policy "at the time the assignment was made" would allow the new policy to be imposed on existing assignees. They are just waffling and waffling. I half expected them to come up with some clear legal basis, but they have not. If there is one, someone tell me (and tell RIPE NCC as they seem not to know).

So, if there is no legal basis for compelling the end user to sign a new contract, what is there? You are left with the simple fact that RIPE could simply remove the route record from their database. That would mean the assignment could not be used on the Internet. So RIPE can cut off the Internet for this customer.

Sorry, but that sounds like a typical scam or protection racket to me. RIPE provide lots of records which are key to the operation of the Internet and in many cases do not charge for them. They do charge for membership (ISPs typically). Many people and organisations provide key parts of the Internet infrastructure with not contract in place with the people that use those services. It is not like we contract with someone to provide root name servers, for example.

So surely, threatening to delete the route record is much the same as blackmail? After all, the assignment has no expiry date - it is an assignment and has been made - that is now a simple historical fact, and is recorded in the database as a record of fact. Is that not the case?

Maybe that is, in this context, completely legal, and blackmail is indeed the legal basis by which RIPE can force existing assignees to enter a new contract. Though it sounds like entering in to a contract under duress to me (i.e. not enforceable).

From the customers point of view, they dealt with us, and we arranged for them to be assigned PI space, job done, contract with us over, all obligations completed... Then some third party (RIPE NCC) is asking them for ongoing money for something they already own and to enter in to a new contract.

This also raises other interesting issues as to why a Dutch company is controlling a key part of a UK business and can just dictate terms and force them to do things? Scarily, I start to see where the ITU are coming from on this now - and that is a tad worrying.

It does make you wonder how much of the Internet has no clear contractual conditions or enforcement. I mean, what is to stop RIPE saying PI shall be £1M/year suddenly? Or what is to stop Whoever runs the root servers deciding they want contracts with all ISPs and charging lots of money? Can't happen? I would have thought it can't happen until this whole PI charging thing came up, and now I wonder...


Major outage, I tell you! MAJOR!

We have a text reporting for major outages. We have lots of monitoring but there is nothing like customers telling you something subtle has broken even when it is 2am. So we have a texting system for major service outages. They text real people, including me!

So you are sat at home, Monday evening, thinking "My broadband has been out since Friday lunchtime, and I don't have a dial tone. This must be a major outage that affects lots of people and somehow my ISP does not know even though it has been over 3 days and I have not mentioned it - I know, I'll text their MSO number to let them know. Who knows, I may even get someone to pause Top Gear to go look in to it."


Please! MSO texts are for major outages.

A phone line fault, or a physical fault on a broadband line, needs an engineer and they will not rush out at 8pm in the evening - they need booking, which can be done just as efficiently at 9am as now, so no need to hassle someone now! This is why we don't do 24 hour support because faults like this are not fixed any more quickly if we did... You can text (not as an MSO) or email and we will get in touch to arrange an engineer.

Now, back to Top Gear.

/me refrains from waiting until 2am to ring end user to arrange the engineer visit - after all, a major outage justifies working at such hours doesn't it?

OK, OK, lets be a bit fairer shall I - and be a tad constructive. It is fair to say that most people with one broadband line cannot tell when it just stops working if it is just them or a major outage, and so do not know if they should alert us or not. That is true, so a few tips :-
  • Most major outages, whether in our network, or far more common somewhere within the carrier network, we know about, and things are being done to fix them. It is worth waiting a few minutes before panicing.
  • If you can get to the internet (e.g. mobile) check the status pages. If nothing on there then go on to the irc channel (via the support pages). If there is a major outage it will be clear on the irc channel and people will say if they have texted already - staff may even be on-line answering questions.
  • If you have multiple lines and they go off together you have a clue something more major may have happened, this is especially true if you have many different sites (as some of our dealers and IT consultant customers have).
  • If you are more technical and can see there are, for example, some routing issues on the internet or something in our network, then that is the ideal time to send an MSO text as it may not have been picked up by our automation. Please do check status pages and irc first still.
  • If the problem has been going on days - it is not an MSO - it is a normal fault report so email or text normally, or call support during the day
  • If the phone line is not working then it is not at all likely to be a major outage (and if it was, it would be a local cabling or exchange issue handled by BT and not us). So just email or text normally, or call support during the day.
Remember, an MSO hassles staff, day or night, and if it is abused it will be removed. It is there to help us and you when there is a major problem. We cannot answer all MSO texts when there is a major outage but will update status pages and irc.

Sorry if I sound a bit harsh on this particular one. I am just gobsmacked that anyone can think a three day old non-dial-tone issue can be an MSO!

Road Kill

After my post http://revk.www.me.uk/2011/11/knee-jerk-reaction-government.html I am pleasantly surprised to find that 5 deaths a day on UK roads is news!

Well done BBC - with an interactive map of road deaths...

"Britain has one of the best records in the world for road casualties. However, hundreds still die on the roads every year. In 2010, the police recorded 1,850 deaths, 22,660 people seriously injured and 184,138 who received light injuries."



Broadband 101

When you are trying to figure out why your Internet does not work you have to understand the basic steps that are involved when it does work and so work out where the problem lies. Of course, in most case, this is the job of the person on tech support asking the right questions to work out what is happening, but it is worth understanding a few of the basics.

1. The actual wires involved: there is normally a copper pair which is carrying phone and broadband. If that is physically screwed up in some way (which could be intermittent interference, broken contacts, poor quality) then that will cause problems. The most severe types - like a simple disconnection - affect the phone line as well, and you have no dial tone - those problems are easy to identify. However various types of interference (which can be audible noise on the phone line, or not) and quality issues can cause problems.

These problems with the wires result in sync problem. Sync (or synchronisation) is the general phrase we use for the operation of the modem (MODulator/DEModulator) which translates data (1's and 0's) in to tones (radio frequencies) on the wire and back again. If the line is broken in some way the modem may not get sync at all (normally shown by a light on the box). It could get sync, but at a very low speed. It could be that it only works when interleaving is enabled and a slow speed. It could be that there is sync, but lots of errors on the line (which the modem can detect).

All of these symptoms relate to the actual wire. They could be anywhere from, and including, the modem (usually inside a router) at the premises, all the way to the modem at the other end (exchange or cabinet). To fix the problem you usually want to start by trying a new modem, and new cables to the telephone socket. Always remember that a power supply fault can cause a modem to be faulty as well so remember to change that. You should try to eliminate the wiring in the premises by trying from the master socket. This usually has a removable front plate which disconnects all extension wiring and has another socket behind.

If none of that works, it could be the wires, in which case an engineer is needed. It could even be the equipment at the other end - but an engineer can confirm that and arrange a lift and shift (moving to other equipment at the exchange end).

It is a very important step to work out if the problem lies with the computer, the wires (modem issue), the back-haul, or the ISP (or even somewhere in the Internet), as this affects who can fix the problem, and how. Getting this initial diagnostic wrong wastes a lot of time and effort for everyone, which is why it is one of the key parts of the training for our technical support staff.

2. If the modem is OK, and in sync, and not showing errors or slow speed or disconnections, then the problem must lie somewhere else. In these cases there is no point in sending an engineer, the problem lies with the data being sent on the wire, not the wire itself (or the modems).

These cases can usually be diagnosed in various ways - a common type of problem is simply getting a username or password wrong (an authentication problem). Tests on the broadband service and at the ISP can usually tell if this is the case. The router will normally say if there appears to be an authentication issue (e.g. on a web interface). These can however be everything from a simple typo by the end user to problems in the back-haul system and finally problems at the ISP. A good test here is to try a test login, such as a test user or speed test user which does not involve the ISP. If that does not work then there are issues in the back-haul system and possible the BRAS (Broadband Remote Access Server) configuration. In some cases the ISP can do the same test from the exchange using test systems - where that shows problems the ISP knows it is the back-haul.

If authentication works, then that means there is a working connection from the router all the way to the ISP. Well, normally that is what it means. There is a special case of a default accept which is quite easy to detect. This is where the authentication has been faked because the ISP cannot be contacted. But generally, if you can authenticate then you are connected to the ISP and you and the ISP can see that.

3. If you connect to the ISP then the back-haul provider is not normally involved in sorting issues. Occasionally there can be issues like mis-configured BRAS causing slow speeds, or some packet loss or latency in the back-haul network. These are usually things the ISP can measure and detect though.

4. Of course the problem may be in the Internet, with the server you are trying to reach. Testing other routes (e.g. other web sites) helps identify if you have a working Internet connection generally.

But, of all of these tests, checking if the problem lies with the wires or backup-haul or ISP is a crucial first step.

So, why the Broadband 101? Well, because our favorite telco are, once again, insisting that they send an engineer to check the sync on an FTTC line that is not working. Yet the line has sync, and we even see two way traffic to the BRAS. The issue is BRAS config (not authenticating and passing to us as the ISP). It is a simple problem with config and a rebuild of the config on the BRAS (which takes a few seconds) would fix it. Sadly their staff have no clue and insist over and over again on sending more and more engineers, all of which confirm there is sync and go away without fixing the issue. Maybe this blog could be a useful reference for them in future...

NOTSCO (Not TOTSCO) One Touch Switching test platform (now launched)

I posted about how inept TOTSCO seem to be, and the call today with them was no improvement. It seems they have test stages... A "simul...