Monday, 16 October 2017

Recording

Audio recording of conversations is a tricky business, and call recording is one aspect. The rules and advice and laws have changed. Some aspects are simple telecommunications and "interception" laws, and some can fall in to data protection where the identity of a living individual is apparent from the recording. Even with data protection laws, caveats like "public interest" and "preventing or detecting crime" come in to play. So it is not simple.

We, as a communications provider, sell telephony services where call recording is a standard feature. If you have a number from us even if connected with a mobile SIM, or VoIP phone, we can record calls and email them to you as a standard feature at no extra cost. It is really very useful.

Personally, I record all calls. As a business (A&A) we record all calls. Indeed, for business it is so common it is to be expected and you don't even have to say that calls are recorded (we think).

There are issues with "why" the calls are recorded and "who" gets to access those recordings.

Now, as a service we offer, it is important that our customers understand the rules on the recordings of calls they make or receive.

So later in the year (or next year), in light of GDPR, we need to work it all out. The plan it to make some proper legal advice on call recordings, when and how. I'll be blogging on the matter, and A&A will have advice for customers as much as we can.

At the end of the day, the fact a call was recorded usually only comes up when someone wants to deny what they said, or agreed. Once you get to that the fact you recorded the call is not the issue, it is the fact someone lied, or broke a contract, that matters. They cannot get out of that by saying they did not know the call was recorded. That is saying "If I knew it was recorded I would have told the truth" which is not going to wash with any judge, I suspect.

So watch this space on that...

But there is something weird that happened today. A public body wants a meeting, but their "policy" is (a) you cannot bring a solicitor, and (b) you cannot record the meeting. The second point is odd, well both points are odd, but especially as they say they will be recording the meeting and will send a copy of the recording...

Policy!

They say this is "policy"! Policy is a lovely term and we see it all the time. We have encountered BT policy as a company. We counter such things saying "A&A policy is X". When anyone spouts "policy" they are dictating something as an immutable rule when not considering that the other party may legitimately have their own conflicting "policy" on such matters.

It is my policy to record all meetings... This is one reason it is not me going tomorrow.

Let's record...

So we have pondered some legal points - if all participants of the meeting know it is recorded and know that we will get a copy of the recording, is there any legal impediment to us covertly recording the meeting? I think not... I am not a lawyer, but it is an interesting legal point. Comments?

You also have to wonder why, though? I can think of two reasons. The main one is for them to be able to edit the recording before providing a copy. That is not, in any way, a stated intention, and would be unethical I feel. The other is to hold copyright on the recording - but one could make your own transcript using the recording to ensure accuracy and hold your own copyright on the transcript - so not a useful right to retain. Either way, something wrong with not allowing both parties to make a recording. Neither party making a recording may be a valid thing in some cases, but hard to see why a public body would want such an "off the record" meeting, and they have not said they do. It just makes no sense to refuse us making a recording when they will and provide us with a copy!

So, what do to... We will have two see...

I find myself in one of those situations where I would love to say more - to say which public body, and what is at stake. As you may imagine, doing so at this stage could be a problem legally. But it is an interesting legal point, and I know several legal minds read my blog - so comment away...

What is the law on recording a meeting?

Blip

I'd thought I'd share one of the challenges of my day today - a very minor thing but it shows why some software can be such a nightmare. Maybe I can explain it in a way that is easy enough for non engineers to understand.

Sometimes a computer may be doing something wrong. That happens. One example which customers will have noticed is our "blip graph".

What is wrong is pretty obvious in that it is meant to have red (logouts) and green (login) bits, and until a few minutes ago it was only green. It is not a big deal, or highest priority, which is why I am looking today and not yesterday. We use it mostly to identify issues with the network, so it is useful and did need fixing.

What did you change?

One of the key steps in diagnosis of something like this is to look at what you changed. You then try and see if there is some link between what you changed and what is going wrong. In many cases you can just look at the changes and the error sticks out like a sore thumb.

A perfect example would be if I had, for some reason, been working on the code that creates the blip graph from the database, or if I had been working on the code that puts the blip counts in to the database.

I would be able to look at my change, and wonder why my own testing had not shown the problem as well. There are tools to show me exactly what I changed.

It is also really useful if a problem is reported quickly as you also remember why you changed something and what you were trying to actually do as well.

We changed everything and nothing!

The problem is that we changed everything because we have done a major upgrade on clueless. We have also changed nothing, in that none of the code has been changed, just built on the new machine.

The code that makes the blip graph has not changed, and the code that displays the blip graph has not changed. Clearly the database is working as we have some of the blip graph. Indeed, it really made no sense.

Error logs?

One of the key things that lots of systems have are error logs, and we check these. But there are no errors being reported by the system that generates or displays the blip graphs after the upgrade, and were not in the past. So no clues there...

How did it ever work?

After a lot of digging I have found the cause, and it leads to one of those special things that can so often happen with software. HOW THE F*CK DID THIS EVER WORK?!

The "digging" took quite a few hours, because there simply was no logic to it. Nothing had been changed recently in the code, and no errors showing.

I quickly worked out that the displaying side was probably OK, but the database has zeros for the "logouts". The code to record the data looked the same for both login and logout, so how could it only be recording one side?

The eventual bug was a stupid mistake on my part in the code, written 8 years ago. I was comparing a data and time value with a time field in one case because of a simple typo. For the login side I did not have the same typo. It was subtle.

The problem is that the database server used to (silently) decide that I meant to just compare the time part, and get one with it and "just worked". Now, some change in the date/time logic in the database means that it considers the comparison not to match - though not an error, so it (silently) does nothing, instead.

The fix was therefore very simple, and now we have working blip graphs. Just one of dozens of small things to check today. So, if you do see thinking no quite right on clueless, do let us know.


I hope that gives some insight in to the perils of programming.

Sunday, 15 October 2017

Clueless

I do feel it worth acknowledging the work of the A&A ops team, and especially Jimi and Brucey, for the upgrade today. They are not alone and we have all been involved in the planning for this. Even those not in the ops team have helped out and tested things, and thanks to customers to ongoing feedback.

We have a core server which has logically been the main database and control pages for everything we do for nearly 20 years. It has had many upgrades, but has got to the stage that we really need to do something new and a big upgrade.

A lot of functions are already moved to new servers, with extra redundancy. The database server moved to a cluster of sql servers. Lots of internal VLANs and VPNs. lots of backup servers. And much more we can now move and diversify.

But today was the big upgrade of "clueless".

It is interesting to think how "clueless" has changed over the years - at the start it was very much "the" key database server albeit only for our dialup services and even then accounts were very much separate. Now it covers many more services but is far less critical being mainly a front end for staff and customer use. Even so, it is an important server.

For those that do not know, this is the origin of "clueless" is a cartoon from June 2000.


It is that old in origin. Yes, we have a "pointy" as a test platform for clueless...

The changes are supposed to be simple, but the upgrade is operating system, and apache, and mysql, and, well everything. Apache config has changed enough that despite of a lot of planning and testing it has taken hours of work today to get it right. Scary how many things run on clueless, at least for now.

But all tools and scripts, and there are a lot, needed rebuilding and testing and fixing,

There will be some things not fixed until tomorrow, but the basics are all working and the important things were sorted first. Well done all.

Friday, 13 October 2017

Another little gem in the OFCOM CoP

There is another little gem in the OFCOM Broadband Speed Code of Practice in 2.23

When network infrastructure providers or wholesalers make available the live access line speed that is actually received on the customer's specific line, ISPs must use this as the basis for speed estimates (rather than using an access line speed range for similar lines) in circumstances where they will be using the same infrastructure and access technology to provide service. This must incorporate the measures of contention derived from the testing outlined in paragraph 2.20, and should still take the form of a range, where possible.

So, let's make sense of this. Normally the requirement is to provide a range of estimated speed that are the 20th and the 80th percentile speed of "similar customers", and set a guaranteed minimum of 10th percentile speed. As I say this makes one in ten lines faulty by definition.

But consider one of those random one in ten that are faulty, getting service. They complain. The ISP "canna change the laws of physics captain" and it gets no better, so the customer gets a refund and leaves to another ISP.

So new ISP ideally gets to see the sync speed, or gets from a carrier new speed figures based on the carrier knowing the actual sync speed. This gives a few problems :-
  1. Knowing the new sync speed it is still necessary to report a "range" ("where possible"). Well, the only range allowed is 20th and 80th percentiles, but this is a sample size of one! The 20th and 80th percentiles are the actual sync speed of that one sample. How could a range be given? What are the rules for working out that range. I can only assume it is going to be not possible, or the range will have to use some other, perhaps saner, criteria than percentiles.
  2. Assuming the ISP just makes shit up and picks a range from below the actual sync to above the actual sync in some arbitrary and undefined way, and then, of course, picks an arbitrary minimum guaranteed speed that is even lower, what then? Well now the customer migrates to a new ISP, using the same modems and the same line, and getting the same speed. All that has changed is that now they no longer has cause to complain.
This helps the customer how, exactly, OFCOM?
This helps the ISPs or gives them any incentive to change things or invest, how, exactly, OFCOM?

Maybe the existing ISP, on complaint, can offer to "migrate you to us, at not charge, here are your revised speed estimate and guarantee"? Who knows...

Wednesday, 11 October 2017

Small world, it is...

So, my Daughter is in Paphos in Cyprus on holiday along with several others in the family, or as perhaps I should call them "minions" :-)


She just bought something and it came with a silly plastic toy, as things sometimes do...


Well, they looked at the bottom of the toy...


Yes, that is right, Bracknell - which is where we live...


Which is pretty amazing, being that she is over 2,500 miles away.

But you think that is fluke, look at the postcode.

Yes...

RG12 1QS...

That looks familiar...

You may know an ISP whose office is in RG12 1QS. They must be one of the buildings next to us!

(Thanks to James for sending me the pics / story).

Concrete example of 10th percentile issue

Given OFCOMs idea that one in ten lines are faulty it may help to provide a concrete example of the problem here and explain quite how daft this really is. For example, it is not, as some may assume, the slowest 10% of lines in the country.

A friend of mind has a broadband line, it is close to the cabinet, so the forecast sync speeds on 20th to 80th percentile are 79Mb/s to 80Mb/s. He gets 79.912Mb/s. He has no complaints.

The 10th percentile is for "similar lines" and so BT will have banded lines that are that close together and sampled them and looked at the range of speeds that such lines can get so as to find 10th, 20th and 80th percentile. This means some aggregation. I don't know for sure but this could a band of line lengths 0-500m from cabinet. BT will have done some level of aggregation - we may even be able to find what, but it does not matter for this explanation, so we'll assume 500m line length bands for now.

The 10th percentile is 74Mb/s. This means that lines in that "band", i.e. "similar" lines sync are a range of speeds from below 74Mb/s up to 80Mb/s. Indeed, many would probably sync well above 80Mb/s if the sync was not capped.

One in ten of these lines will get below 74Mb/s - that is the very definition of "10th percentile". Whilst the occasional line will actually have a fault (less likely on such short lines) and still sync at a lower speed, the main reason for being below 74Mb/s will be the line length from the cabinet.

So, assuming this is, say, a 0-500m band it could simply be that everyone over 450m from the cabinet gets less than 74Mb/s, a simple fact that they are a certain line length away. Not something anyone can change.

So imagine such a person at say 490m away is getting 73Mb/s. The line may be perfect. The modem may be perfect. There may be nothing that can be done to make the line better of the sync faster whilst using this technology.

Yet, that person is one of the "one in ten" deemed faulty by OFCOM. They can insist the ISP tries to make the line better, engaging engineering time and effort. They can even insist on a refund. Simply because they are one of the "one in ten of lines" below the 10th percentile for "similar lines".

Now, let's look at their neighbour, who is, say 510m away. They may find they are in a 500m-1km band, and get lumped in with such lines for their forecasts. Being so close to the top end they may be in the top 10th percentile, even though they sync at a lower speed, say 70Mb/s. So their line is not deemed to be "faulty". Indeed, they could find themselves with a 50Mb/s guaranteed minimum, have an actual fault on their line dropping it to 51Mb/s and not be caught by the code of practice.

Please explain to me how this mad system where arbitrary bands of one in ten people at various line lengths (depending on arbitrary choices of "similar" line groupings BT do) are to be deemed to be faulty is meant to actually help consumers? It is not like these people are more or less likely to have a fault, or that they are good candidates for some changes in technology so as to improve speed, or even that they have "slow" lines, they are simply "one in ten".

A graph may help explain...


Tuesday, 10 October 2017

One in ten UK broadband lines are faulty, says OFCOM ?

We have had this in the past and once again we seem to be facing another (voluntary, phew) broadband speed code of practice from OFCOM.

Our reply to the latest consultation is here (pdf).

But once again the big issue here is that OFCOM consider any lines where the speed is below the 10th percentile of speeds for similar lines to be "faulty".

This means :-
  1. The customer can expect the ISP to try and "fix" the line, taking up to 30 days to do so.
  2. The customer can expect to be allowed to exit with no penalty and to get a refund of upfront costs if not fixed within 30 days.
Now, if the line is actually faulty, as some will be, this is all very reasonable. But the threshold is not a "fault threshold" as determined by measuring the speed of similar lines that are not faulty. It is set to the 10th percentile of speeds of similar lines.

This means OFCOM are defining that one in ten lines are faulty, end of story... In fact, this is a moving target. If some part of those lines are faulty and fixed, all that does is push up that threshold.

In fact, assuming the ISP can get a refund from Openreach or the carrier then it is in their interests NOT TO TRY AND FIX such lines. If they do, they will end up with more and more lines that are NOT FAULTY but below the 10th percentile if they do start fixing the genuinely faulty ones. Those lines simply cannot be "fixed" and so just cause even more hassle for the ISP. An ISP will actually want a load of low speed faulty lines that are not complaining so as to reduce the 10th percentile level.

The problem is that if you are unfortunate enough to be in that bottom 10th percentile, and bear in mind that one in ten people will be, you may well have a service that is indeed doing the best it can and there is no fault whatsoever on the line that can be fixed by anyone.

It is as bad as trying to say that every school should be above average or some such. It makes no sense. Why on earth do OFCOM still insist on this nonsense in the code of practice. Why do so many large ISPs agree with OFCOM by signing up to their code of practice? Are BT plc really saying one in ten of their lines are faulty. I am glad A&A don't say that to be honest.

I wonder if any other countries in the EU or the world publish stats on broadband take up, and how many of those lines they consider faulty. The UK must be leading the way with 10% of all lines being faulty by definition of the regulator.

I have to wonder if there is any other industry in the UK, or in the world, where the regulator defines that one in ten of the things you sell are faulty, regardless of what you do, even to the extent that customers can get a refund on that basis? Imagine if OFWAT defined that the lowest 10th percentile of water pressure was a fault and water companies had 30 days to fix or else refund the customer. This is basically what OFCOM are saying about broadband.

We'd love to sign up to the CoP, as it has many good things, but until this fundamental issue is fixed I don't see how we can. We simply do not agree that one in ten UK broadband lines are faulty, sorry. When there are faults we fix them (whatever speed that means you line gets when faulty).


The fix?

Have the modem providers, e.g. Openreach for most FTTC/VDSL, and BT Wholesale or others for most ADSL, define a realistic "fault threshold" which is the lowest speed for non faulty similar lines. Use that as the reference, and have them guarantee that to the ISPs who can pass on that guarantee to the end users. Not complicated!