Tuesday, 28 March 2017

The new £1 coin

The new £1 coin comes out today!

It has a lot of nice security features, making it hard to copy. Obviously not impossible to copy, but that is not what is needed. What is needed is something that it is not economical to copy.

All but one of these "features" are explained. They are quite clever, from micro writing to a lenticular feature that appears too change an image depending on the angle (described as "like a hologram").

But there has always been listed this one "special" security feature to make forgeries detectable.

The problem is that it appears to be "secret".

This is bad!

Firstly, it is a concern that they have to keep it secret. That suggests that if it was not secret then a forger could reproduce it. That makes it "security by obscurity" which never works. If a forger could not reproduce it even if they knew the secret, then why keep it a secret?

Secondly, how is anyone supposed to check the coin is not forged if they do not know what this secret security feature is. It may as well not be there if people cannot actually test it.

Thirdly, assuming some people get to know the secret so they can check it, that means the secret has to be distributed to a lot of people, even if it is "only banks", and so that means the secret will definitely get out.

Who thought any sort of "secret" security feature could possibly be a good idea?!?

And as if to prove a point: The secret is out!


Sunday, 26 March 2017

Misleading customers?

Speaking as director of A&A I can say for sure that we absolutely do not want to mislead or trick customers in any way whatsoever, at all, ever.

We actually want to make the products and services we sell "intuitively" reasonable and fair if we can. The sad state of things is that some things have some caveats. The main one being the minimum term on some of our broadband services and so some early termination charges.

In some cases we have taken a pragmatic and statistical view, so FTTC on BT backhaul we sell on 6 month terms even though we are stung with 12 months terms from BT. This is because what we pay BT for the circuit is only part of the cost - it does not cover the bandwidth usage, for example. We feel that overall, even for those ceasing within 12 months, we do not lose out, much, on average, by having a 6 month term and a low monthly early termination charge for remaining term.

Sadly, for the TT FTTC backhaul, not only are we stuck paying a to a 12 month term, we even get stung with a nonsensical "early termination fee" on top of that - even though TTB get the full cost for the term without having to provide service. It is actually more expensive to have service for 355 days than 356 days. It is a crazy situation which we have tried to re-negotiate without success several times. As a result we sell these services with a 12 month term and a higher monthly early termination charge just to make sure we usually cover our costs.

If only Openreach did not do FTTC/FTTP on 12 month min term and if only BTW and TTB did not insist on 12 month min term too, we would not insist on it, simple as that.

However, on the bases that we are stuck with some shit like this, and hence some services we sell that are not as "intuitively fair" as we like, we are really keen to ensure transparency and making sure no customer is fooled or tricked by what we sell or misled, or even misunderstands by mistake.

The web site explains. The order form lists "key contract terms" and requires ticking a box to agree the minimum term and early termination costs. These are the very first of these terms. Not hidden away with all the bureaucratic crap we have to do anyway - but up front, at the top. We then email those terms as an order confirmation just to re-enforce what was agreed.

Recently a customer purchased one of these 12 month term services at a premises where they would be living there for only a few weeks. And as such was shocked at the cost to terminate or move to new address within 12 months.

I was horrified! How did this happen? We try to be so clear. We even have services (with higher install costs) that have only 1 month term that would have been way better. How did this happen?

The customer did speak to sales, and so far I have confirmed that the sales staff have reviewed the calls, but I want to as well. It is all very well saying "you ticked this box", but the whole customer experience matters.

What is the way A&A handle this? To be honest this is rare so comes down to me to decide. What I have said is that if we, in any way, on the phone call (for example), misled the customer, then the customer should not lose out and not pay the early termination. If we said something on the call, we stand by it. If that is wrong, then I have to make sure that staff have the training they need to not make that mistake in future.

But, as it seems to be, assuming we are right, and we were not misleading in a call or website or order form, this is still a big concern. How did this happen, and importantly what can we do to stop any sort of misunderstanding like this in future? Bigger text, bolder font, what?

I take this seriously - we are not here to trick customers. When there are caveats, like minimum terms, we need to be really clear. I really wish we did not have them, but we are not trying top "trap" customers. Customers should stay with us because they want to - because the service is worth every penny. Definitely not because we tricked them.

So my job this week is going through the call recordings, the web site, the order form, and doing all I can to ensure no customer ever has this misunderstanding again. I have offered this customer the option to spread payments (though she declined). If my investigation finds any hint that we misled her in any way, I will refund the early termination charges.

If A&A do not trade on integrity, we should not be trading at all.


So far we have added this extra box above the "Key terms", tweaked wording a bit, reviewed call recordings, and discussed further ways we can improve. Thanks for all the suggestions.

Saturday, 25 March 2017

Was I wrong on the oven?

I am puzzled. I checked the manual for the oven (here).

The instruction is

"Operating the Ovens
Push in and turn the knob controlling the oven to gas mark 9 (Fig.2-22). The oven will light automatically.
Turn the oven knob to the desired gas mark (Fig.2-23)."

OK, no suggestion of delay, or that going to 9 will be a way to get to temperature faster - just to ensure it lights by the look of it. It does not say why.

Fair enough, but I really have to wonder why. The manufacturer of the oven should have a reason for this, surely?

I could understand if there was some aspect of the lighting mechanism that was somehow more reliable if you do this.

But that cannot be the case. The oven has to be able to self light at any gas mark because it can be set for a timer. If you set for a timer, it will not have the "turn to 9 first" aspect. That is why the spark is automatic and not a button like the top burners. So it must have a reliable mechanism to light anyway without turning to 9 first.

Yet, they have put it in the manual.

Perhaps I'll write to them and ask them.

Critical thinking

There are a lot of memes around - ideas and rituals that get passed on to people and spread by word of mouth (or worse, the Internet), even distorting and evolving and they spread.

Some of these are plain silly.

My kids all seem to have got the idea that you should never, ever, set the volume control on your car stereo or TV to an odd number. Why? because it is bad luck! Personally I think it is bad luck to be superstitious.

Some may have had a good reason once.

Our industry is not immune to this - BT would always set Ethernet ports on leased lines to no auto negotiation, fixed full duplex and fixed speed. Why? nobody really knows. Many "CISCO people" say the same. The best explanation I ever got is that when auto negotiation first came in it had a lot of problems and this was the work around. These days that is simply not the case, but the work around persists causing its own problems. A better work around of fixed settings announced by auto-negotiation never seems to get used. Historical problems have led to a meme, one we are stuck with mostly and will be for years to come. To be honest, we have much the same feeling in the office when it comes to spanning tree having been badly bitten by that years ago and with no real reason to think it is actually a problem now, but we don't really want to risk it.

Some may be a basic misunderstanding.

Setting a thermostat to max to get to desired temperature more quickly. This drives me nuts and I posted on how it was even applied to an oven. In the car, if it is cold the temperature gets set to max (like 30 ℃), then it ends up too hot, so it is set to min (16 ℃) and this is repeated! Setting to 21 ℃ does that for you, and gets to 21 ℃ just as fast as if you set to 30 ℃. Somehow people don't understand how things actually work, not realising the binary nature of the output of most thermostat based systems. This "solution" works for them, well, sort of, so they don't question it. Indeed, people refuse to understand why they may be wrong! We had much the same when air-con installers said never use "auto" mode, but could not in any way explain why that was a problem, or, if it was a problem, why manufacturers include the feature. At work the auto mode allows the two control temperatures to be set and to allow a narrow range. At home there is one setting an around ±2 ℃ making an annoying 4 ℃ window. Apart from an "economy" mode making the window wider I have no other control - so I end up using "heat" in winter and "cool" in summer so as to maintain a more controlled temperature. But this was not an explanation the installers had, and would not apply to the more controllable office system they were installing.

Some come with their own explanation.

The first thing I do is ask "why?". This is the critical thinking, and I think I have managed to get my kids to do the same. The odd numbers on volume does rather surprise me, but I think that falls in to "playing a game" rather than a lack of critical thinking. If there is no explanation then I don't do it or pass it on. If we were all like that then stupid memes would die out. However, some memes come packaged with their own plausible explanation. This is clever as it is basically targeting the critical thinkers. Ideally it is an explanation that is plausible but hard to verify. Sadly the explanation may be totally made up, as part of the way memes are spread - someone once asked why and someone made a guess or invented an answer and it stuck. There are examples like "why never boil the kettle twice" which often comes with "because every time you boil it you reduce oxygen, and so boiled twice does not taste as good". See this (with some nice graphs) for a clue why that is almost certainly bullshit. One clue is when the same meme comes with a variety of unrelated "explanations".

Sometimes the explanation is the meme.

Sometimes the whole meme boils down to the made up explanation - and this spreads because people want to be smart and tell someone something they know and the other person did not. Of course, when the explanation is made up, there can be a counter meme to explain why actually that is a common mistake. I suspect there are cases where these exchanges can go on several times.

Even so, memes, these stories and "facts" and rituals that float around, are an interesting topic. I am sure that once upon a time, before books and the Internet, they were essential in ensuring collective knowledge allowed people to learn from other's mistakes.

Friday, 24 March 2017

Thermostats (again)

(Update: Sorry to my wife. According to all of my five kids I am being tactless, again, which I know I can be. But some interesting comments have been made on this post. Perhaps the best is don't argue with the cook!)

Blazing row with my wife, again. This time over gas oven.

She is adamant that you start by setting the oven to max (gas mark 9) first and then down to what you want, (e.g. gas mark 6) to get it to temperature quicker, and apparently the man that came to repair the oven today said the same.

I am sorry, but the gas oven, like most devices using a thermostat, has a binary output (on or off). It is either a low level pilot or full blast. That is it. I am sure some systems work differently, but most thermostats are binary output, as that works, and is simple.

They have a target temperature, a current temperature, some level of internal hysteresis, and they have a binary output. Pretty much end of story. Fact.

Of course, being a bit scientific, I went and checked. I would not want to be stating fact when I am wrong. I turned oven on to gas mark 1, door open. There is an initial low temperature state where output is in "low" pilot light, but after a few seconds it goes to the expected full blast. Changing dial to gas mark 9 does not change level of blast. There clearly are only the two options, pilot/low and full blast. That is it, as expected.

This means even if you only want gas mark 1, selecting gas mark 1 gets to temperature just as fast, at full blast, as selecting gas mark 9 to start with. There simply is no point in this ritual. Indeed, it can lead to the oven being over temperature and having to cool down. I think, for a cook, that is also bad. I am not a cook.

This simple fact about the way thermostats work in so many systems, whether the car air-con, the house air-con, the cooker, whatever, is apparently beyond my wife's comprehension. I don't know why, she is not stupid.

She is apparently sick of me lecturing her! Well, duh, if she finally got this really simple fact of life I would stop doing so. That is sort of how lecturing people when they are wrong on something works! You keep doing it until it sinks in. Well, I think that is the rule. It is so frustrating. As I tried to explain, black is black and white is white and if she was insisting that was not the case I would lecture her until she accepted it. This is not different. It is a simple fact of life, not opinion, not something that is unknown, is is so simple.

Why is this hard?


First off, yes, absolutely, I may be wrong in some (or all) aspects of my understanding of how this thermostat works in this case. Having played around with the oven a bit (not done proper tests with thermocouples and the smart meter measuring gas flow yet) I think I am not, but I may be. Even so, that does not mean what she is saying is right. However, if I am wrong, I am prepared to listen to reasoned arguments and references, and change my view and learn something. I'll even apologise.

What makes it so frustrating is that I have something to back my point of view - I can explain how a thermostat works, and why there is a binary output, and so on. The other side of the argument comes down to "stop lecturing me" rather than any explanation of why I might be wrong. I might indeed be wrong, but that is not the way to have any rational discussion, and that is what winds me up, sorry.

Why would I be wrong? Well, firstly, I don't like to say how something works without knowing how it works. I am sure that happens some times, when I have deduced how something works but got it wrong, but obviously I don't like to be wrong - who does? I am pretty sure I dismantled something with a thermostat in it when I was a kid - so I saw, first hand, how it works. In most cases it is the same today, I am sure, where the output is binary - on or off. This means in the case of an oven you do run at "full blast" until you get to temperature and then stop, and the cycle on/off in some way. Interestingly someone has suggested that this may not be the case, so I am hoping some time to test that. It was suggested there is a linear control near temperature. I can believe that, but it seems unnecessarily complicated and expensive. It is also suggested that the oven will not be fully at the right temperature when the thermostat in one point thinks it is. Even so, getting to temperature by setting a wrong and higher temperature and letting it then cool is likely to take longer over all, so my overall conclusion would be right. The idea of being hotter before opening the door or putting in a cold container is an interesting one, maybe that has some merit.

The additional heater on the thermostat is interesting. I have heard of this before, and I am pretty sure that it effectively reduces or cancels out the temperature hysteresis inherent in a sprung bi-metalic strip based thermostat, possibly even introducing a more controlled time based hysteresis. This would mean the switching between on and off at the target temperature is more rapid even if the temperature is stable thus creating an (albeit slow) pulse width modulation effect to maintain a proper temperature without the temperature swings you would expect with the simple temperature based hysteresis. This is clever, but won't change what happens in the time before you get to the selected temperature - which will be full blast all the way.

In light of the comments so far, I feel I am quite correct in saying that putting it up to 9 to get to temperature fasters is just silly, a meme, and old wives tale, and not needed. Even so, I may have something to learn about the details of how it does work, and will see if I can find the time to test and learn something.


I am not alone :-

OFCOMs plan for automatic compensation for broadband faults

OFCOM have published their proposals for automatic compensation for broadband and phone line faults (here). It is a consultation and we are replying. Even if it goes ahead it will be some time before it comes in to force.

The proposals have some good ideas, and as a consumer it really helps to understand where you stand.

The key aspects are pretty simple...
  1. Delay fixing a total loss fault, £10 per day (after end of second working day after fault report)
  2. Delay providing service (if promised a date), £6 per day
  3. Missed appointment, or cancelled/changed within 24 hours before, £30
This is simple, easy to understand, and not that daft...

The good news for us, as an ISP, is that all three of these are basically someone else's problem. Appointments to visit a customer for any reason are not by us as such, we don't go to customers, we arrange through carriers, which arrange with Openreach. So missed appointments will be 100% down to Openreach. The same is true for delays in provision, and delays in total loss fault repair in most cases. We're careful not to promise an installation date, so the delayed provision pretty much bypasses us, but we may as well pay out if we get compensation from carriers.

The fault stuff is good for us as it is "total loss of service". This is important. There are a lot of grey areas, and a lot of cases of vague faults. We spent a lot of time arguing with carriers about cases of low level packet loss, unexpected throughput issues, extra latency, intermittent service and frequent drops. None of those are "total loss". All of these are, thankfully, out of scope. A total loss of service is generally pretty clear cut. So OFCOM have got their head screwed on there.

In theory a total loss of service could be us, but unlikely for such a thing to ever last for two working days. So basically, this is all down to carriers and Openreach. Yay!

To be honest I like the simplicity of it all. Well done OFCOM. But...

Carriers not forced to pay ISPs!

The biggest issue is that whilst it is clear that every one of these cases is someone else's problem it is us that have to pay out. It is right that we pay out, we have the contract with the end user. But OFCOM think we can negotiate with carriers to get the same compensation from them so not be out of pocket if not our fault, which it won't be. OFCOM clearly have no fucking clue. We may be lucky - we may find the big players manage to beat up BT Wholesale, Talk Talk Business, etc, and they do in fact offer us the same compensation. Given that we struggle to negotiate for BT Wholesale to actually have a process to even fix faults in the first place, let alone define what a fault is, I do not hold out a lot of hope. It it works, and we get the same compensation for the same things, we are fine. We will have an admin cost of recognising and automating the payouts, and checking that we get the compensation from carriers, and disputing with carriers when they have not paid us (there will be disputes, guaranteed). That is a cost, but overall this will be OK, we are good at this stuff. Fingers crossed.

Broadband depends on a phone line!

As you probably know most broadband services needs a working phone line. If the phone line breaks then the broadband cannot work. This is pretty basic stuff, but OFCOM think most services have same provider for both. That may be true on aggregate but for small ISPs it is not, and most of our lines have someone else providing the phone line part. Oddly the phone lines we provide are out of scope as they don't allow calls, so if we did a line for someone else and it failed (killing their broadband), we would have no obligation to pay compensation for the phone line being dead. Crazy or what?!

The way it is worded now, someone can complain of no broadband to us. We look, see PSTN fault, and say "report to phone line provider". That is all we can do. We have no contract with the part of BT doing the phone line. We cannot do anything to fix it. Forcing us to pay money won't change that! The end user has to contact their phone line provider. But according to the proposed rules, if the broadband is not working in two working days the we have to pay out £10 a day until it is, even though not our fault and we have no means to fix it. There is not even an obligation on the end user to report the fault to the phone line provider. They could sit on it, raking in £10 a day, from us, forever!

There has to be an end to payouts at some point?

OFCOM decided on no cap on daily payouts. I understand, but ultimately we could find there are services we cannot provide. The checker may be wrong, the line may be just too long. In rare cases even a fault may not be possible to fix. In these cases there is the option for us, or the customer, to cancel the order or cease the service. But the wording as proposed does not clarify that this stops the daily payments. If we accepted an order and gave a date (based on carriers telling us), and then cannot provide, could we be paying £6 per day forever? Scary!

Using ADR if not compensated - bad idea!

OFCOM propose that ADR should cover cases where a customer thinks automatic compensation is due and not paid. This is a huge problem.

OFCOM have sensible limited faults, for example, to "total loss", which is pretty clear cut. But a customer with an intermittent fault or other speed issue, that takes time to fix, may want compensation. It will be a very easy mistake to make, or a tricky argument over "total loss of service". It could even be someone with a router problem, and needing a new router, feels we should pay when no compensation under this scheme is due.

The problem is that even with many days delay the compensation is way less than the circa £350 cost of an ADR case (even if we win). So the customer simply has to say they feel compensation is due and threaten ADR. Any ISP will have to pay up else pay more for ADR even if they win the case.

So the limit to "total loss" is moot once you factor in the threat of ADR. However, assuming we get compensation from carriers they won't pay us for such cases and we cannot take them to ADR.

Ultimately this extends the compensation to cover every fault, not just total loss, but intermittent and customer router issues, and the ISP pays out with no way to "improve the service" which is what this whole scheme is trying to do.

Post takes time!

The normal process for most faults is to start by eliminating end user equipment. The carriers make us do this, and it is not totally daft. So usual process is send a router out, which takes a working day, at least.

OK, so we do that and it turns out that it really is the broadband at fault. So we report to carrier. But we have already wasted one working day on sending a router.

Even if we negotiate with carriers to pay compensation to us, they will not allow for the wasted working day in such cases, even though they insist we replace equipment first.

This means we have to change our processes and report a fault to carriers at the start, then send router while they are saying "no fault, book an SFI", and then we say, yes, a fault as we have now tried a new router. All a bit iffy, but process needs working out. I can see the carriers being pissed over this as it gives them at least one fewer working days to fix things.

It costs money!

At the end of the day, if we have to pay out more because of all of these gaps we have more costs, and have to consider higher prices. If the carriers have to pay us more to pay out, they may charge us more, and we have to consider higher prices. At the end of the day the consumer pays. Well done with that OFCOM!

Wednesday, 22 March 2017

Keeping Customer Informed

We are starting on a fun project at A&A. Well, to be fair there are a few major projects going on, but this is likely to be one of mine rather than the ops team, at least to start with...

There is this horrid term BT use, "KCI", which is Keeping Customer Informed. They have these stages KCI-1, KCI-2, and so on.

But, in spite of the annoying term, the principle is reasonably good, and we are working on a system for A&A.

The basic concept is that there are a lot of cases where automated systems (usually) need to update a customer on some progress of something - whether an order or a fault or something else. At the moment we have a lot of systems, some of which get KCIs from BT or TT, and trigger events, some from our own systems, some as a result of an action by staff or a customer, etc. There are some consistent systems for some subsets of what we do, but nothing as a whole.

So the plan is to make a new system, a general purpose system, that can easily be bolted in to all of the systems in place of what we do now, and be consistent and helpful to customers.

The first issue we identified is there are two main grades of notice to customers. The simple "short message" type thing such as "order accepted by BT", "appointment booked for the 3rd, 8am-1pm", etc. The second type is more detailed long messages we currently send by email, such as the detailed order confirmations, or notices about open DNS servers, etc.

The short messages can be sent in many ways, and we currently, for some parts of our network, have messages with a choice of SMS, Twitter or email. These are all ideal for the simple short message type notifications.

So the plan is to allow customers to define, at various levels, e.g. a control page login, where they want notifications sent, and maybe even more than one place at once. Also, especially with text messages, time windows such as (8am-8pm Mon-Fri), etc, so not woken by unimportant messages. We may in future be able to extend to Signal, or WhatsApp, or whatever, where there are APIs.

We have to allow for message to be time sensitive, e.g. no point sending a message about an appointment after the appointment has happened. We may have to delay some messages, e.g. if a line is flapping, the line up/down messages get delayed (for both cost and annoyance reasons). We may also have to pair and cancel messages, e.g. if you have texts 8am-8pm but at 3am your line drops and reconnects a minute later then those two messages can cancel out and not be sent at 8am. We have to also consider load and rate limits on things like texts.

Now, when we get to emails it is also a bit fun. These can be used for the simple texts and for longer notices we send. We already try to sign most emails with an automation signature, but we are considering encrypting emails. We have been asked about this by a few customers, and we need a central system to handle this (makes no sense for everyone to have their own keyring). Our ticketing system could do it for us even.

So how would customers register a public key. Well, the plan is they email it to us, and we send them an encrypted email with a link to confirm the email address. Once done, we have a database of customer public keys and email addresses to use for sending email.

That is pretty simple, and the wonders of GPGME library have been impressing me for the last two days.

The huge problem is turning it off. Technically simple, and we can have have a staff interface for that, but the issue is policy. If someone wants encryption and has any risk of emails being read in transit, they do not want someone to just be able to phone up and turn off encrypted emails from us. Indeed, we cannot even sent a link to confirm which is not in fact encrypted to be safe. Sending an encrypted link will work for someone simply wanting to turn it off, but what of when someone loses a key??

Indeed, even accepting a replacement public key is tricky as it could be sent by someone that has means to intercept email, and they can then extract the confirmation link from our reply as they made the key.

Obviously the traditional face to face key signing is not practical on scale.

We could use customer login, 2FA, and so on, but how do we know the email they are using is really them. They could use their account but with someone else's email address which they have means to intercept, even if temporarily.

I am slightly at a loss on best practice on this at the moment. Comments welcome.

I suspect the best we will do is create policy and a good practice which minimises risk, but can never be bullet proof.

No guarantees on timescales yet, and it will be a gradual deployment, but watch this space...

P.S. S/MIME PGP is a pile of shit, IMHO, so far.

P.S. One issue is that we are not expecting to register public keys "per account" or "per login" but "per target email address". We would send an encrypted reply with confirmation link. This makes validating changes or removal more complex.

P.S. After a lot of work on the library, I am the S/MIME king - all working as expected.