Friday, 28 April 2017

Slartibartfast tour!

I have a couple of very long standing (aka suffering) mates, and we did a thing last year which we hoped to be the start of an annual thing. We went to LA and Vegas.

Mike organised it and we had 1st class flights to US. That was pretty cool. It made the flight pretty bearable, but still for like 9 hours, was hugely expensive. It was good though. We had fun in LA, and even introduced a barmaid to marmite! We had fun in Vegas. I went hunting a new camera, nightmare, and I probably caused annoyance to my mates in doing so. I have it now - Canon, please don't do a new camera in next three weeks.

It was an awesome holiday, and I was really appreciative that I was not doing the driving (LA to Vegas) to be honest. I also appreciate how much Mike paid, but the whole trip was great fun.

This year, I am carrying on the tradition, and have us booked in a couple of weeks on an 8 day cruise. We were lucky, Mike found the trip after we all tried the web site(s) repeatedly. We assume a cancellation to get this cabin for this price on two weeks lead time.

This is not cheap, and neither was last year. It is each of us saving up to afford this, and then only just. It is all about making something memorable and worth the money. You only live once, and it is really nice if we can afford to live, occasionally. This year a series of favourable events mean I can manage to pay. I suspect Simon is bricking himself for next year, but to be fair it is not about the price but the experience and being with friends for a week or so, and so no pressure :-)

The "cabin" (aka "stateroom") we have booked is fucking awesome, sorry. Hot tub, piano, ten times the size of all but two other cabins on the ship (one of which is the other one like this, there are only two). You could throw parties in this. I'll get pictures and videos. I could not believe our luck when it came up. The web site then made it impossible to book it and I spent well over an hour in Thomas Cook today while the agent there (that we know quite well now) sorted it all. Well done.

It is the Slartibartfast tour - and yes, spell checking understood Slartibartfast! Norwegian fjords. With any luck some northern lights. We'll all be working to some extent - that is the nature of each of us running a business, but the open bar and wifi will help with that. I may be working from the exclusive hot tub with open bar, sorry...

Really I can't wait. It is going to be so cool. My wife is jealous and I just know I'll be paying for that somehow.

"I don't want porn coming in to my home"

As you know, I always get very concerned if any customer has a disconnect with what they expect and what we provide. I take it very seriously and always try to improve how we work to avoid it in future. It does not happen often, but we had one today that was, for want of a different word, "special". I initially assumed it was a wind up even!

The gist of it was that there were a list of things that she did not want coming in to her home including porn, suicide, self harm... She was amazed that "the internet" has not taken down such videos. Who? Apparently "blue wale challenge" is real and "every school in the uk has sent a formal email about it to parents and children have been told about it in assemblies" - really?!? Not the schools round here, sorry.

OK, personally, I think that if her kid's school has told kids in assembly not to google for suicide videos, that would be something to complain about as it would be very irresponsible. Tell a kid not to do something, that works every time, duh!

Though, her kids are apparently quite savvy, as her son "innocently googled some games the other day and found a pretty hard core sex games internet site"... Err, OK... Safe search, anyone? Given a later comment I hope this is a son in late teens... If so, I suspect he knows how to "innocently" google many things by now. Time to talk to your kids about what you may find on the internet I think.

OK, lets be a bit fair here. Not everyone knows how to fact check stories. People do need some help understanding how they can filter content, or even just turn on google safe search on the browsers their kids are using! It is actually quite a concern that parents get very little help in this area - it is bad enough learning about everything you need to know when being a parent but for a whole generation, this stuff is new and complicated. It is not something parents could have been taught in school even.

To be clear: we are more than happy to offer advice, and even set alternative DNS servers as default on the router. We're not irresponsible here. What we do is make sure every customer is well aware that they are buying an unfiltered service with an active choice you cannot ignore on the order form, as well as confirmation in the key terms you have to tick, and on our web site and the order confirmation and the information pack we send.

It is also very important that parents understand that no filtering is 100%, so is a "false sense of security" to some extent. It is also the case that a teenage boy (and girl I expect) will be more than capable of bypassing filters if they want to access something. What you need is education and openness, not cotton wool. Else you create innocent blobs that go out on their own in to the wild world of the internet at 18 and don't know how to handle what they find and probably with nobody to talk to.

But the icing on the cake was the last bit...  "My boys play 18 rated pc games...... all killing and horrrible stuff......Hopefully they will still be able to play these games?"

OK, now I am not sure what to say, sorry... Please, just be a parent!

Sunday, 23 April 2017

The plot thickens...


Today I decided to try something else with the Apple TV. I set up another Apple ID with iTunes store account. I then family shared my normal Apple ID to that.

I logged in as the new Apple ID and played episodes via the family sharing.

First episode fine.

Next episode - wants my iTunes password before playing...

And before someone asks, I also tried fixed IP config IPv4 only already.

Next step: physically swapping for another Apple TV

Update: didn't help - Apple TV using my account asked for iTunes password on playing one of the episodes. Arrrrg!

Friday, 21 April 2017

Cursed apple

So where are we with my Apple TV saga...

They are referring to "engineering". That is it...

Latest things I tried...
  • Change apple ID, so if any machines I forgot are trying to log in then they will not know new apple ID.
  • Only have my Mac, Phone and Apple TV on the new Apple ID
  • Turn 2FA back on
  • Turn off home sharing, just in case
  • Log off iCloud, just in case
  • Again, confirm payment details, and actually buy a film on the Apple TV
Guess what - not helping. Still asking my iTunes password, around every hour, on play of episode, or rewind within it. Randomly will not even show episode with "unexpected error" requiring restart.

This is what makes pirating movies easier! If I pirated movies I would not have this hassle.

Thursday, 20 April 2017

I expected better

I had an interesting exchange with a customer this evening.

His internet has been down for an hour and he texted our "Major Service Outage" number saying so and saying he "expected better".

I'm not sure what to say to that - if we really have a customer that expects that a fault, which they have not even reported, is fixed within an hour, out of office hours... What does that mean? If we really have people with that expectation I am tempted to try and sell to them. I expect it will be many thousands of pounds a month for such a service, with multiple redundant leased lines, diversely routed, and a lot of 24/7 active monitoring and staff. I suspect we could do it, for the right price.

Of course, that is not what we normally sell. So if one of our customers expects that they have a serious misunderstanding as to what they have purchased. I try to be honest in what we sell, so I am worried that someone expected more.

At the end of the day things calmed down. I was unhappy as he "abused" our MSO system. It alerts many staff for what could be a major outage. In the evening, not so bad, but middle of night, not good at all. Either way it is staff that are not at work and trying to spend time with family. The reaction from staff that do get the alert was not good, and I took over and was, I have to say, rude...

As per the web pages on this, I was rude. You are asking for a rude reply if you abuse the MSO system! That is what we say!

It turns out he read of the MSO text details on the status pages, which does not have as many caveats. We need to fix that. It is clear it has to be "multiple lines" and so was clearly abusing the system. It was not as clear as the main web page on this. So we can be clearer, I conceded. We'll fix that. Sorry.

Ironically, this fault looks like it may affect 3 lines. It is even an "incident" that TT have spotted. So ironically not something we need to take any action on as it is all in hand. The issue is that this person did not know it was impacting more lines. Our aggregate systems do not spot three lines going down together as that is too few to recognise as a pattern. Good news is it should be fixed soon.

I hope the mix of rude and reconciliatory messages was the right level in the end. We'll see. At the end of the day, expecting a line issue to be fixed within an hour of *not* reporting a fault is special, in my opinion - and I'd be happy to sell such services for the price they deserve...

Let's hope this line is sorted soon.

Wednesday, 19 April 2017

Talking to apple support

I went through the iTunes email based support, took ages, and no help.

So I called 08001076285, as they suggested, to get support. It is a horrid voice based IVR thing. So I have to say my serial number the first time. This time I said my case number.

Oddly, they are actually very pleasant, so well done on that. I have a dread of calling any tech support normally. I hope our (A&A) tech support does not create the same sense of dread for customers. We have no call gate!

On the first call I explained the issue, and what I had tried. Eventually they suggested one thing I had not ("restoring" via iTunes and a cable). So I have done that now, and hence another call.

I am tempted to publish the call recordings (and annoying hold music). I will have to change my security questions first :-)

Basically, I have tried everything so far...
  • Changing password
  • Turning on two factor authentication
  • Turning off two factor authentication (twice now)
  • Removing all devices from account and starting again
  • Turning off family sharing
  • Factory resetting (many times)
  • Restarting using iPhone, and using manual setup
  • Using cable to "restore" the Apple TV via iTunes
  • Upgrading, many times
Still broken.

They think "something somewhere is asking for an iCloud password, or something" and that is upsetting it.

I removed all devices from my apple ID already, if this is what it is that is mad. I have just logged in to two iMacs and set new passwords. If that solves it Apple are fucking stupid! No way a third party failed login attempt should break my Apple TV. Just broken logic there Apple!!!

P.S. If this works I should do a more controlled test - what if someone just used my apple ID and wrong password, would that break it? If so Apple really are broken!

Tuesday, 18 April 2017

Julian Huppert - Cambridge

I have met a few MPs, and Julian is (was) one of them (see wikipedia).

The party he stands for is the Lib Dems, and the area is Cambridge.

I have met him, and discussed several policy issues with him. Overall he is sensible in my opinion. More so than many MPs and way more so than some MPs or the PM.

He is educated, understands science, and even rides a bike!

I was not really in to politics until some of the more recent stupidity that started to affect me and my customers. So far Julian has been quite sane in the discussions on the many issues that have come up, and so have the party for which he stands.

Now, some would say this sounds like a mediocre endorsement - but please, those that know me -  this is far from mediocre - I do not endorse someone lightly.

Julian is someone with which I feel that I could have a sane debate, and would listen to my views. If I lived in Cambridge, regardless of the party for which he chose to stand, that would be good enough. He should represent his constituents, and I am confident he can do that. He has integrity and sincerity.

Right now, in politics, we have some crazy shit going down, and the Lib Dems actually seem to be the only ones being vaguely sane right now.

Even if you are not massively in to one party or the other, if you live in Cambridge, please do look at the candidates and take Julian seriously. If nothing else, he is one of the few who could be in parliament with some decent education and understanding of science.

So, if you can, vote for Julian in Cambridge. A voice of reason in these troubling times.

Tuesday, 11 April 2017

PGP usage

PGP (Pretty Good Privacy) has been around for quite a while now, and the GNU code for it (GNU Privacy Guard, or GPG) is free.

Amongst other things you can encrypt and/or sign emails using PGP.

Unfortunately it has yet to catch on for a common usage. We use it a lot in A&A. We sign the emails we send in almost all cases and have done for decades (I like how I can say "decades" now when referring to A&A). We are just starting more comprehensive encrypting of emails we send as per another blog post.

But it is still uncommon. It is not properly supported in almost any common email clients. I use thunderbird and there is a good plugin (enigmail) which works well, but still a plug-in. It puzzles me a bit as to why it is not a lot more standard in major email clients yet, after all this time.

Of course, one of the big problems, is the "trust" of keys. There is (deliberately) no central authority. Sadly, a central authority model, like that used for https, is way easier for end users. They could automatically trust an email claiming to be signed by their bank because it would be signed by a chain of authority their email client knows to trust. This is the same as the way you can go to https for you bank and know it is them.

There is an email system for this, S/MIME, but even support for that is complicated and not simply included in major email clients, as far as I can see. It also has the problem that individuals want keys, and a central authority model makes that a pain and probably involves paying to have your key signed.

I do think some organisations could do more to encourage PGP. It would be great if Companies House, for example, would sign company keys as a service that is part of managing company registrations. They already have security measures, and they could use the fact they can trust a signed company email as an added feature in dealing with companies. That may encourage more companies to check signatures, and maybe even use company key signed emails as signatures for contracts.

You still have the issue with individuals, but again, organisations that already do security checks, like banks, could easily include key signing. It would be a way to advertise their bank as a source of trust.

Anyway, enough of solving the problems of the world for a moment, the main reason for writing this is that I have to assume more people are using PGP at last...

How do I know? SPAMMERS!!!

I am seeing more and more spam that includes a PGP signature block or a PGP public key block in the spam email. These are usually broken or bogus, which is silly, but they almost certainly look close enough to get an improved anti-spam checking score, and give an impression of more credibility to people.

That would only happen if people really are using PGP more. So, interesting times.

Of course, if people do use PGP more, then spam checking can start actually checking signatures and trust chains, as part of the scoring. Get enough people using PGP, even if only for signing, and we could ultimately eliminate spam (ha!).

Monday, 10 April 2017

Serious rants at apple now

Progress on iPhone roaming

For whatever reason, the instances of the roaming issue have massively reduced in my house. The main difference was that all APs on same PoE switch, but could be the phase of the moon for all I know at this stage. It is a bugger to track down this one.

This means it is taking days to "catch it in the act". The good news is that this happen last week, and I confirmed there was good signal but no connectivity - no IP or anything even to a device on same AP. So I changed config to be fixed IP.

Today it has happened again and we have learned some concrete details of the problem. Also, it has happened in my study, and so I have the phone in the state, captured, and on charge, sat here. It is not between two APs, so should stay broken.

So what have we learned so far?

The phone was set completely static IPv4 config, so no DHCP. This means the problem is not trigged by the way DHCP works or by the FireBrick or gateway doing DHCP in an odd way - that eliminates a load of possible concerns from previous testing. The fact that many people came forward with the same issue on non FireBricks was also a relief.

The controller for the APs claims the phone is not attached, it shows it was, but that it is not now. This is a clue. The phone thinks it is, and shows full signal. So the underlying issue here is a mismatch so the phone thinks it is associated and the APs think not. This has to be a big step forward and suggests it is the roaming process itself failing somehow.

In this state (perhaps unsurprisingly), even with the fixed config, we cannot get any packets to flow, even to another devices on the same AP (and subnet).

What next?

At this point, I am keeping the phone on charge in here in the broken state as long as possible, and have set up firewall access for Ubiquti engineers to have full access the APs and the controller and see what they can find. I hope they find more clues to the problem, but I appreciate it is tricky with some issues like this.

We're doing all we can to get to the bottom of this.


The phone was in the same state having left it all night. So I started to do monitor-mode wifi dumps on my MacBook as requested (wireshark is working quite well on MacOS now). On the AP in here I did not see the MAC of the iPhone at all. I've sent them the dump anyway.

Sadly, trying to get laptop on another channel to dump that I made a config change to APs, which made the phone spring in to life... That has to be a clue for them I suspect.

  • Not DHCP related
  • Failure mode is phone things associated and AP thinks not
  • We know wifi off/on on phone fixes
  • We know roam to another AP on phone fixed
  • We now know reconfigured of AP (even leaving SSID in place) fixes it
Ubiquiti think that any packet from the phone which thinks it is associated should cause a de-auth from the AP which should cause the phone to re-connect. They can't dump that on the AP, hence monitor mode. Sadly I did not capture any packets from the phone on that channel so not conclusive.

Friday, 7 April 2017

Apple TV has gone terribly wrong for me

First off, my Apple TV appeared not to work - I swapped cables, nothing! So I got a new one, same model. Worked! Yay!

Simple enough, though sadly since then the new one did the same a few days later and I now conclude it may be the port on the TV that is being flaky, so on a new port - bugger. I may have chucked out a working Apple TV. Idiot!

However, that aside, the new apple TV is being odd.

Yes, I signed in to iTunes. I have 2FA set up, but that is not actually the root of this issue as I tested today. That all worked, all my stuff shows.

Problem is that every new episode of something I want to watch, even though already purchased, is asking for my Apple ID password, every fucking time.

To start with it popped up on my iPhone as well, so I had a keypad to enter it, but even that has now stopped.

It is making watching anything on Apple TV unbearable and I have no idea why. I have already removed the old Apple TV off my account. I tried removing 2FA (and have since put back). The best I have now is dictating my password for EVERY FUCKING EPISODE!!!

I have no idea why - I even told the Apple TV not to ask for password for purchases, yet it does, many times, and the still asks every time you try to watch every episode.

I will have to ditch it if this keeps up.

P.S. tried all the suggestions, thank you all for them. Finally removed from iTunes, factory reset, upgraded, and re-set-up. Seems to be being sane now again. But really, what a messed up failure mode.

P.P.S. Bollocks, that was fine for two episodes and then again is asking.

Wednesday, 5 April 2017

Customer Privacy

We have completed the first step in providing customers with extra privacy by encrypting emails to customers if they wish.

This is currently only the accounts system. We are extending it step by step to other systems.

I spent some time working out the best way and I think I have something sensible. On the accounts system, once logged in, you can update contact details by a link on the main page.

This allows contact detail to be updated, including email address, but also allows you to paste in a PGP public key to use.

We don't care what UIDs, emails, or trust there is set in that, as long as not expired or revoked, we will use it to encrypted accounts email to you.

We then email you using new contact details (including encryption key) to confirm, and you have to follow a link. That proves you control the email address and the secret key. Once done the account is updated to use those details from then on. We also email the old details (email and encryption settings) to advise of the change just in case it was not really you!

RevK, thanks, first ISP to use PGP for communication with me (and i work for one) :)

We also allow some controls of emailed content, so text email plus optional PDF and optional XML. You can select PGP/MIME or not (i.e. just signing and encrypting the main body). You can even select if we include a confirmation link in the email or not.

The next step is to cover two main areas - call recordings and KCIs. KCI is Keeping Customer Informed and relates to all the texts/tweets/emails from the control pages. It will take some time to get everything on the control pages moved to KCI.

The principle is likely to the the same - load a key and we will use it.

I think this is an important step for privacy for customers.

P.S. We have had options for a long time on what is emailed, e.g. no itemised bill and no link to get it embedded in the email. This is extra protection to protect the entire contents of the emails. We may add extra layers to protect subjects in due course.

Gravity plating

In sci-fi there is artificial gravity.

But it occurred to me gravity is more complex that just saying the "gravity plating" on your Starship Enterprise is set to 1g.

1g, or 9.80665ms2 is the nominal average on Earth. It is a result of being 6371km (ish) from a centre of gravity of a mass of 5.972 × 1024 kg.

Basically the force is based on a constant times the mass of each object divided by the distance squared. So the force per mass of the object on the surface (i.e. you) is based on this constant times the mass of the earth divided by the radius squared.

But you get the same end result if the mass of the planet on which you are standing is much higher and the radius much lower. If you stood on a super dence object only 1km wide you could experience 1g.

There is a difference though, now your (near) 2m height means that the radius is noticeably different when considering the gravitational force at your feet and your head. You would feel the difference I expect! If you jumped in the air you would quickly have much lower gravitational force - reaching escape velocity would be way easier, surely?

Take it to the extreme, a 1m radius object with enough mass to be 1g at your feet, what is that at 3m from centre of gravity, 1/9th g? Light headed or what - jump and you are gone!

So the gravity plating on starships is not just a matter of being 1g, it is also about the apparent distance involved, surely. But the gravity is not exerted much beyond the ship, if at all. Lots of episodes show this, so it must be the like 1m radius Earth... Emulating a super dense but small structure.

Do all star trek people get used to being so light headed?

If we lived in such a world, would we evolve to be midgets?

These are questions we need to ask, people...

P.S. As per one of the comments, creating gravity by spinning part of the ship has the same issue. You can have a small radius spinning fast or a large radius spinning more slowly. The effect will be 1g at your feet but the change in gravity at your head would depend on the radius. I really had not though that gravity at a point in space has both a force and a rate of change like that, but it is obvious when you think about it! It also means that gravity because you are simply in a box that is accelerating is different yet again, indeed, you should be able to detect that this is not the same as gravity on earth, from inside a sealed box, even if you measure 1g, because it is not different at different heights in the box.

Barclays on-line banking bug

I reported a bug to Barclays months ago and they just ignored me. I have now complained about the time taken to respond to a complaint, and they are looking.

The bug is pretty simple, and very stupid.

The business accounts summary page shows "Last night's balance"

But it is not in fact last night's balance, it is in fact the current cleared balance. The actual last night's balance is shown when you go in to the account.

It is a stupid bug, and should be very easy for them to find and fix.

What got me was , now they are looking in to it, is that they have asked me:-
  1. Computer details - e.g. Windows/Mac and what version of OS
  2. Browser details, and what version
  3. ISP used
Now, this is a web page, via https. Why would the computer or browser matter. At a stretch, they may format the content differently for different browsers or operating systems, perhaps. Seems unlikely to me, but that could be the case, just maybe. So OK, ask that I suppose.

But why ask about the ISP?

Surely they do not think that the ISP can, in any way, influence the content of a secure web site? If they do, then why do they trust the use of secure web sites in the first place. Why ask the question? I have asked why and not told them as I suspect it would confuse them.

Tuesday, 4 April 2017

Next step in AP testing here

I have tried quite hard to get the three APs here to break when using a FireBrick FB2700 as gateway on a separate subnet (i.e. WAN side of FB2700 on my main LAN here).

What we did is move from a set-up that broke on my main LAN, to a separate subnet off the main LAN and a Ubquiti EdgeRouter. That worked! So I tried an FB2700 instead in same set up, and that worked too. So it was splitting off to a separate subnet with some sort of gateway that seemed to fix this somehow (rather than specific choice of gateway equipment).

My working theory was that there must be some network set-up aspect that is somehow triggering this issue (whether that set-up is a bug or error or not). This would account for why FireBricks seem to be a common factor as well as Unifi and Apple. FireBricks are not an off the shelf linux system so have very different default settings, and maybe that leads to the problem set-up to be much more common. Well, it was an idea.

Ubquiti had the problem immediately with an FB2700 that we sent them, so sounds like a default setup with very few changes would trigger it, but it did not do so here. I have now gone through matching settings to the gateway on my main LAN. This includes things like leaving DNS to automatic which announces the FireBrick itself as one DNS server only on each of IPv4 and IPv6. I even set up the extra VLAN for guest WiFi which is separately firewalled but on the same subnet with proxy ARP/ND between the two LANs, just in case that was a trigger somehow. After some days of doing this now, it really is "just working", which is rather frustrating.

So this morning I am back on the main LAN as before. Hopefully this will "break" things once again and hopefully quite quickly. It may be a few days to be sure.

The techies at Ubquiti have advised that a pcap on the actual AP itself may help, so the plan is, when it breaks, leave my phone in the broken state (don't move it) and try and diagnose with pcaps on the APs.

To further diagnose I also plan to set the iPhone with static IPv4 config, as some sort of "DHCP throttling" may supposedly be to blame for this. I have double checked with the other developer on FireBrick, as we have both worked on the DHCP server, and neither of us know of this "feature". However, it is worth investigating every avenue. Previous tests (albeit years ago I expect) showed the issue still happened with no DHCP involved. The problem may have changed since, so I'll repeat those tests to confirm. I'm not going to dismiss any ideas.

In case it is not obvious, when this started, years ago, the first assumption we had is that it has to be the FireBrick at fault, and I spent a long time testing things like static config to eliminate DHCP, and checking packet dumps very carefully for DHCP, ARP, ND, RA, RS protocols to try and find anything that would point to FireBrick as the cause. Only after all of that testing did we raise with Ubiquti.

I'll keep you posted...

P.S. Finally (Thursday) my phone failed, I confirmed even a static config could not send or receive packets, even to a device on same AP. I confirmed roaming to another AP does fix. I am leaving on static IPv4 config now to test.

Monday, 3 April 2017

Working with ubiquiti

This is a separate post as something seems to have kicked off on twitter this morning. And first off I'd like to apologise to Brandon from Ubiquiti for swearing.

Ubiquiti have been very helpful trying to get to the cause of a long standing issue impacting a small number of people, but including myself. It is a very frustrating issue which has led me to consider scrapping using the Unifi APs on more than one occasion, but I do like the Unifi kit and I would like to get this actually resolved and continue selling it.

What do we think we know?
  • This only seems to impact Apple - it is seen on iPhones mostly - not android.
  • This only seems to impact Unifi APs - not seen using other APs yet.
  • This almost always seems to be FireBrick as gateway router (at least one case of not FireBrick)
  • This is a rare situation, with many people using hundreds of Unifi APs with no problem. Similarly lots of people using Apple with no problem. Similarly lots of people using FireBricks with no problem.
  • It seems sticky - when a set up has the issue, it stays. When a set up does not have the issue, that stays OK. It is also very intermittent and can seem to take days to be sure if fixed or not.
  • This seems to be only where IPv6 is on the network, which is one reason most people don't see it, and may also be a reason why cases where an IPv6 friendly router sold by an IPv6 friendly ISP is the most common case we have seen (i.e. why FireBricks in almost all cases).
As I say, Ubquiti have been very helpful - they sent us two switches, and edge router and a security gateway. I was only expecting a switch from what was said, so thank you. It has allowed more testing. We sent an FB2700, which has also allowed more testing. The results are interesting, to say the least.
  • Brandon has advised that using FB2700 they see the problem right away. This is good, we have created a set up with the problem. He confirms that using other gateways he does no see it. So something about the network when using a FireBrick seems to be able to trigger this somehow. Oddly he has also seen up to 60 seconds "delay getting an IP" which is not one we have seen. The problem we have seen is permanent - you lose all IPv4 and IPv6 on a roam (intermittently) and do not get an IP even after 60 seconds, all you see is the 169.254 address for when you don't get a reply. I assume that is not what Brandon was seeing, but actually a "delay", which is rather odd. If it is, then that explains the phantom delay and means he has exactly reproduced the problem.
  • Here, we tried moving all APs on to a unifi switch connected to our main LAN (and using FB6000 as gateway). It did not help. That eliminates the switches I have which could have been messing with multicast or something.
  • So I set up a separate subnet for the APs, connected to a Unifi switch, and that then connected via their EdgeRouter. Sadly I needed help setting up IPv6, but got there, in spite of some of my typos. It seemed to fix things - great.
  • So I changed to using an FB2700 on the same separate subnet and same Unifi switch, just swapping one box, and again it is working. I have made the set up as close to the main LAN as I can, same VLANs etc, and the APs are the same config exactly - not changed.
This means the separate subnet appears to be the fix rather than change of router.

It also means a really simple set up of FB2700, switch, and three APs here just worked, but Brandon, with presumably a similarly simple set up, immediately failed. That would be nice to try and compare.

The roaming also seems to happen, apparently as expected, with no interaction with the gateway. No DHCP or anything, just switches over from one AP to another. So it is hard to see how any gateway can be the cause of the problem.

At this point I am wondering if somehow it is a specific configuration of a network that breaks it - I hesitate to suggest the actual IPs in use somehow. I also wonder if it is something else on the LAN causing this - but that does not fit with Brandon's comments.

Unfortunately we have reached an impasse with Ubquiti - they have been very helpful up until now, and thanks for that. But even though this only happens with their APs, and only happens with Apple products, they have now concluded it must be FireBrick and "So at this point I don't think it's fair for you to ask us to help you resolve this.  In doing so your are asking us to help your company make a competing product, for free." and now "So I'm out. Refuse to interact under such disrespectful terms."

We'll continue to look for the issue. I suspect, when we find it, it will not be something where any finger of blame can be pointed at a single bit of kit. But nice to know the spirit of co-operation is alive and well, up to a point. Thanks for your help so far.

FYI, I don't care that Ubuiti have a "competing product". As an ISP we work with competition all of the time for the greater good. I'd be happy to continue to work together to get to the bottom of this anyway - all of our customers would benefit from that. I will, of course, share our findings, even if we find a bug in something FireBrick is doing.

P.S. My next avenue of investigation is differences in configuration, no matter how small, to try and see if we can find a network set-up difference. It is very likely that a typical (mostly default) FireBrick network will have some notable differences to a typical (mostly default) non FireBrick set-up...

P.P.S. You gotta love it - Brandon has complained to FireBrick about one of their employees (me) swearing at him. This is from the country that actually believes in free speech.

Sunday, 2 April 2017

AAISParty - 20 years old

Victoria did well co-ordinating it all. Pictures (here).

We are obviously interested in feedback - we do not do this sort of thing often.

There are a lot of things to arrange with a party...
  • Food: We had a professional BBQ laid on, but also sweets which went down well (more with the adults than the children I think) and at the end of the evening pizza. We have some sweets left over, but mostly I think we had the right balance and nobody was left hungry.
  • Music: Some music to start then a live band on until the end. What I liked is we had the marquee which was louder and the live band, but the meeting room in unit 2 close enough to hear the music but quiet enough to talk to people. A good set up.
  • Drink: A range from 7up, water, fizzy water, diet coke, coke, fanta, J2O, alcohol free Becks (nobody wanted), Becks, Kopperburg, more cider later in evening, whisky (nobody dared open, sorry), bottled ales, coffee, tea - I think we had everyone covered one way or another.
  • Geek: We had a special event amateur radio call sign GB2OAA set up, and (supervised) use of radios. We had a slide show showing staff and a history of 20 years of A&A which was surprisingly popular. We had WiFi. We had detailed Q&A on how the network works (by me). We had our museum of old routers and phones (and "the internet") as well as showing people bare board FB2900 PCBs. We had a DALEK and Orc!
  • Cuddly toy: Many FireBrick dragons, and the kids that came loved it all.
  • Weather: As ordered, sunny and warm and dry, perfect... Thanks for that deity of your choice.
It was nice to meet some customers.

What could be improved - well the name badges went down well and we managed almost everyone, but a key lesson is not leaving it to eventbrite. We need irc handles and maybe twitter handles on badges next time as well as WiFi details, etc. But they are a souvenir for those that came.

On a personal note - I was worried because of Thomas. We had him on one of the slides even (thanks to my daughter for allowing that). One person said they had not spoken to me since this happened and asked if I was all right. Even with my daughter there, and her partner, nobody made any fuss. Thank you all for that. I was really actually quite worried. Nobody wants to cry in front of a tent full of customers!

Encrypting emails to customers

We are making progress...

The new email library is working, and the accounts systems was updated last week. This was not without some snags. On Friday we had some people unable to open PDF attachments. This turned out to be a quirk with an old exim on the server, which is being replaced soon, and created additional unexpected whitespace in the base64 coding of the PDF. From what I understand whitespace is valid anywhere in base64, so I don't think we were wrong, which explains why the tests with various email clients before hand (Thunderbird, Apple, etc) were fine. It was failing on some Microsoft mail clients though. We fixed that on Friday before main billing run Saturday.

A typo in the fix, or rather after the fix was tested, meant no invoices Saturday, which has been fixed today and all April invoices have now been emailed. One person with an old mutt script broke but otherwise all looking good.

The next step is part of the development work on the KCI system. (KCI is Keep Customer Informed). It will be used for status updates from the control systems tracking orders and faults and appointments. It is not used for invoicing or call recording. However, a key part of the KCI system involves customers registering public keys with us.

Customers will be able to email a public key to us with their email address on one or more of the UIDs. Once done, they will then be able to go to priceless (accounts), or clueless (control pages), and select from one or more public keys we have seen with that email address. That will select the specific key we should use for encrypting email to them.

This should avoid issues with rogue keys that happen to be created with the same email addresses. The customer, once logged in, with any 2FA and so on, will select the specific key to use.

This is important for privacy as invoices can include itemised bills, and call recordings could have sensitive information. The plan is all systems, including the new KCI system, will start using this key management to send encrypted emails, and our existing systems for ordering and faults will move to the new KCI system.

This may take months to complete fully, but should allow a lot more security and privacy for customers.

Friday, 31 March 2017

"Up to" speeds

I have just had a very interesting conversation with a Canadian, moved to UK and looking to get our services.

Step 1 was to say, please, please, please, read the terms carefully - we don't know what is "normal" for you and how that compares to what people in UK expect, so read them and ensure you are happy with the terms... I think that is fair.

But he went on our order system and saw we quoted VDSL at 64Mb/s. The fact I had to explain "Mb/s" is same as "Mbps" is a concern.

What was interesting is we quoted the 64Mb/s and he asked how that compared to Plusnet's "up to 76Mb/s".

This is the crux of the matter. The "up to 76Mb/s" is a general UK wide "up to" for VDSL, I assume. We quoted 64Mb/s for a specific address.

The point I had to make was that for VDSL, the speed will be the same for anyone using the same copper pair and the same modem in the cabinet. The forecasts and estimates are just that and ALL WILL BE THE SAME regardless of the ISP.

Now, we know the backhaul and ISP network matter. AFAIK Plusnet are no that bad. But for forecasts on the last mile VDSL sync speed it does not matter who he goes to.

So finally, having explained that, he asked why buy from us?!

19:44 <RevK> We are better in many other ways, best to ask customers on #A&A for
             unbiased opinion.
19:44 <RevK> We usually have better backhaul - you have looked at VDSL sync speed
19:44 <RevK> We have better technology (working IPv6)
19:44 <RevK> We have better politics (I sat in parliament talking to a committee to 
             help protect the freedoms of our customers)

Thursday, 30 March 2017

Where are we with Unifi and iPhone roaming?

As you will know I have spent a long time trying to understand the issues we see with the Unifi access points and roaming between them using an iPhone.

A&A sell these, and some of their PoE switches as well. We may start selling more stuff in due course. Overall the Ubiquiti stuff is pretty impressive and there is an increasingly large range of devices. The WiFi is technically very good at the hardware level, and we sell in boxes of three even for businesses.

So it is important to us that they work. I also use them at home, and my family treat me as tech support (obviously) so it is important to me if I want a quite life. They were all round this Sunday - we had sort of cancelled Mothering Sunday for obvious reasons, but everyone came round and we had pizza and chatted. They all told me in no uncertain terms that the WiFi here is crap and they even turn off on their phones and use 3G/4G when round the house. They all use iPhones. That really is a bad sign.

I myself spend a lot of my time in my office at home, but whenever I leave for the rest of the house I find I have to turn wifi off and back on. Though, technically, it is far from every time and can even be the odd day with no apparent problem, whilst other days I see many times. The problem is, as always, you remember the times it breaks.

This also makes testing hard - something changes and you watch it, and see you spend all day with no issues and think it fixed, when actually it is just intermittent, still, just as before.

I have an AC Pro and two AC LR in the house, and they are now on latest firmware. I thought that may have helped, but no. We also tried changing switches, and thought that had helped, but no.

The current state is that I have managed to mess with wiring enough in the house to actually have all three APs off a single Ubiquiti EdgeSwitch8 - one of their switches - so as to eliminate the switches as the cause of the issue.

Tip: Some of the Ubiquiti kit is still passive 24V PoE, and their switches are great as they support that, but you have to configure on the switch! It is not automatic as PoE normally is.

We also did tests with just IPv4 on the LAN, only for a few days, but that seemed to just work. This means the current thinking is that it is the IPv6 being present that is causing the issues. It could be some combination of bugs in iPhone, Ubiquiti, and even FireBrick code, for all we know. Reports from others that use this kit say no problems. We did a lot on FireBrick to try and eliminate that as the cause. However, with IPv6 on the LAN, even with IPv4 being static on the iPhone and no DHCP, it can still fail. Setting up DHCPv6 on the LAN does not seem to change things, we normally use just RA/SLAAC.

The symptoms are a sudden lack of connectivity when it roams. For a few seconds the phone may show the old IP addresses, but quickly switches to showing no IPs and then to showing the 169 auto addresses. Wait as long as you like, it is broken. You need to turn WiFi off and on (on the phone) to fix it.

Part of the reason for writing this up again is for the engineers at Ubiquiti - they are trying to fix this. Good news (though I seem to have to poke on twitter to get things progress, sorry guys). They sent me some switches and a router and gateway. Big thank you - nice to eval the kit as some of it we may start selling. We sent them a fully loaded FireBrick FB2700.

At this point the next stage is for me to try and create a setup using their kit as the gateway on the LAN and so doing IPv4 DHCP and IPv6 RA/SLAAC, and see if that breaks still. It is a pain as I cannot exactly replace my router as it is the office router. So I have set up a new IPv4 and IPv6 subnet for WiFi use. Not ideal, but will do for testing.

They, for their part, need to try and set up with a FireBrick to do the same. Can they make it break? Obviously I am on hand to help them set that up.

So setting up the Edge Router. It is a simple set up. No NAT. Fixed IP /24 IPv4 and /64 IPv6 on LAN with DHCP serving IPv4,and RA for SLAAC doing IPv6. On WAN is a simple IPv4 which can be DHCP client or static, and a simple IPv6 which can be SLAAC or static. Obviously need to set IPv6 DNS servers for RA on LAN.

So far I have managed to set up:-
  • Firewall off
  • NAT off
  • Static IPv4 on WAN (a /24 for testing)
  • Gateway route on WAN, can ping out to internet
  • Static IPv6 on WAN (a /64, obviously, from my PI block)
  • Gateway for IPv6 on WAN
  • Static IPv4 on LAN
  • DHCP IPv4 on LAN
  • Static IPv6 on LAN
  • RA on LAN configured by ubiquiti for me
And I am stuck. So waiting on Ubiquiti at this stage. Suffice to say I don't think they are a threat to FireBrick as this is all pretty simple on a FireBrick.

No word on where they are with FireBricks. Obviously keen to help them test the other way around. To be fair, if this is either a bug in FireBrick in some way, or more likely, something we can work around by changing FireBrick in some way, I am more than happy to do the work to make that happen. We have implemented a number of "pragmatic" aspects to the way the FireBrick works (sometimes on a config setting so as to be "standard" by default) and I'd really like this WiFi kit to work...

I think best if I update this post as we make progress for a bit rather than new posts. Let's get to the bottom of this, shall we?

  • From comments, it is not just FireBrick, but is some rare combination of things clearly, and seems to be Ubiquiti APs and iPhones and "something" else.
  • IPv4 gateway not working was user error, I mistyped as for some reason
  • Someone from Ubiquiti, in Austin, Texas, in the middle of the night, is working with me on this now.
  • IPv6 gateway was not working as I was using the zero address in the /64 which the ER had assumed it can have making it a router on the WAN side, which is unexpected. I changed to the ::1 in the /64.
  • Now wifi all on ER not using FireBrick, thanks to guys from Ubiquiti working in middle of the night. Roaming appears to be working, more testing to do. I am being sent a cap of working roaming as seen by ER, and will get same from FireBrick.
  • We now have two interchangeable set-ups. Both on same sets of IPs as a separate subnet for my WiFi run as a LAN side of a router. I have the ubiquiti EdgeRouter set up, and the same set up on an FB2700. At present both seem to "just work" but as I say, this can take a whole to see the fault. I have lots of logging. One clue is that I am sure I have seen the iPhone re-do DHCP on roam, and the current testing (on both set-ups) does not do that - it just flips over to new AP basically seamlessly. So, just more testing for now. If both these "just work" we have to go back and see what else on the main LAN could be upsetting things in any way.
  • This morning (Saturday), still no apparent roaming issues! This is using a FireBrick but on a separate LAN the same as the ER set-up. Again, if the roaming happens without involving the gateway router, no way the FireBrick can be to blame. If it is OK for a few days I look to swap back to main LAN and see if that shows the problem again.
  • Sunday, still using a separate FireBrick as gateway, and have set up the second VLAN that was being used before on it. Still not failing. This makes no sense at all.

Tuesday, 28 March 2017

The new £1 coin

The new £1 coin comes out today!

It has a lot of nice security features, making it hard to copy. Obviously not impossible to copy, but that is not what is needed. What is needed is something that it is not economical to copy.

All but one of these "features" are explained. They are quite clever, from micro writing to a lenticular feature that appears too change an image depending on the angle (described as "like a hologram").

But there has always been listed this one "special" security feature to make forgeries detectable.

The problem is that it appears to be "secret".

This is bad!

Firstly, it is a concern that they have to keep it secret. That suggests that if it was not secret then a forger could reproduce it. That makes it "security by obscurity" which never works. If a forger could not reproduce it even if they knew the secret, then why keep it a secret?

Secondly, how is anyone supposed to check the coin is not forged if they do not know what this secret security feature is. It may as well not be there if people cannot actually test it.

Thirdly, assuming some people get to know the secret so they can check it, that means the secret has to be distributed to a lot of people, even if it is "only banks", and so that means the secret will definitely get out.

Who thought any sort of "secret" security feature could possibly be a good idea?!?

And as if to prove a point: The secret is out!

Sunday, 26 March 2017

Misleading customers?

Speaking as director of A&A I can say for sure that we absolutely do not want to mislead or trick customers in any way whatsoever, at all, ever.

We actually want to make the products and services we sell "intuitively" reasonable and fair if we can. The sad state of things is that some things have some caveats. The main one being the minimum term on some of our broadband services and so some early termination charges.

In some cases we have taken a pragmatic and statistical view, so FTTC on BT backhaul we sell on 6 month terms even though we are stung with 12 months terms from BT. This is because what we pay BT for the circuit is only part of the cost - it does not cover the bandwidth usage, for example. We feel that overall, even for those ceasing within 12 months, we do not lose out, much, on average, by having a 6 month term and a low monthly early termination charge for remaining term.

Sadly, for the TT FTTC backhaul, not only are we stuck paying a to a 12 month term, we even get stung with a nonsensical "early termination fee" on top of that - even though TTB get the full cost for the term without having to provide service. It is actually more expensive to have service for 355 days than 356 days. It is a crazy situation which we have tried to re-negotiate without success several times. As a result we sell these services with a 12 month term and a higher monthly early termination charge just to make sure we usually cover our costs.

If only Openreach did not do FTTC/FTTP on 12 month min term and if only BTW and TTB did not insist on 12 month min term too, we would not insist on it, simple as that.

However, on the bases that we are stuck with some shit like this, and hence some services we sell that are not as "intuitively fair" as we like, we are really keen to ensure transparency and making sure no customer is fooled or tricked by what we sell or misled, or even misunderstands by mistake.

The web site explains. The order form lists "key contract terms" and requires ticking a box to agree the minimum term and early termination costs. These are the very first of these terms. Not hidden away with all the bureaucratic crap we have to do anyway - but up front, at the top. We then email those terms as an order confirmation just to re-enforce what was agreed.

Recently a customer purchased one of these 12 month term services at a premises where they would be living there for only a few weeks. And as such was shocked at the cost to terminate or move to new address within 12 months.

I was horrified! How did this happen? We try to be so clear. We even have services (with higher install costs) that have only 1 month term that would have been way better. How did this happen?

The customer did speak to sales, and so far I have confirmed that the sales staff have reviewed the calls, but I want to as well. It is all very well saying "you ticked this box", but the whole customer experience matters.

What is the way A&A handle this? To be honest this is rare so comes down to me to decide. What I have said is that if we, in any way, on the phone call (for example), misled the customer, then the customer should not lose out and not pay the early termination. If we said something on the call, we stand by it. If that is wrong, then I have to make sure that staff have the training they need to not make that mistake in future.

But, as it seems to be, assuming we are right, and we were not misleading in a call or website or order form, this is still a big concern. How did this happen, and importantly what can we do to stop any sort of misunderstanding like this in future? Bigger text, bolder font, what?

I take this seriously - we are not here to trick customers. When there are caveats, like minimum terms, we need to be really clear. I really wish we did not have them, but we are not trying top "trap" customers. Customers should stay with us because they want to - because the service is worth every penny. Definitely not because we tricked them.

So my job this week is going through the call recordings, the web site, the order form, and doing all I can to ensure no customer ever has this misunderstanding again. I have offered this customer the option to spread payments (though she declined). If my investigation finds any hint that we misled her in any way, I will refund the early termination charges.

If A&A do not trade on integrity, we should not be trading at all.


So far we have added this extra box above the "Key terms", tweaked wording a bit, reviewed call recordings, and discussed further ways we can improve. Thanks for all the suggestions.

Saturday, 25 March 2017

Was I wrong on the oven?

I am puzzled. I checked the manual for the oven (here).

The instruction is

"Operating the Ovens
Push in and turn the knob controlling the oven to gas mark 9 (Fig.2-22). The oven will light automatically.
Turn the oven knob to the desired gas mark (Fig.2-23)."

OK, no suggestion of delay, or that going to 9 will be a way to get to temperature faster - just to ensure it lights by the look of it. It does not say why.

Fair enough, but I really have to wonder why. The manufacturer of the oven should have a reason for this, surely?

I could understand if there was some aspect of the lighting mechanism that was somehow more reliable if you do this.

But that cannot be the case. The oven has to be able to self light at any gas mark because it can be set for a timer. If you set for a timer, it will not have the "turn to 9 first" aspect. That is why the spark is automatic and not a button like the top burners. So it must have a reliable mechanism to light anyway without turning to 9 first.

Yet, they have put it in the manual.

Perhaps I'll write to them and ask them.

Critical thinking

There are a lot of memes around - ideas and rituals that get passed on to people and spread by word of mouth (or worse, the Internet), even distorting and evolving and they spread.

Some of these are plain silly.

My kids all seem to have got the idea that you should never, ever, set the volume control on your car stereo or TV to an odd number. Why? because it is bad luck! Personally I think it is bad luck to be superstitious.

Some may have had a good reason once.

Our industry is not immune to this - BT would always set Ethernet ports on leased lines to no auto negotiation, fixed full duplex and fixed speed. Why? nobody really knows. Many "CISCO people" say the same. The best explanation I ever got is that when auto negotiation first came in it had a lot of problems and this was the work around. These days that is simply not the case, but the work around persists causing its own problems. A better work around of fixed settings announced by auto-negotiation never seems to get used. Historical problems have led to a meme, one we are stuck with mostly and will be for years to come. To be honest, we have much the same feeling in the office when it comes to spanning tree having been badly bitten by that years ago and with no real reason to think it is actually a problem now, but we don't really want to risk it.

Some may be a basic misunderstanding.

Setting a thermostat to max to get to desired temperature more quickly. This drives me nuts and I posted on how it was even applied to an oven. In the car, if it is cold the temperature gets set to max (like 30 ℃), then it ends up too hot, so it is set to min (16 ℃) and this is repeated! Setting to 21 ℃ does that for you, and gets to 21 ℃ just as fast as if you set to 30 ℃. Somehow people don't understand how things actually work, not realising the binary nature of the output of most thermostat based systems. This "solution" works for them, well, sort of, so they don't question it. Indeed, people refuse to understand why they may be wrong! We had much the same when air-con installers said never use "auto" mode, but could not in any way explain why that was a problem, or, if it was a problem, why manufacturers include the feature. At work the auto mode allows the two control temperatures to be set and to allow a narrow range. At home there is one setting an around ±2 ℃ making an annoying 4 ℃ window. Apart from an "economy" mode making the window wider I have no other control - so I end up using "heat" in winter and "cool" in summer so as to maintain a more controlled temperature. But this was not an explanation the installers had, and would not apply to the more controllable office system they were installing.

Some come with their own explanation.

The first thing I do is ask "why?". This is the critical thinking, and I think I have managed to get my kids to do the same. The odd numbers on volume does rather surprise me, but I think that falls in to "playing a game" rather than a lack of critical thinking. If there is no explanation then I don't do it or pass it on. If we were all like that then stupid memes would die out. However, some memes come packaged with their own plausible explanation. This is clever as it is basically targeting the critical thinkers. Ideally it is an explanation that is plausible but hard to verify. Sadly the explanation may be totally made up, as part of the way memes are spread - someone once asked why and someone made a guess or invented an answer and it stuck. There are examples like "why never boil the kettle twice" which often comes with "because every time you boil it you reduce oxygen, and so boiled twice does not taste as good". See this (with some nice graphs) for a clue why that is almost certainly bullshit. One clue is when the same meme comes with a variety of unrelated "explanations".

Sometimes the explanation is the meme.

Sometimes the whole meme boils down to the made up explanation - and this spreads because people want to be smart and tell someone something they know and the other person did not. Of course, when the explanation is made up, there can be a counter meme to explain why actually that is a common mistake. I suspect there are cases where these exchanges can go on several times.

Even so, memes, these stories and "facts" and rituals that float around, are an interesting topic. I am sure that once upon a time, before books and the Internet, they were essential in ensuring collective knowledge allowed people to learn from other's mistakes.

Friday, 24 March 2017

Thermostats (again)

(Update: Sorry to my wife. According to all of my five kids I am being tactless, again, which I know I can be. But some interesting comments have been made on this post. Perhaps the best is don't argue with the cook!)

Blazing row with my wife, again. This time over gas oven.

She is adamant that you start by setting the oven to max (gas mark 9) first and then down to what you want, (e.g. gas mark 6) to get it to temperature quicker, and apparently the man that came to repair the oven today said the same.

I am sorry, but the gas oven, like most devices using a thermostat, has a binary output (on or off). It is either a low level pilot or full blast. That is it. I am sure some systems work differently, but most thermostats are binary output, as that works, and is simple.

They have a target temperature, a current temperature, some level of internal hysteresis, and they have a binary output. Pretty much end of story. Fact.

Of course, being a bit scientific, I went and checked. I would not want to be stating fact when I am wrong. I turned oven on to gas mark 1, door open. There is an initial low temperature state where output is in "low" pilot light, but after a few seconds it goes to the expected full blast. Changing dial to gas mark 9 does not change level of blast. There clearly are only the two options, pilot/low and full blast. That is it, as expected.

This means even if you only want gas mark 1, selecting gas mark 1 gets to temperature just as fast, at full blast, as selecting gas mark 9 to start with. There simply is no point in this ritual. Indeed, it can lead to the oven being over temperature and having to cool down. I think, for a cook, that is also bad. I am not a cook.

This simple fact about the way thermostats work in so many systems, whether the car air-con, the house air-con, the cooker, whatever, is apparently beyond my wife's comprehension. I don't know why, she is not stupid.

She is apparently sick of me lecturing her! Well, duh, if she finally got this really simple fact of life I would stop doing so. That is sort of how lecturing people when they are wrong on something works! You keep doing it until it sinks in. Well, I think that is the rule. It is so frustrating. As I tried to explain, black is black and white is white and if she was insisting that was not the case I would lecture her until she accepted it. This is not different. It is a simple fact of life, not opinion, not something that is unknown, is is so simple.

Why is this hard?


First off, yes, absolutely, I may be wrong in some (or all) aspects of my understanding of how this thermostat works in this case. Having played around with the oven a bit (not done proper tests with thermocouples and the smart meter measuring gas flow yet) I think I am not, but I may be. Even so, that does not mean what she is saying is right. However, if I am wrong, I am prepared to listen to reasoned arguments and references, and change my view and learn something. I'll even apologise.

What makes it so frustrating is that I have something to back my point of view - I can explain how a thermostat works, and why there is a binary output, and so on. The other side of the argument comes down to "stop lecturing me" rather than any explanation of why I might be wrong. I might indeed be wrong, but that is not the way to have any rational discussion, and that is what winds me up, sorry.

Why would I be wrong? Well, firstly, I don't like to say how something works without knowing how it works. I am sure that happens some times, when I have deduced how something works but got it wrong, but obviously I don't like to be wrong - who does? I am pretty sure I dismantled something with a thermostat in it when I was a kid - so I saw, first hand, how it works. In most cases it is the same today, I am sure, where the output is binary - on or off. This means in the case of an oven you do run at "full blast" until you get to temperature and then stop, and the cycle on/off in some way. Interestingly someone has suggested that this may not be the case, so I am hoping some time to test that. It was suggested there is a linear control near temperature. I can believe that, but it seems unnecessarily complicated and expensive. It is also suggested that the oven will not be fully at the right temperature when the thermostat in one point thinks it is. Even so, getting to temperature by setting a wrong and higher temperature and letting it then cool is likely to take longer over all, so my overall conclusion would be right. The idea of being hotter before opening the door or putting in a cold container is an interesting one, maybe that has some merit.

The additional heater on the thermostat is interesting. I have heard of this before, and I am pretty sure that it effectively reduces or cancels out the temperature hysteresis inherent in a sprung bi-metalic strip based thermostat, possibly even introducing a more controlled time based hysteresis. This would mean the switching between on and off at the target temperature is more rapid even if the temperature is stable thus creating an (albeit slow) pulse width modulation effect to maintain a proper temperature without the temperature swings you would expect with the simple temperature based hysteresis. This is clever, but won't change what happens in the time before you get to the selected temperature - which will be full blast all the way.

In light of the comments so far, I feel I am quite correct in saying that putting it up to 9 to get to temperature fasters is just silly, a meme, and old wives tale, and not needed. Even so, I may have something to learn about the details of how it does work, and will see if I can find the time to test and learn something.


I am not alone :-

OFCOMs plan for automatic compensation for broadband faults

OFCOM have published their proposals for automatic compensation for broadband and phone line faults (here). It is a consultation and we are replying. Even if it goes ahead it will be some time before it comes in to force.

The proposals have some good ideas, and as a consumer it really helps to understand where you stand.

The key aspects are pretty simple...
  1. Delay fixing a total loss fault, £10 per day (after end of second working day after fault report)
  2. Delay providing service (if promised a date), £6 per day
  3. Missed appointment, or cancelled/changed within 24 hours before, £30
This is simple, easy to understand, and not that daft...

The good news for us, as an ISP, is that all three of these are basically someone else's problem. Appointments to visit a customer for any reason are not by us as such, we don't go to customers, we arrange through carriers, which arrange with Openreach. So missed appointments will be 100% down to Openreach. The same is true for delays in provision, and delays in total loss fault repair in most cases. We're careful not to promise an installation date, so the delayed provision pretty much bypasses us, but we may as well pay out if we get compensation from carriers.

The fault stuff is good for us as it is "total loss of service". This is important. There are a lot of grey areas, and a lot of cases of vague faults. We spent a lot of time arguing with carriers about cases of low level packet loss, unexpected throughput issues, extra latency, intermittent service and frequent drops. None of those are "total loss". All of these are, thankfully, out of scope. A total loss of service is generally pretty clear cut. So OFCOM have got their head screwed on there.

In theory a total loss of service could be us, but unlikely for such a thing to ever last for two working days. So basically, this is all down to carriers and Openreach. Yay!

To be honest I like the simplicity of it all. Well done OFCOM. But...

Carriers not forced to pay ISPs!

The biggest issue is that whilst it is clear that every one of these cases is someone else's problem it is us that have to pay out. It is right that we pay out, we have the contract with the end user. But OFCOM think we can negotiate with carriers to get the same compensation from them so not be out of pocket if not our fault, which it won't be. OFCOM clearly have no fucking clue. We may be lucky - we may find the big players manage to beat up BT Wholesale, Talk Talk Business, etc, and they do in fact offer us the same compensation. Given that we struggle to negotiate for BT Wholesale to actually have a process to even fix faults in the first place, let alone define what a fault is, I do not hold out a lot of hope. It it works, and we get the same compensation for the same things, we are fine. We will have an admin cost of recognising and automating the payouts, and checking that we get the compensation from carriers, and disputing with carriers when they have not paid us (there will be disputes, guaranteed). That is a cost, but overall this will be OK, we are good at this stuff. Fingers crossed.

Broadband depends on a phone line!

As you probably know most broadband services needs a working phone line. If the phone line breaks then the broadband cannot work. This is pretty basic stuff, but OFCOM think most services have same provider for both. That may be true on aggregate but for small ISPs it is not, and most of our lines have someone else providing the phone line part. Oddly the phone lines we provide are out of scope as they don't allow calls, so if we did a line for someone else and it failed (killing their broadband), we would have no obligation to pay compensation for the phone line being dead. Crazy or what?!

The way it is worded now, someone can complain of no broadband to us. We look, see PSTN fault, and say "report to phone line provider". That is all we can do. We have no contract with the part of BT doing the phone line. We cannot do anything to fix it. Forcing us to pay money won't change that! The end user has to contact their phone line provider. But according to the proposed rules, if the broadband is not working in two working days the we have to pay out £10 a day until it is, even though not our fault and we have no means to fix it. There is not even an obligation on the end user to report the fault to the phone line provider. They could sit on it, raking in £10 a day, from us, forever!

There has to be an end to payouts at some point?

OFCOM decided on no cap on daily payouts. I understand, but ultimately we could find there are services we cannot provide. The checker may be wrong, the line may be just too long. In rare cases even a fault may not be possible to fix. In these cases there is the option for us, or the customer, to cancel the order or cease the service. But the wording as proposed does not clarify that this stops the daily payments. If we accepted an order and gave a date (based on carriers telling us), and then cannot provide, could we be paying £6 per day forever? Scary!

Using ADR if not compensated - bad idea!

OFCOM propose that ADR should cover cases where a customer thinks automatic compensation is due and not paid. This is a huge problem.

OFCOM have sensible limited faults, for example, to "total loss", which is pretty clear cut. But a customer with an intermittent fault or other speed issue, that takes time to fix, may want compensation. It will be a very easy mistake to make, or a tricky argument over "total loss of service". It could even be someone with a router problem, and needing a new router, feels we should pay when no compensation under this scheme is due.

The problem is that even with many days delay the compensation is way less than the circa £350 cost of an ADR case (even if we win). So the customer simply has to say they feel compensation is due and threaten ADR. Any ISP will have to pay up else pay more for ADR even if they win the case.

So the limit to "total loss" is moot once you factor in the threat of ADR. However, assuming we get compensation from carriers they won't pay us for such cases and we cannot take them to ADR.

Ultimately this extends the compensation to cover every fault, not just total loss, but intermittent and customer router issues, and the ISP pays out with no way to "improve the service" which is what this whole scheme is trying to do.

Post takes time!

The normal process for most faults is to start by eliminating end user equipment. The carriers make us do this, and it is not totally daft. So usual process is send a router out, which takes a working day, at least.

OK, so we do that and it turns out that it really is the broadband at fault. So we report to carrier. But we have already wasted one working day on sending a router.

Even if we negotiate with carriers to pay compensation to us, they will not allow for the wasted working day in such cases, even though they insist we replace equipment first.

This means we have to change our processes and report a fault to carriers at the start, then send router while they are saying "no fault, book an SFI", and then we say, yes, a fault as we have now tried a new router. All a bit iffy, but process needs working out. I can see the carriers being pissed over this as it gives them at least one fewer working days to fix things.

It costs money!

At the end of the day, if we have to pay out more because of all of these gaps we have more costs, and have to consider higher prices. If the carriers have to pay us more to pay out, they may charge us more, and we have to consider higher prices. At the end of the day the consumer pays. Well done with that OFCOM!

Wednesday, 22 March 2017

Keeping Customer Informed

We are starting on a fun project at A&A. Well, to be fair there are a few major projects going on, but this is likely to be one of mine rather than the ops team, at least to start with...

There is this horrid term BT use, "KCI", which is Keeping Customer Informed. They have these stages KCI-1, KCI-2, and so on.

But, in spite of the annoying term, the principle is reasonably good, and we are working on a system for A&A.

The basic concept is that there are a lot of cases where automated systems (usually) need to update a customer on some progress of something - whether an order or a fault or something else. At the moment we have a lot of systems, some of which get KCIs from BT or TT, and trigger events, some from our own systems, some as a result of an action by staff or a customer, etc. There are some consistent systems for some subsets of what we do, but nothing as a whole.

So the plan is to make a new system, a general purpose system, that can easily be bolted in to all of the systems in place of what we do now, and be consistent and helpful to customers.

The first issue we identified is there are two main grades of notice to customers. The simple "short message" type thing such as "order accepted by BT", "appointment booked for the 3rd, 8am-1pm", etc. The second type is more detailed long messages we currently send by email, such as the detailed order confirmations, or notices about open DNS servers, etc.

The short messages can be sent in many ways, and we currently, for some parts of our network, have messages with a choice of SMS, Twitter or email. These are all ideal for the simple short message type notifications.

So the plan is to allow customers to define, at various levels, e.g. a control page login, where they want notifications sent, and maybe even more than one place at once. Also, especially with text messages, time windows such as (8am-8pm Mon-Fri), etc, so not woken by unimportant messages. We may in future be able to extend to Signal, or WhatsApp, or whatever, where there are APIs.

We have to allow for message to be time sensitive, e.g. no point sending a message about an appointment after the appointment has happened. We may have to delay some messages, e.g. if a line is flapping, the line up/down messages get delayed (for both cost and annoyance reasons). We may also have to pair and cancel messages, e.g. if you have texts 8am-8pm but at 3am your line drops and reconnects a minute later then those two messages can cancel out and not be sent at 8am. We have to also consider load and rate limits on things like texts.

Now, when we get to emails it is also a bit fun. These can be used for the simple texts and for longer notices we send. We already try to sign most emails with an automation signature, but we are considering encrypting emails. We have been asked about this by a few customers, and we need a central system to handle this (makes no sense for everyone to have their own keyring). Our ticketing system could do it for us even.

So how would customers register a public key. Well, the plan is they email it to us, and we send them an encrypted email with a link to confirm the email address. Once done, we have a database of customer public keys and email addresses to use for sending email.

That is pretty simple, and the wonders of GPGME library have been impressing me for the last two days.

The huge problem is turning it off. Technically simple, and we can have have a staff interface for that, but the issue is policy. If someone wants encryption and has any risk of emails being read in transit, they do not want someone to just be able to phone up and turn off encrypted emails from us. Indeed, we cannot even sent a link to confirm which is not in fact encrypted to be safe. Sending an encrypted link will work for someone simply wanting to turn it off, but what of when someone loses a key??

Indeed, even accepting a replacement public key is tricky as it could be sent by someone that has means to intercept email, and they can then extract the confirmation link from our reply as they made the key.

Obviously the traditional face to face key signing is not practical on scale.

We could use customer login, 2FA, and so on, but how do we know the email they are using is really them. They could use their account but with someone else's email address which they have means to intercept, even if temporarily.

I am slightly at a loss on best practice on this at the moment. Comments welcome.

I suspect the best we will do is create policy and a good practice which minimises risk, but can never be bullet proof.

No guarantees on timescales yet, and it will be a gradual deployment, but watch this space...

P.S. PGP/MIME is a pile of shit, IMHO, so far.

P.S. One issue is that we are not expecting to register public keys "per account" or "per login" but "per target email address". We would send an encrypted reply with confirmation link. This makes validating changes or removal more complex.

P.S. After a lot of work on the library, I am the PGP/MIME king - all working as expected.

Tuesday, 21 March 2017

Proud dad

I have done a few posts like this, and I understand it is a bit much for some people. I appreciate all of the comments, thank you. But it will carry on for a bit I am sure, at least until the coroner's report and funeral. Sorry if it is not your cup of tea. This is new to me, and new to the whole family.

We spoke to the funeral directors and they said neither they, nor the crematorium, charge for an infant funeral. I was happy to pay whatever it costs but it is free, which is not something you expect.

But with all of that I want to say how proud I am of my daughter, and how she is coping. In some ways I think she is coping better than I am. Today she has been home and been in to the nursery that they had set up, with all the presents from the baby shower and all the knitting from my wife. They are both getting there and coping really well. They are so strong.

What really made me realise quite how grown up my daughter is, and I did not realise this until now, is that she has the hard choice over Thomas. After over a day not sleeping, in pain from an emergency C Section, and in the early hours of the morning when his heart stopped for the third time, she had to decide if to try and restart it yet again. She said she realised he was gone. She made the right choice, I am sure, but I struggle to understand how I could have faced such a choice myself. There is nothing more grown up I can imagine for my baby girl, or anyone.

And typing this I have set myself off again... Blurred vision once again.

I hope she does not mind me posting this. I really am proud of her.

Think of the Children (again)

Once again, the nanny state is on about censoring the internet because "Think of the Children".

See ISP review article (here).

I have said it all before, but it it may be worth explaining a bit about what AAISP do here.

Firstly, all of our customers are adults, we do not sell to minors at all. And all broadband customers have actively selected that they want no filtering. So I suspect we comply with even these latest suggestions.

But there is a lot that our adult customers can do to take responsibility for their children using the Internet. Remember, if you have teenage kids that want to access porn, they will always be able to - it is not clear that there is evidence that this does any harm to be honest, and a solution to any harm (such as skewed ideas on relationships) is better education - talk to your kids, explain that porn is fiction just like thrillers on TV, and sci-fi. Talk to them about relationships.

However, for younger children, every computer system these days provides a range of "parental controls". Indeed, sometimes it is hard to set up a machine with these turned off! Use these tools.

Also, you can set up additional free and even paid for tools if you need - just search for them!

We can also help - we can set the default DNS servers on your broadband to be OpenDNS, which allows you to set up DNS level filtering that will help stop young children stumbling across the more dodgy parts of the Internet. If you are unsure, call us and ask for help (or chat on the web page, or irc, or email, or SMS). The only thing we don't do is filtering in the connection we provide to you.

Ultimately, as our customer, you are in control of your computers and have access to a range of tools to help.