Good news for privacy - Investigatory Powers Act vs CJEU

As reported by the BBC, the European Court of Justice has made a ruling that could seriously impact the powers in the Investigatory Powers Act to collect data on everyone in the UK.

The IP Act has provisions, much like the Data Retention and Investigatory Powers Act (DRIPA) it replaces, and the Data Retention Directive (DRD) before it, to retain data about use of communications systems.

The IP Act actually pushes this much further - previously telcos/ISPs could have been asked to retain certain data they processed (e.g. telephone itemised billing records) but could not be required to actually generate data they were not processing. The IP Act allows much more and it has been made clear that the government wish to log usage of the Internet in some detail - down to the level of recording every web site everyone has accessed. This is far more than just retention of data, and would apply to everyone, even those not suspected of any crime.

The good news is that the ruling from the CJEU is that this sort of mass retention of data is not consistent with our basic human rights and EU law. These apply regardless of whether we leave the EU or not.

The BBC article is not ideal in its analysis, and Open Rights Group have a much better analysis (here).

Retention is an invasion of privacy

The key point of argument here is that the UK Government considered that indiscriminate retaining of data should be allowed as long as access to that data was restricted and controlled in a suitable way. However, that is not the case. The court ruled that indiscriminate retaining of data was simply not acceptable. You have to be much more specific about whose data is to be collected to target suspects in a crime.

Only to be used for serious crime

The court also looked at the issue of controls over access to the retained data. Again, this did not go well as the access has to be restricted to only serious crime. The IP Act tries to even redefine serious crime to include things that are not serious, so that will have to change too.

Proper independent authorisation of requests for data

On top of that - the access to the retained data should be approved by an independent body, such as a court, and not simply by the current system of a Designated Senior Officer. This could finally mean we see proper court warrants for access to retained data.

No more secrecy

As I have long said, the secrecy around data retention and collection of data is not really acceptable. The ruling says subjects of access should be told about it once there is no longer a risk of prejudice to the investigation.

We can still catch criminals

None of this stops wire taps (or the Internet equivalent) on suspects in serious crime, set up and accessed with the proper controls. All it stops is the indiscriminate logging of everything we all do on the Internet - and that is a good thing - we are all meant to be innocent until proven guilty, after all.

Read more

Read the ORG article for a lot more useful insight in to this ruling.

IEC18004 QR Codes

I said I had my mojo back :-) Yesterday afternoon I decided to have a bash at writing a QR code encoding library, from scratch.

Yes, this is re-inventing the wheel as there are QR encoding libraries out there. It was fun, and it is always nice to have source code that is ours, especially if we may put it in the FireBrick (I am looking at making the TOTP logic in the FireBrick a lot easier to use).

Thankfully Cliff had already written a Reed/Soloman ECC generation function for me, and has made me a very simple BCH coding function. Whilst I understand Error Correction Code, it really is just beyond me in terms of the maths.

I found a copy of IEC18004 on-line. You normally have to pay for a spec, and I may do so at some point, but the court ruling on reading stuff on-line using your browser makes clear that I am not breaking copyright simply be reading it in my browser - whoever is hosting it is. It is 118 pages long!

What really annoys me about this whole specification is the tables of numbers. Instead of saying that the alignment marks will be evenly spaced with spacing between 16 and 20 units starting on unit 6, or something like that, they have a table that states the positioning for each. I played around and worked out a simple algorithm to work out the table and so did not have to use the table - yay. I double checked my calculations only to find one barcode size does not follow the same logic and is a special case for no apparent reason. Why not just make it a simple algorithm?

You then have the same for the level of ECC coding - rather than say "medium ECC uses X% of the space for ECC words" and work that out for each size, there is a table, for four different ECC levels for 40 different sizes of barcode. Then the number of blocks used for ECC is not something simple like "use more blocks when data encoding size is 32 bytes or more" or something simple, no, again a table, for all four ECC levels and all 40 sizes. It drives me round the bend. It could be one line of C rather than typing and double checking and testing hundreds of numbers in to a table.

Anyway, in the end, I have myself a nice little library that codes in 8 bit, Alphanumeric, or Numeric (not Kanji, but I could add that I guess). It codes the input all in one format only - I may, later, make something to work out optimal coding of the string changing coding in the middle as needed, like I did for the IEC16022 barcode library I wrote years ago, but I suspect there is no point.

It is very useful having QR readers on my phone to test it, and the reference coding in the specification was really useful too. I like specs that do a worked example like that.

All in all a fun little project for a Monday afternoon.

This was published at the time on the A&A site but is now a GitHub project (here). And yes, we did put it in the FireBrick for TOTP.

Boiling a frog, and old age

We know the story of boiling a frog - you start with cold water and gradually make it warmer, that way the frog does not notice and jump out. [who would do that?!]

Well, I have noticed that getting old is like that. Several times now I have discovered a change in my life that only strikes me when fixed. Being diagnosed diabetic was scary as only when I was on medication did I realise how much all the symptoms crept up on me over the year before. These are symptoms I knew to look for and had drummed in to me by my mother since I was a child, and still they eluded me. Mostly tired and thirsty. I had got used to taking a glass of water to bed - which stopped being necessary as soon as I was on medication. Now I am on insulin, and my diabetes is well under control, or so I thought. Indeed, the annual reviews and HbA1c tests are all good.

The latest example is one where, over time, I have realised that whilst generally feeling reasonably heathy, I was going to bed tired sooner, and feeling much more apathetic and doing less work. If I was up at 9pm there was a joke in my family that it was past my bed time. That should have been a clue. I would sleep for like 9 hours a night, and not do a lot of work during the day.

Then I was put on indapamide as my blood pressure was getting higher, as I blogged recently. The 2.5mg dose was too high and I felt like crap, but now on 1.25mg, I feel better. I realise that since I started on the indapamide I am feeling "better". Over the last few weeks I have designed, coded, and deployed the whole 2FA systems for A&A (four separate systems), whilst also coding a load of other stuff including a Monzo API library and a few other things - documenting it all, and testing it all.

To my surprise, I look back at last week, and realise a couple of days ago I was up at 5:30 am and working solidly until 11:30 at night with no problem, only to be up at 6:30 the next day. I am finding I am bored just watching TV or going to bed, and instead am doing stuff. All last night I was designing in my head new code for a feature on the FireBrick which I ended up getting up and documenting first thing this morning. I feel like I have my mojo back.

The issue is not, as I see it, the blood pressure, which is what the indapamide is for, but it has changed the way my diabetes is working - I am having to take more insulin, about twice what I was, but I am much more stable now. It will be interesting to see my HbA1c in a few weeks time. Indapamide is not listed as a treatment in any way for diabetes, just that it can impact blood glucose levels. What is encouraging is that, having mentioned on a recent blog, I am not alone. Others with diabetes found they were "revived" (which I feel is a really good description) once on indapamide. So maybe it should be a diabetes related treatment?

Now I wonder what the next thing will be - something that will creep up on me over many months before I realise.

Change Freeze

Tricky subject, and the very fact the subject has come up means something for the size of A&A now.

We have a change freeze, started this afternoon, and going on until after new year bank holiday.

The principle is fine - we have a lot of staff off, especially some of the senior technical staff, and none of us want major issues whilst we are at home with family if it can be avoided.

So the idea is we don't make major changes or deploy new systems over the change freeze. Nice idea.

There are, however, a few problems, and it is a change for the way I work for a start.

I am very keen to do a job and finish and deploy it - I hate having any job interrupted by a big gap - I lose track of what I am doing and spend a lot of time catching up and things can be missed. So this means that where there is a job in progress before xmas, I have been rushing to make sure it is all deployed before the change freeze. This is not to say taking short cuts as such, but rushing. I don't want a half finished job not deployed. And no, finishing on a dev system and deploying next year is not good - I like to deploy things as I go and pick up issues and fix them whilst still fresh in my mind. If I did the work and did not deploy for two weeks that would be horrid.

We also have the fact that xmas can be a quiet period from a technical point of view - it is (was) an ideal time to deploy and test changes with lower than usual impact. For a start, a whole bunch of customers are not even there - businesses shutting down. And whilst I don't mean to say business customers are more important than residential, there is a difference for a business customer disrupted in their business for a few minutes, or a home customer disrupted whilst eating mince pies. So traditionally the xmas break has been a good time to work on some major projects and iron out the bugs before everyone is back to work or doing anything serious with what we sell.

Ironically, whilst a few months ago, I would almost be happy to sit around doing nothing all xmas break, we now finally have me on medication for my blood pressure which has had some sort of impact on my diabetes which means for last few weeks I feel much more like I was in my 20's, bored of watching TV, and coding from dawn to dusk (well much later with it being winter). Seriously, this is great, even if it won't last (5mg perindopril + 1.25mg indapamide, FTW).

We had only one snag with stuff rushed through yesterday, and it was not actually due to rushing at all, it was a VoIP issue, which is a complicated set of issues where a recent change, which had been tested on several boxes, was deployed as part of an urgent update to address a customer issue. Sadly, when load got to a certain magic level on the live VoIP servers we go drop outs. Our normal testing on several other boxes did not pick it up, and would not have had we not had the change freeze and hence done the update next week instead. Sorry for the inconvenience on that - the VoIP servers are a pain as reloading means dropping calls but waiting means people with dropouts in calls until we do - we managed to move calls and so only drop a few to get the new code deployed during the afternoon.

But overall I feel rushed by the change freeze and not entirely convinced it will help with issues cropping up or not. I guess we'll see over next couple of weeks, if I go crazy, and/or make a huge list of changes all done on Jan 3rd and consequences of that.

If I do have my mojo back, I am damn well going to do stuff, but maybe not A&A stuff. My son has a load of web/app sites that could be tidied up, and my mate Mike has loads of stuff he wants re-inventing from scratch (probably including "the wheel", knowing him). I may find stuff to do.

So, happy freeze everyone.

How not to do 2FA?

We have purchasing cards with Barclays and the statements come in on a web portal.

The site has a secondary question, which I think they pick from various questions that were asked when originally set up.

So, I logged in, and was asked the extra security question, and got it wrong. The problem is that it was "What is your mother's middle name?". This is a horrid question! (a) not everyone has a middle name, and (b) she has two of them. So I forgot what I had put originally and got it wrong.

So, locked out. Great.

I have spent literally a month trying to get it sorted, with email replies taking a week, and eventually a lot of phoning and getting our business relationship manager to chase, finally, the login was reactivated. Not a good user experience at all.

Same question, same mistake, locked out again, arrrrg!

OK, one more time with the shouting and chasing, and what do I get.

Seriously?

Yes, an unsigned, unencrypted, plain text email with a plain text password quoted that is valid for 2 months! (Yes, I have changed it).

Anyway, this time I guessed the right answer to the question.

To be fair, a password reset process is tricky, we send a link valid for a few hours, but that too is as good as plain text in a way as someone could use it. Just seems so very wrong sending a plain text password by email somehow. I am glad we are setting up the proper 2FA stuff on our systems.

Even so, this looked so much like some sort of spam I nearly deleted it.

OFCOM Broadband USO

Arrrrg! This is a knee jerk reaction - I'll do a proper response to them shortly.

OFCOM published guidance to Government on technical specification for the Universal Service Obligation for broadband. Here (PDF).

In it they have three scenarios:-

1. a standard broadband service, characterised only by a 10Mbit/s download speed; 
2. a more highly specified standard broadband service, adding upload speed (1Mbit/s), latency (medium response time), maximum sharing between customers (a ‘contention ratio’ of 50:1), and a defined data cap based on current usage profiles (100GB per month); and 
3. a superfast broadband service, with download speeds of 30Mbit/s, upload of 6Mbit/s, fast response times, a ‘committed information rate’ of 10Mbit/s (i.e. guaranteed 10Mbit/s at all times) and an unlimited usage cap.

Now, I am pleased there is mention of latency, but no mention of the latency being when idle (i.e. not when a queue on the customers own router adds latency), and no mention of packet loss when idle. They also don't define where latency is measured.

But again, they talk of "speed" and "contention" and even "committed" rates.

SPEED TO WHERE?
COMMITTED TO WHERE?
CONTENTION TO WHERE?

Speed (of what?, and to where?)

So, let's start with speed. In the industry and since the days of modems, this has always been the modem link speed (or radio link, etc). It is the raw speed of the link (and usually before overheads like ATM, but that is just a matter of percentage adjustments). It is the speed from the modem at the customer premises to or from the modem at the other end of a wire or radio link.

This speed is important as it was usually the key factor in the user experience - but times have changed. These days users may find that they have an 80Mb/s VDSL sync, but cannot download some film or game at more than a few Mb/s. This could be down to congestion from cabinet to exchange, from exchange to BRAS, from BRAS to ISP, from ISP to internet, within the internet backhaul, or at the serving end. Only some of these factors are within the ISPs control, and some are within the back-haul carriers control. Often you find you cannot fill your 80Mb/s because of other factors. So are OFCOM actually only talking of sync speed still?

They do mention speed as between ISP access network and premises. But this is not a lot of help, as one could have the "ISP access network" in the exchange and/or contention within the ISP. If they are talking of Openreach here, it would be to the exchange only. They ignore congestion in the Internet back-haul or at the serving end of communications. They also talk of it varying depending on contention in the network when actually the issue is congestion. They also talk of it varying depending on home wiring - but that is not between the ISP and consumer premises - but within the premises and not ISPs job!

Committed speed?

This is special. The back-haul providers may offer committed speeds in parts of their network, and will charge a lot for it. In practice it is simply not needed as both BT and TT back-haul are generally not congested. But you, once again, have to ask where this 10Mb/s is committed too?

On the modem to modem link, if the sync is over 10Mb/s then you have a committed speed of that sync all the way to the other end of the wire where the other modem is, simple. Does that meet the requirements, or does this commitment have to go further?

What of to the ISP? Well, as an ISP we have lots of people on what we would call "super fast" links. But we do not buy 10Mb/s times the number of such lines. We buy capacity to avoid being the bottle neck (and we are better than many ISPs in that respect). Where the average download of customers at peak time is around 0.5Mb/s then obviously the capacity we buy is of the order of 0.5Mb/s * number of lines and a bit of headroom. Buying 20 times that will not make anybody's downloads any faster, it just means we charge 20 times as much to cover extra costs (well, not quite, but a lot of the price is the back-haul bandwidth). It makes no financial sense for an ISP to dedicate 10Mb/s on a service where the usage is way less, even if a back-haul provider would offer that as a service. And the back-haul providers have the same model - to get a dedicated 10Mb/s would be stupidly expensive. I may price it up on BT in my reply to OFCOM. But it is silly - why have a service offering that needless costs 20 times what is needed to offer no extra performance and just the nice feeling of links at only 5% utilisation?!

But what then - this dedicated 10Mb/s gets to the ISP. Maybe the ISP even buys transit and peering links to accommodate that, what then. That adds again to the expense - meaning even a small ISP like us would need many 10Gb/s external links - around 20 times what we need for customer traffic.

Well, then there is the Internet. Let's say there is a web site on the Internet. Let's say that server has a 1Gb/s link to the Internet. Let's say we look at UK only, and there are what, 4 million people in UK that can get super fast broadband? If they all have 10Mb/s dedicated, that one web site needs a 40Tb/s link to the Internet to ensure all of the UK super fast users have a dedicated 10Mb/s to that one server...

But that is crazy, clearly. This is not a USO on web sites is it? Well if not, then what is the metric for? Where does that 10Mb/s have to go. End users are buying internet access, so anything short of 10Mb/s dedicated to their favourite website is not actually 10Mb/s dedicated, is it? Is 10Mb/s dedicated to their street cabinet acceptable? This is going to cause customer confusion.

Contention?

Contention is no longer a sensible measure, and even when it was, it only make sense if you specific the two points where the contention is measured.

Specifying 50:1 makes no sense. For a start, it is massively different if you have 50:1 or 5000:100. If you have 10Mb/s and there are 49 others sharing a 10Mb/s link, then two people downloading at once get 5Mb/s each and see "slow" Internet. If there are 5000 people sharing a Gb/s link, then you need 100 other people downloading at 10Mb/s to slow your link 1%, so access seems much faster.

In practice, what matters is a capacity for each end user that reflects normal end user access rates at peak times, e.g. 500kb/s (as it seems now). That means you normally have uncontested links. That measure needs to change over time. You need links at least a number of times the size of the end user min size, so if 10Mb/s is min, then links that are 1Gb/s. Specifying contention is not the way to do it.

But also, where is that contention measured. Once again, modem to modem is 1:1 and not shared. Cabinet to exchange will have some contention, but if not congested the contention does not matter. Contention in backhaul and back to ISP is another factor.

Also, with line speeds going up so much - contention is more problematic. If a line is 80Mb/s at 50:1 then that means an average capacity per user of 1.6Mb/s per user, when in fact all you need us around 0.5Mb/s. But if users are 10Mb/s you only need 0.2Mb/s per user, which is poor these days in the netflix generation.

We still come back to this being an Internet service and so that web server with 1G/s link, and accessible by a billion people around the world, what is that contention ratio?

They also talk of contention "at a node" but do not say at what node, or why they have picked that node rather than any of the other points elsewhere between some Facebook server and the end users laptop. Again, user confusion.

Conclusion

It does look like they are talking just access network broadband and maybe back-haul here, but I do not think the document makes that at all clear. I assume they hope if good access networks exists then Internet access will simply follow and not be an issue. That may be true in some aspects, but ISP models depend on the pricing of the back-haul, and that is expensive, so some ISP models involve congestion at the ISP. That model is not helped by this USO at all.

I am not sure they are addressing user confusion on this at all - even someone paying (at lot) for 10Mb/s committed rate cannot expect to download 10Mb/s from their favourite web site, but they probably will expect that from the ISP somehow if that is what they are paying for.

Bear in mind, one of the providers here is Openreach, and they only operate from master socket to the exchange. So do they meet a USO if the speed to/from the exchange meets this spec? What of BTW/TT back-haul? To be fair, that is usually fine, but won't meet defined contention or committed speeds without stupidly big links from them to the ISP being mandated somehow.

It needs work!

What we did in the end for A&A 2FA

The system is OATH/TOTP 6 digit 30 second authenticator codes, set up by QR code. We have TRNGs we use for seeds that are 320 bits long.

On the accounts system we have gone for some flexibility. Option to SMS codes instead, but configurable, and configurable trust level to decide when to ask for a code. It is also a seed we hold so staff can ask for a code to check you are who you say you are (a useful feature on phone, irc, web chart, etc).

On the control pages (and the internal staff A&A systems) we have gone for encrypted TOTP seed and no SMS option. The seed is binary data, XOR'd with a stretched Argon2 hash of the password and a seed set for that purpose (i.e. the seed also has a random seed for its encryption), so no way to check you have right answer other than doing the Argon2 hash and checking an authenticator code, so not a shortcut to crack the password hash.

This means that on control pages the password change needs old password if you have 2FA set up, and expects an authenticator code as well. Some staff can override, but they will also look at account settings as part of deciding you are you!

I think, overall, we are doing well. Hashed passwords and 2FA with encrypted 2FA seeds.

There is always more to do, and more security to add, but this is an ongoing process.

Customers can now set up 2FA on A&A accounts and control pages if they wish - have fun.