Sunday, 29 November 2015

Logging DNS lookups

One of the interesting questions in relation to the Draft Investigatory Powers Bill is whether it would allow a retention order to require an ISP to log DNS lookups.

What is a DNS lookup?

The Domain Name System is a key part of the Internet - its primary use being to convert the names you use on web sites (like to the addresses used within the protocol itself (e.g. 2001:8b0:0:30::51bb:1e51).

It is actually a pretty good distributed database system, and can hold more than simply name to IP address lookups. It can do reverse lookups (IP to name), and hold text records and mail server records, and a number of other record types.

Why would you want to log DNS?

Well, the government have made it clear that they would like to like the web site names people access. Usually, when accessing a web site, before you access it you have to convert the name to an IP address, and hence to a DNS lookup. Trying to extract the name of the web site from the web site access itself it a lot harder than just logging the DNS lookup.

How easy is it to log DNS lookups?

Mostly the ISP runs DNS servers for their customers, and such servers could produce logs. To be honest, that would mean beefing up the servers, as they typically are not logging (it would be a lot of logs). Also it would mean finding a good way to store and search the logs, but it is possible.

What gets a tad more complex is when people do not use the ISPs DNS servers. Normally this is a simple thing to do, and some people use googles or OpenDNS which can provide some parental control filtering. There are ISPs that do not run DNS at all themselves and subcontract it.

However, DNS packets are not encrypted, and are always on the same port, so it is technically possible to log the requests as they go past. This is a headache to do - you cannot easily divert these packets or copy them on a normal router - you have to look at a switch mirror port of all traffic and filter out the DNS packets. The only good news is that you probably do not have to do session tracking, simply catching the DNS replies would allow you to see the (apparent) requester and the answer they got. You'd also get all DNS reflection attack traffic.

Of course, it is easy to see how protocols could advance to allow encrypted DNS lookups, and I am sure that will come.

Should DNS lookup logging be allowed?

This is where it gets tricky! In the telephony world a call to Directory Enquiries is essentially the same function as a DNS lookup - however telcos are not expected to record, listen to, and log the content of that call any more than they can log the content of any other call. So it seems obvious that DNS requests should not be logged.

Will the bill allow DNS lookups to be logged?

The bill tries to define content and meta data (communications data) - which is a complex task. In principle, an "identifier" or data about a communications address is considered meta data and so could be logged. On that basis, maybe they could ask to log the content of these DNS lookups.

The problem is that DNS can be used for more than just a name/IP lookup. Only some types of DNS request will come within that somewhat loose definition of communications data. Any other type of lookup would be "content" which the ISP must definitely not be logging and retaining.

Even more complex is that you do not know for sure that a name/IP lookup is actually be used to look up a protocol address. The IP address returned could be used to signal something else - one common usage is a blacklist lookup. This is using DNS as a database query system, and the reply indicates a yes or no, and not actually an IP address, even though it looks like an IP address is returned.

Ultimately the ISP has no way to know for sure what purpose exists for the DNS lookup - it is simply a database. With that in mind, and with a ban on logging the "content", I do not think any ISP could legally log the content of DNS lookups under a retention order.

How would we know?

One huge problem here is that if this is not clear in the bill, once passed an ISP could be asked to log DNS requests. If they don't appeal that, then end up making and retaining such logs. If that is in fact not allowed (and presumably even one logged request which was not a protocol "identifier" would make it illegal) then that could cause problems. The issue here is nobody knows - the retention order is secret.

Indeed this is a more general issue with the secrecy - the definitions are not crystal clear and if the government decide something is in scope of "communications data" they could include it in a retention order and simply get away with it. One level example was the idea of grabbing from emails the details of calendar events. These seem obvious that they are "content" except the define the time of an event, and that is something that is defined as "communications data" in the bill. The fact that it was within the "content" part of an email may not matter. This is yet another reason that retention orders must not be secret.

What did the Home Office say?

They seemed unsure. As per my written evidence I think this needs spelling out in the bill that DNS lookups must not be logged.

Thursday, 26 November 2015

Snooper's Charter 101 Please share

To all of the normal people that read my blog. I am sorry this is another post on that snooping crap, but do please read it. I'll try and get back to 3D printing daleks or something real soon.

There is a law that is being considered right now, and may be proper law some time next year.

You should care about it! You can help fix it!

It tries to update some of the existing laws, and make legal some of the stuff done by our "intelligence services". You know, James Bond stuff, except they don't just spy on our enemies (who exactly are they?) they spy on us as well.

It also tries to make some new powers to help the police. In theory these might help the police, and in general I am all in favour of helping the police, but it is not that simple.

Might be worth a small bit of history - phone systems. Originally they were a bit mechanical, and even had operators at the start. Charging for calls used a "meter" that clocked up units. That was it. But things got smarter and people understandably wanted to know where all these units of charge came from, so the phone companies started logging the calls you made and created the wonder that is Itemised Phone Bills. We kind of take them for granted now, but I am old enough to remember a time when we did not have them. This was all done for the benefit of the phone company and arguably their customers.

The fun then starts - the police realise that they can ask the phone company (there was only one) for details of phone calls made from a phone. In some cases this is really useful to some investigations. Later they were even able to ask about calls made to a phone, which is also useful. Of course, even before these itemised phone bills they could ask to "wire tap" a line so they could listen in. At one time this really meant connections to the physical line. This was for serious criminal suspects, obviously.

These days it has got more complex. There are mobile phones, and the police can ask where phones were (at least based on cell towers). As time has gone on, the technology to "snoop" on us all has improved a lot.

The big concern is where the line is drawn - how much snooping is too much, and there is a really big fear now that we are getting to that point. There is a bit of a clue when new laws actually have clauses to exclude MPs - even they feel that this would be too far for their comfort. The fact that someone knows the location of your phone, and hence probably you, every minute of the day for the last year is a tad scary.

Where do we not have privacy?

When we are out in public, we expect that the public can see us, and hear us, and know where we are.

This is usually that we only expect a few people can see us, but they can tell others, so overall the idea that there are cameras all over the place is no huge surprise really.

Basically, we don't have an expectation of privacy, that is what "being in public" means.

The laws on photography are also quite clear - as a photographer I can take a picture of pretty much anything and anyone from a public place - I am just recording what I myself am quite legally allowed to see. (Yes, there are a few caveats on that, but not the point here)

But where do we expect privacy?

When we are at home, or pretty much anywhere behind closed doors, we expect privacy.

Now there are those that say "if you have nothing to hide you have nothing to fear", which is, to be frank, bullshit. None of those people want a public web cam in their toilet or bedroom, strangely enough, and they won't tell me their card details and first pet's name either.

So, I think we can agree that whilst some things we do are not basically private where we have no right to privacy, there are places where we can go and things we can do where we expect privacy and to be quite frank we are entitled to it.

So how does this new law cross the line?

These days when in private we may use of technology a lot - phones, computers, TVs, games consoles, and all connected to the Internet. What we do on the Internet says a lot about us.

Now, with phone call records, the content of the call is not logged by the phone company. Unless you are a targeted suspect of a serious crime your calls are not being tapped, or at least should not be.

The problem is that what we do on the Internet is a lot more revealing about us that what phone calls we make. Privacy International have loads on this (here) and a great video on metadata, which is supposed to be the what, when, who, how, but not the content of what you do on the Internet.

The new law wants to collect a lot of this metadata about all of your Internet access. What is worse is that they want your Internet Service Provider to collect it and store it for a year and make it available to the authorities if they ask. Do you trust that your ISP will not get hacked? Even if they are pretty good now, they will become a juicy target for hacking very soon.

Don't they need this to keep us safe?

There are bound to be cases where knowing everything about everyone can help stop a crime, and if that is what you want then we really should go for cameras in your toilet and bedroom. There is a trade off to be had between the rights we enjoy, the way of life we want to live - with that degree of privacy, and with keeping us safe.

But let's try some facts here shall we...
  • Terrorist attacks, one of the main justifications for all of this, remain one of the lowest threats to your life. There are way more people that died from suicide because of changes to the "Fit to work" assessments than died in recent terrorist attacks in Europe. The justification is scaremongering and bogus. Let me be clear - I do not need protecting from terrorists! What I need is protection from heart disease, cancer, and car accidents.
  • The recent terrorist attacks did not lack this data - they had suspects and even had people under surveillance - the area we need to focus on is not "getting the data" it is what we do when we have it. In fact, having more data will make things harder.
But it gets worse - the Internet is just not like the phone network, and the logs they want don't exist. What logs they can get are likely to be unhelpful (they seem confused that a phone does not just connect to twitter, but actually stays connected all day every day). And over time they will get less and less data as changes in the Internet make it more secure (to combat criminals).

It is also true that criminals can cover their tracks with ease. Simply using secure messaging systems like iMessage, but with a bit of googling you can be way more secure. So the real targets, the serious criminals, and the terrorists, can hide already and always will be able to hide.

What can you do?

One is to spread the word - share and repost this blog to your friends. I have a lot of techie friends and they really get this already - what we need is all of the normal people, the non techies, the people fooled by the "Think of the children" news headlines. People need to think - do I really need the government, and worse, my ISP, spying one me?

Secondly, and this is more work, which is why spreading the word is important, contact your MP now. tell them you are unhappy about this. If you really want, look at my other blog posts and you'll find out a lot more, and even how to formally respond to the consultation and evidence processes, as I have done.

You can also contact people like the Open Rights Group, tell them how you feel. Join up, and stand up for some of these last remaining rights which we all enjoy before they get eaten away bit by bit. AAISP are a corporate sponsor. All that is necessary for the triumph of evil is that good men do nothing.

Wednesday, 25 November 2015

Home Office meeting re IPBill

Thanks to the Internet Service Provider's Association (ISPA) I got the chance to visit the Home Office yesterday and hear their briefing on the Draft Investigatory Powers Bill and ask lots of questions. There were a number of small ISPs at the meeting. Obviously these are my views as I don't speak for ISPA.

Firstly, as you can imagine, security is pretty tight. There was an X-ray screening, and a two door air-lock entrance thing to get in, and constant escorts, and locking up phones, laptops, and any recording devices on a separate floor before going to the meeting. Obviously I was told to bring photo ID, and as I got to the desk I went to get my driving licence when the receptionist said "Ah, I can see your photo ID" and handed me my visitor's pass and sent me on my way. They even let me keep my pen knife. Yes, I got in on my work's photo ID around my neck, which I printed myself on the work Matica card printer - I could have been anyone!

However, apart from that amusement, things were quite interesting. We asked a lot of questions around data retention - this is one of the main areas of concern for small ISPs as the bill seems to allow an order to retain data that could only be obtained by somewhat expensive deep packet inspection (DPI) equipment. It also does not say we'd get paid for this kit, just that the "contribution" would not be "nil".

What we heard was somewhat "civil servant" waffle, but overall was quite reassuring. They basically said they already have retention orders with the large ISPs under the existing regime, and would expect to serve new orders only on them. They have already discussed with them what they could retain. They even said that an ISP would not be expected to log things for which they don't have the capability, or to log any "third party data", or "over the top services". From what we can tell, the logging of "Internet Connection Records" would come from operators that have web proxies and/or CGNAT equipment. They also said they currently do 100% cost recovery and intend to keep that the same.

Of course, they could not rule anything out. We basically said we need some of that re-assurance on the face of the bill some how (see my written evidence at the end of this post for more details of what I would like). The key points in the bill now are that they do have to consider cost and impact on the ISPs business when making an order, and they do have to consult us first. That should probably rule out doing any DPI stuff on cost grounds. Mind you, after yesterday, I would be surprised if A&A do not have a red flag and "don't go near with a retention order"...

At the start of the briefing the the bill was explained, and we heard a story very similar to Theresa May’s comments along the lines of:-

Consider the case of a teenage girl going missing. At present we can ask her mobile provider for call records before she went missing which could be invaluable to finding her. But for Internet access, all we get is that the Internet was accessed 300 times. What would be useful would be to know she accessed twitter just before she went missing in the same way as we could see she make a phone call

Now, I am sure this is a well practised speech, used many times before. I am sure the response has been nodding of heads and agreement with how important “Internet connection records” are, obviously.

However, I, and other ISPA members immediately pointed out the huge flaw in this argument. If the mobile provider was even able to tell that she had used twitter at all (which is not as easy as it sounds), it would show that the phone had been connected to twitter 24 hours a day, and probably Facebook as well. This is because the very nature of messaging and social media applications is that they stay connected so that they can quickly alert you to messages, calls, or amusing cat videos, without any delay.

This seemed to fool them somewhat and they had no real answer - we were not just nodding and agreeing, and that was unexpected :-)

I asked about Data Protection Act Subject Access Requests for retention data, and they don't know.

We asked if DNS logs might be wanted, and they don't know.

I asked about my canary and if the law could compel me to lie - they could not answer that either.

We asked what an "Internet Connection Record" is meant to be, and they confirmed that it is basically down to what they agree with the ISP when they do the consultation before the make a retention order, and will depend on what the ISP can log. We all expressed concern that the bill makes out that an "Internet Connection Record" is a real "thing" and not just some vague term.

I asked about the gagging clause - not allowed to disclose retention orders, and they said the large ISPs asked for that clause, which makes no sense as they could simply choose not to disclose anything.

I asked if the audio content telephone calls to directory enquires counted as "content" and not "communications data" and if so, the content of DNS packets should be treated the same. They were very non committal on that and I wonder if they will be wanting DNS logging. One ISP there outsources DNS to an American company so would have no logs!

I pointed out that if asked to log email I can simply move email to a foreign email service to avoid the hassle. That caught them out - almost like they have never considered that anyone would do that.

Overall - it looks like small ISPs probably have nothing to worry about, but...
  • We'd like that a lot clearer on the face of the bill
  • None of this addresses the privacy issues, but I have been invited to working group on that in a few weeks.
There is a call for written evidence - here is what I have submitted (pdf).

P.S. No, I did not see Theresa; No, they did not hypnotise me; No, I have not yet wiped my phone after being in their hands for two hours... yet; Yes, they had coffee and biscuits; No, I don't think Theresa is a goa'uld; No we have never been and are not subject to a retention order; No we have no "black boxes" of any colour.

Tuesday, 24 November 2015

Changes to IP Bill?

What changes would I like to see to the Draft Investigatory Powers Bill - particularly with regard to data retention?

Obviously I'd like it dropped, but given the push on this in DRD, and DRIPA, I can see that may be a challenge, so simple changes :-
  • I'd like to see transparency of retention orders - they are not specific to individuals or cases and so have no reason to be secret - however, sharing the details between ISPs helps establish best practice, common solutions, and so on. We need the gagging provisions dropped for these.
  • I'd like to see retention only apply to data which the ISP is already logging to some durable medium, or that is reasonably practical to do so. I.e. existing logs but kept for up to a year. This would greatly simplify what was logged. This does mean that email and VoIP and so on end up kept for a year if logged at present, and if the services are provided in the UK.
  • I'd like to see the "processed or generated" clause be included as per previous regulations, but also "processed" exclude "simple passes through". A definition such as "data is only 'processed' if it is logged already or used in some part of a decision process by the CPs systems". This stops us having to look deeper in to any packet than we already do, and hence avoids the possibly huge cost of DPI equipment, and risk of third party control of such kit and feature creep of logged data.
What would this mean? Well, it would not stop all of the intrusions in to privacy, and it would mean :-
  • Anyone using any UK email server will have their emails logged
  • Anyone using any UK VoIP server will have their calls logged
  • Anyone using a CP that operates a transparent web proxy, as some mobile providers do, will have some of their web pages (not full URL, just site name) logged
This last point appears to be what the government want as far as we can see.

However, it also means that the logging is even easier to bypass. A&A can, for example, stop providing email in the UK and move to a foreign data centre and company - bingo, no hassle with logging. We could do the same with VoIP, but getting it to be on the bills may be harder - perhaps a link to an off shore https that provides the itemised bills. We don't run a web proxy so no logging there. Transparency of orders would allow end users to choose ISP based on the level of snooping without the small extra hassle of having to VPN or Tor everything.

I am not trying to make the provisions useless - IMHO they already are useless, as criminals can use Tor and VPN and many other measures. I am trying to make it easier for normal innocent citizens to have the same level of privacy as those criminals without quite as much hassle (not that such things are a lot of hassle).

Monday, 23 November 2015

Poisoning the well

One of the things I did say about the Draft Investigatory Powers Bill is that people could easily create false "Internet Connection Records" by sending packets that from their machines.

I even suggested that this could be an app or virus people could use, though obviously a simple Tor exit node would create loads of bogus traffic.

It has, however, occurred to me that there are other ways people can be rebellious - if someone includes images in a web site, even 1 pixel by 1 pixel, they will be loaded. Those images can be from anywhere in the Internet - radical web sites, terrorist web sites (do they exist?), porn sites, anything.

Now, it seems the government are quite keen to log the web site name but NOT the full URL, which means that even though this is just an image grab it logs as a "visit" to the site - they cannot tell it was just an image and not something else on the site.

This means people can put these image tags on their web sites, or in HTML emails (even emails sent to politicians) and create false data in the logs.

P.S. As someone else pointed out, some browsers pre-cache links, fetching pages that the user may never visit.

P.P.S. Someone ask why would *I* do this - well people will have lots of reasons, not least of which is to rebel against the invasion of privacy - but I am also pointing out that criminals can be doing this to make the database less useful.

Saturday, 21 November 2015

How can terrorists and pedophiles bypass the IPBill?

One of the issues with the Draft Investigatory Powers Bill is how pointless it is, given that its measures can by circumvented easily.

Of course, what I mean is "How can NORMAL PEOPLE THAT WANT TO MAINTAIN SOME PRIVACY IN THEIR OWN HOME bypass the IP Bill"?

So, I'll explain a few ways you can use the Internet and communicate reasonably privately. These are not new. These are explained in guides for journalists and freedom fights in oppressive countries. As an oppressive regime is something the UK is clearly aiming for, it is no surprise that these methods are the same. They can also be found in terrorist manuals, again, unsurprisingly.

Firstly, if you really are a terrorist or a criminal, please stop it now.

Simple instructions - time/place, etc.

If you want to send a simple instruction to your friends, perhaps because you are starting a video game or something, maybe “On est parti on commence.” then there are simple ways to do this and you can easily encrypt that message in totally uncrackable ways without even using a computer (see my simple encryption video). Of course, you can even just pre-arrange that when you say "elephant" in any message, that is the message to get started - you don't need encryption in any way at all. So none of the following really matters if you are sending something really simply like this - you could even use plain old SMS.

Equipment Interference

This is just hacking your computer, but legally! If you are a suspect they may have hacked your machine, or your web cam or whatever, so you are probably stuffed. Using good practice for security and firewalls and sensible use of the Internet may help avoid that happening. You may want good locks on your doors too. The best thing is not to be a suspect in a crime, if you can.

Accessing web pages

The simplest way to access web sites privately is to us Tor. This is a development funded by the US navy originally. You can download a Tor browser and use it. The browsing is bounced around multiple nodes on the Internet, many of which may not be in the UK, and all of that communications is encrypted. Each node only knows the next node, and they do not log anything. Eventually the data leaves an exit node - which could be anywhere in the world, and goes to the web site. The web site does not see your IP address, it sees the exit node's. The browser may leave some fingerprints of who you are, but a Tor browser would try not to. Obviously if you give a web site any details yourself then they will know who you are or claim to be. But the IPBill will only log that you are connecting to random nodes on the Internet, and that maybe you are using Tor. The ISP retention stuff will not show where you went on the Internet.

Using secure sites

Using an https (secure) web site outside the UK should be safe from the content being logged, but the fact you visited the site can be snooped. At present the name of the site may be too, but protocols are improving. Depends if you want to protect the content or meta data.

Sending and receiving email

For the content of email this is easy, get one of the PGP email plug-ins for your mail client. May be listed a GPG or GNUmail or similar. They talk an encrypted email protocol. Read up on handling keys properly and check the keys of your friends are really theirs. This protects the content of the email. Importantly it does not protect the subject line or the from/to email addresses. That could all be logged.

However, there are simple ways to protect the to/from and subject and so on - using encrypted links from your phone/PC to your email servers. This is normal, using imaps and smtps, and many mail services allow this. Or use https to a web mail system. But beware - the mail server may have logs, and if in the UK they could be collected under the IP Bill. To avoid this you need to run your own mail server - which is really not that hard (google it). You also need your friend to run their own mail server too. The snag then is that they can see this encrypted connection between your email server and your friends so assume you are communicating. Using Tor will help hide some of that too.

An alternative is use a common mail server in a sane country and use smtps and imaps to talk to it, and hope that country is not handing over logs to the UK. I don't know if there are email services in North Korea, but if there are you can bet they don't send logs to the UK.


There are a number of end to end encrypted messaging apps for phones, but even iMessage should be mostly safe unless Apple get coerced in to unlocking it. All the snooping will show is you are talking to Apple - it may not even be obvious you are using iMessage. There are also Tor messenger that makes use of message systems like irc but encrypts message and hides the parties to the chat channel.

Phone calls

Tricky - some things like Apple FaceTime are as safe as iMessage, to probably quite good. Some apps exist like Signal which help ensure content of calls is secret, but the fact you are using Signal will probably not be. The biggest issue is that any calls to or from the normal phone network are already logged. Same with SMS to or from the normal phone network. Using foreign SIP gateways and a VPNs to get to them could make it hard to link the calls to you though. It depends a little if you just want to protect the content of the calls, or the meta data (the fact you made a call, when, and who to).


One on the all encompassing methods it simply to make use of a VPN. This is an encrypted link to some point on the Internet. From there it is normal Internet traffic and all of the above may be useful, but if the VPN endpoint is in another country then that bypasses the IP Bill. The snooping shows only that you connect to that foreign endpoint using a VPN, not what you are doing.

Two main ways to make a VPN. One is to buy a VPN service. There are quite a few now, and some will allow connections via various countries. For a few quid a month you can make all of your Internet go via this.

The other way is to buy a cheap VPS (a virtual server) which is a computer on the Internet, and then install a VPN application on that server. Again, only a few pounds a month. This is then in your total control, but works in the same way. Of course if you and your friends all connect to a dedicated VPN endpoint like this, then the snooping shows you are connected somehow. Using a commercial VPN endpoint will hide that.

Either way you can make your phone or PC talk directly to the VPN endpoint, or you can even get some home routers now that handle IPSec (a VPN protocol) to put your whole house and wifi on the VPN.

The other end

Remember, if you are communicating with anyone, even a web site, the other end sees the communications. If they are compromised, hacked, or simply untrustworthy, they can reveal your communications. In some cases, such as Tor to a web site, they don't know who you are or where you are, but for email and messaging and so on, that is not so easy. Anonymity is a who other area of privacy which I am not going to try and cover here.


Yes, there a load of ways to make the logging in the IP Bill totally pointless. A lot of people would not bother with even these simple steps, but any criminal with any sense whatsoever will be able to hide what they are doing with ease. The real victims of the invasion of privacy will be the innocent citizens of the UK.

However, please, politicians, take this in the way I mean it - as an example that shows the futility of this endeavour. Concentrate the effort and money where it matters - police on the ground - following the leads you already competently get - stopping crime without invading privacy.

Quote of the day from the A&A irc channel:

I actually already do tunnel almost all my internet stuff 
through a VPS, to deal with general local ISP rubbishness (e.g. 
dynamic IP address, lack of IPv6) and very localised 
surveillance/tampering (e.g. a dodgy wifi hotspot) rather than 
to try to hide from the UK government.

Cost of Data Retention

The Draft Investigatory Powers Bill has a requirement for ISPs to retain data, but the wording is so wooly it could literally be any data.

One of the important points to be debated about the bill is the cost impact. Obviously people are asking what the cost of retention will be. Unfortunately I don't know, because unless, and until, we get a secret retention order, we don't know what is expected of us. Even if other ISPs get orders, we will not know as they are secret.

So we need to get a handle on what they intend. Unfortunately it is more important than that though - it is not just what they intend, but that intention has to be then put in the bill. If not, then the second the bill passes the secret orders could be very different and have totally different costs to those debated in parliament before the bill passes. If even the politicians are honest (choke!) a change of government puts someone else in charge and they can use the act based on what it says, not what the intentions were. What is worse, as they are secret, nobody will know that the orders are not as per the intentions explained to parliament.

To try and put this in to some sort of logical order, I have listed below some of the things that could possible be requested and an idea of complexity. What would be useful it to know which of these they are after, and have that writing in to the bill now.

Keeping existing logs for a year

Some things an ISP already logs. Examples are email server logs, or call server logs. If the ISP already logs something to a durable medium such as a hard disk, and keep logs logs for a period (a few days for email logs, for example), then simply asking that they keep the logs for a year, and provide a means to access via RIPA requests, is not too hard. It has some costs (bigger hard disks), but is technically relatively simple. I am not too worried if such orders are made, especially because we could move such services outside the UK if we did not wish to make logs at all.

Making some new logs

In some cases an ISP will have equipment which has some means of creating some logs, but they don't log at present. Assuming the equipment is capable of making logs that can be stored in some durable medium, then it could be possible to turn on that logging and keep those logs and have them for a year. This is slightly more work. If the logs are particularly sensitive data, the ISP may have to have extra security measures that would not be present if simply "not logging" as now. It is a step further than just keeping existing logs, but may be possible.

New equipment to make more logs

There are ways in which some equipment can create additional logging, such as sampled IP headers. This is usually used for network diagnostics, things like working out where a denial of service attack is coming from and going to or planning network upgrades or configuration changes. It may not be enough to be that useful for intelligence services as it is more statistical than a proper connection log, but it may be. Installing new equipment or upgrading existing equipment may be possible to provide this sort of additional logging. This will have some costs for the new equipment, and again for the logging itself, so is a step further. The cost will somewhat depend on the extent of logging required. In the case of A&A, one of the big costs in any new equipment is the fact that the rack in question is full and the data centre in question has no spare racks - that could make installing one cheap piece of kit very very expensive.

Logging TCP sessions, UDP exchanges, etc.

It could be that they would like a log of all "sessions". Note that a "session", or "Internet Connection" is not a hard concept - it exists for TCP, but not for UDP or ICMP. It sort of exists for IPSec with key negotiation. For some protocols like SCTP or MOSH it is somewhat more complex as the single "connection" can change endpoints like Trigger's broom and stay up for years. Even with TCP, a "connection" could last days or months or years - it could be that when the session ends and is logged it is already older than the 12 month period of logging. Just trying to define what a "connection" is will be hard, but some sort of Deep Packet Inspection (DPI) kit could track sessions. This is very expensive on any scale at all - ISPs routers use specialised hardware (ASICs) to keep up with just forwarding packets - to track "connections" is a lot more work and cost.

Logging stuff from TCP sessions, like web or email addresses

Ultimately, what was said in parliament, is that they want web logs - logging the web site names. This is much harder still - you don't just have to track a TCP session over multiple packets but have to track the clean data stream within it, understand higher level protocols like http, and extract information from those protocols like web site host name or email headers. This is another level of expensive and complex over and above session tracking. Note that this level will be increasingly thwarted by the use of encryption.

Logging all content

We don't think they yet want to log all content, but basically that would be impossible. The storage requirements would be vast and impossibly expensive - the data flowing over the Internet is just too vast to log.

In addition to these various levels of logging, there are some other key issues :-

Denial of Service attacks

One small point is that there are denial of service attacks - these will look like millions of separate connections a second. Any system that tries to log "internet connection" records will need to be able to keep up and log these. The issue is, of course, that these are enough to break the network normally - having a logging system that does not break in the face of trying to log this traffic will be even more expensive. Now, you could take the view that we don't need to log a denial of service attack, but (a) surely you do as it is illegal activity and that is the whole reason for making these logs, and (b) the DOS could be targeted at the logging - not enough to damage the ISPs network but enough to look like a shit load of connections and be too much for the logging systems to keep up with - thus losing real connection logs. Being able to cope with such new DoS targets will mean even more complexity and cost for the ISP.


One of the big issues, and costs, with any of the more complex solutions for tracking "connections" and especially tracking data from within those connections, is the changing nature of the Internet.

Already we see more and more systems using encryption - so even something a simple as sending or checking email will now be impossible for the ISP to "see" in to and identify the sender and recipient of the email by email address unless they are themselves providing the email service. https which is used for many web sites now currently allows DPI to "see" the website hostname, but that too is changing and it will soon be encrypted too.

But even without encryption, the protocols change. This is not just because standards change, and they do, but because of the very nature of the Internet. It allows packets to go from one place to another and does not care what protocols are used. As long as both ends understand, it does not have to be any sort of "standard" at all. An application on a phone could talk some completely new IP protocol to its server over the Internet, or even talk something that looks like an existing protocol like TCP but actually in a totally non standard way. That is all valid in the Internet. Web sites generally have to follow some standards but games and apps can do what they like, and often make up their own unique protocols for communication with game servers. One of the key things that may want to be tracked is things like in-game chat - but there is no way an ISP can sensibly do that looking at the packets as they pass, even if not encrypted.

Interestingly Network Address Translation (NAT) is responsible for limiting the protocols commonly in use (typically to ICMP, UDP and TCP) because that is what NAT boxes understand. Even with this limitation, the protocol then used over TCP and UDP can be whatever you like. However, IPv6 is finally taking back the Internet as simply a means to get IP packets end to end (as it was designed) - it now allows new protocols and misuse of existing protocols without the limitations of a NAT box having to understand what you are doing.

So, the equipment that does any sort of session/connection tracking or DPI will have to be constantly updated and maintained to handle the new protocols coming along, and even guess at some protocols it has never seen. If looking in to higher level protocols, that will be a constant battle with innovation on the Internet, and with rebellion at the monitoring that is being done.

However, in summary - we need to know what level of logging is intended by the bill, and we need the bill updated to be clear on that, else the cost estimates are a joke.