Apologies in advance for this being a tad long and rambling. But one way to try and work on debating this sort of thing with the government is to try and put yourself in their shoes for a moment and understand why they are asking for this. So I am going to give it a try, but it may be hard.
I have tried to break down some of the basic challenges with managing society so that it works.
A fundamental idea is that some things are unacceptable to society, they are bad things and as such we should try and stop them happening. What exactly is a bad thing will vary over time as society changes, though some are pretty ingrained such as "murder" and "theft". It is a lot less obvious when when you get to things like "copying the CD I purchased to an MP3 so I can play it on my iPod on the train". The very definition of bad things is a matter for ongoing and rational debate.
Deterring people from doing bad things
The basic principle that surrounds most law is the idea that punishing those that do bad things should result in the anyone planning to do a bad thing deciding it is not worth the risk. That anyone considering doing a bad thing may reasonably expect that they could be caught and punished. It is basic human nature to avoid pain, to remember pain, and to predict pain - it is how we learn, and even how animals learn. We need people to be so afraid of being caught that they do not do the crime. For that to be realistic, we don't just need laws, and punishments, we also need very efficient means to detect crimes and identify the criminal that cause them.
What about people pulling the strings?
There is, of course, an issue that some people are smart enough to arrange for bad things to happen but themselves not actually do them. So if the crime is detected and the criminals identified, they can step back and try again with some other mugs doing the dirty work for them. To allow for this you have to then have offences for helping someone do a bad thing. This comes under conspiracy to commit, aiding and abetting, and so on.
What about catching people that are going to do a bad thing?
This is where it starts to get complicated. If the bad thing has not happened, you are looking at punishing people for something that has not happened. You really have to be sure that it would happen, and that is tricky as it is predicting the future.
I was thinking of a possible example here - if a few of us were caught with detailed plans for how we could rob a bank, then there would be an assumption that we actually were planning to rob a bank. But what if we had a half finished on-line game called Bank-Heist or something, and these details were simply research for a computer game? Then suddenly there is no realistic risk that the bad thing of robbing a bank would actually happen. Indeed there are other good reasons for having plans to do a bad thing, and that is where people are trying to devise ways to protect and defend against such things. Of course, a smart gang of bank robbers would hire a s/w engineer that thinks he is in fact making an on-line game so that they would have that defence :-)
The problem is that punishing people for something that they might do is a very dangerous game, a slippery slope in to thought crime. What if two office workers were upset with their boss and discussed how they wished he would be hit by a bus? What if he is hit by a bus? What if all communications is logged in a police state and someone finds that conversation? Thought crimes!
What about security?
The police will investigate crimes and find evidence to identify and convict a criminal, but that is generally after the fact, the idea being that convicting criminals deters other criminals.
But there are situations where that does not work. There are special types of criminals, typically terrorists, that feel that what they are doing is "right" and so much so that it overrides they fear of being caught or punished. Indeed, in the case of suicide bombers this overrides one of the most basic fears of all - fear of death. There are plenty of other risks where fear of being caught and punished is remote, such as cyber attacks from foreign countries.
So we have to consider the idea of the security services, who are trying to keep us safe from threats like that. They cannot use the traditional "fear of being caught" to deter people, so they need other means.
What they would ultimately like (one assumes) is a way to find people plotting to do a bad thing, and take some action against them before they do it. As I say, we are well and truly in to thought crimes here and we have to consider this only makes any sense for really serious bad things and where we are really sure that they would in fact do the bad things they are plotting.
In their ideal world they would have surveillance on everyone, all the time, audio, video, logs of everything they type and say, and the vast computing power to sift and sort that to find any hint of people doing bad things.
One of the difficulties here is that it if they had that, it would not stop at thwarting terrorist plots, like someone threatening to blow up and airport and saying so on twitter. No, once they have that power it would apply to office workers "plotting to kill their boss", or any number of minor things. So many laws make everyone a criminal already. A complete police state like this would be unacceptable to the general public.
What can we do?
Trying to be in their shoes - we know they would like a total surveillance police state, obviously. It is the only way to be sure that people are safe. You probably need to restrict people's movements and communications as well, just to be sure.
But we know that will never fly, we are in some sort of a democracy (though the way some Lords are behaving this week, you would not know it). Going that far would amount to civil war, or at the very least losing the next election.
So the real question is where you draw the line - how far can you go before what you are doing is impractical or unacceptable. That is their problem.
So what would I do?
I think what we had was not that bad, but as it was the status quo, I am conditioned to find it acceptable. In some ways it already goes too far.
But I have some ideas of tests for this - to decide how far I could go:
- Is what you are asking for causing people and companies to do more than they would normally do? After all, getting telephone records was only possible when BT started itemising bills and so had the data - they were not asked to do any more than they already did, just check something on the data they already kept for commercial reasons. In think you may have gone too far if you are expecting people and companies to police other people and companies - to seek out and collect and retain data they would not normally need to. Making people police their neighbours is a very old system of government and will always cause suspicion and resentment.
- Is what you are asking for targetted? This is important as otherwise you are essentially treating the population as criminals in the first place. This is also an issue with lines that have already been draw - human rights conventions and the EU Court of Justice where it is clear that surveillance has to be targetted. It is one thing to say "You have these phone/email records, can we see those for this person who is a suspect?" and another to say "Collect all this extra personal data for us for everyone just in case we want it later".
- Are you invading privacy? This is a complicated one. Reading someone's private communications is clearly an invasion of privacy, and only really justifiable as a targetted action against a suspect with proper oversight on the process. Again, this is enshrined in human rights conventions. Now, if such action is to be targetted you should not be expecting everyone else to give up their right to privacy. This is tricky with encryption, which is now common. You would have to pick individuals who are suspects and say that they alone are not allowed encryption and hence privacy for a period of time. Everyone else should be allowed proper security and encryption as they are not suspects. Such a move is not practical in many cases.
- Is what you are asking practical? This is another important concern. There are many cases where the wishes of the security services are not actually practical. I have a video [here] that shows step by step how to send truly secret messages with no more than pen, paper and dice. Banning something that simple is like banning someone picking their nose, it is a nonsense. But even if you are asking ISPs to do something, it has to be something they can do, and also something that will not compromise the integrity of their network.
- Is what you are spending value for money? This question comes back to the fundamental roles of security services. Terrorism is a serious threat but not one that should be such a high priority for effort and expenditure as it causes so little harm and death compared to so many other areas which could be improved. If we are spending public money it should be a good return on investment. Even preventing an horrific crime killing thousands of people is only really sensible when compared to reducing accidental road deaths by that many. In the US, more people die from slipping in the bath than from terrorist attacks. So, yes, spend money stopping terrorists but only where it is value for money compared to other places on which it could be spent.
I think you will find that the snooper's charter, and even the DRIPA, fail on several of those simple tests already. This does not mean that there are not further steps which could be taken even when considering those tests.
Genie is out of the bottle
A huge problem though is that the genie is out of the bottle - private, secret, communications is fundamentally possible. Even with seriously oppressive governments in the world, journalists, whistle blowers, spies and government agents, manage to communicate without being caught.
This means that, ultimately, a terrorist cell can communicate and plot something, and even the most extreme police state could not spot that in advance. It is also the case that one nutter could just decide one day to walk in to a school and shoot everyone (if guns are not handy, as we are not in the US, poison everyone using household chemicals). Sadly, we will always have some nutters, and some awful things like this could happen. The only answer to that is upgrade us all to cybermen.
So, please, let's make laws that are fair, rational, practical, value for money, evidence based, targetted, not invading privacy or treating everyone as a criminal, but are still some help to security services.