Wednesday, 22 November 2017

Openness

As an ISP we (A&A) try to be very open about what we do.

I worry that sometimes my own staff can be concerned that we are too open. But I appreciate that some times we are just "asking for it" if we are too open, and that is why I have had to be so coy with all of the work we are doing on DoS attacks. It is frustrating that we are doing a shit load behind the scenes but we cannot really say anything about it. It would just open up other ways to attack things. I am really sorry.

I would say to some of the dealers and peers we work with that I am more than happy to discuss more detail off the record, ideally when shit has stopped hitting fan.

But, in the interests of openness I just want to say I am sorry that we cannot say much.

As both an ISP (A&A) and equipment developer (FireBrick) we take these issues very seriously. I wish I understood the motives and psychology of such attacks.

It may be worth saying a few things about my background, as I was using computers long before the Computer Misuse Act and the illegality of these sorts of attacks. I was, what kids may call leet, or 1337, long before that was a term. I fully understand the fun of proper hacks - the thrill of basically solving impossible puzzles. It was a game. The game is the same for any coder. And anyone trying to debug someone else's code, especially "black box" style remains. Working out how and why some system will be vulnerable is a game.

Even now I see behaviour in equipment we are trying to work with and I can imagine or picture in my mind the bad coding that must exist for that particular bug to happen. It is so frustrating.

When I am working on code, I now have to think how someone may exploit it, how they could send a packet that is not quite right just to find a weakness in my code. The very things I may have once done.

I remember the days of unix /etc/passwd files where hashes had no salt, and just comparing them to other passwords allowed one to find matches. I remember simple setuid shell scripts being allowed and making them as root to allow access later when root password changed. I remember then being employed to work on sorting such issues on systems.

In many ways it was fun, but these days it has lost a lot of appeal and not just because of the illegality of it all. Yes, expecting and finding (and reporting) bugs in systems we work with is still an important skill, as it coding systems to expect the "attack" vectors, but just not as "fun".

I wish I could understand the motivations so that I can try to resolve the issues for just someone paying some bot net in bitcoins to DoS some target, to be honest. Where's the challenge? Maybe I am too old to be "leet" now. Oh well.

P.S. I am not asking you to DoS us more - your skilz are there, and cause the hassle you want, I am sure. TBH I would love to know why - what can I do to help? What have I done to upset you? Talk to me? Can I be more open than that?

Moving on

Handling DoS attacks is hard work, takes up a lot of staff time, and involves a lot of changes to code, and config, and policy and procedures and involves meetings and continues to do so, sadly. We are, of course, continuing to work on issues in a lot of ways and improving. There are likely to be even more short notice planned works over the next few days, so watch this space.

However, I hope we can move on to more pressing matters, which are that the Talk Talk backhaul is starting to struggle again, and we are really not happy. We set high standards for the back-haul carriers. We know we have high per user peak loads as I mentioned before, and they have to cope.


This is not all lines, indeed, I am not sure it is a lot of lines, but it is a concern that we have even one line like this. It is not acceptable, and we have been working with Talk Talk for some time on this sort of thing. It looks like, for now, some bits of their old network are suffering and that is a graph that shows one such line.

The good news is their new network is making progress. There have been a lot of significant snags, and Juniper have been working with Talk Talk to address these. I have to say I am shocked that a system like L2TP, which has been stable as a protocol for like 10 years, is hard work for the likes of Juniper, but there you go...

The new platform is under test and we appreciate everyone's help with this. We hope it will be live soon and the issues on the TT back-haul put to bed for a long time.

Thanks for your patience.

Monday, 20 November 2017

This is why I will never be an alarm installer

Today, with some help from Jamie (thanks), I installed an alarm system for someone. They are guinea pigs in this, using the SolarSystem alarm system I created (on GitHub). They know what they are getting in to and have paid cost price for it. It actually seems to work well, but let's give it a few months. My system at home has worked for 5 months now with no issues.

Whilst I am capable of doing this, no way I would do this every day. Yes, maybe, I could run a company that does, but seriously, I have managed to avoid this "hard work" thing quite well, and my knees are now killing me. All I did was spend half the day standing on a step!

And what can I say without swearing? Fucking screw connectors. They are evil. Really, horrid things.

TBH I am thinking that I need a shitload of simple crimps for wires to then use in screw terminals. Better than solder tinning them, as I understand it. Better than twisting them together. And something you can do on a panel at ceiling height whilst standing on a step. It has to be done, to do it right, and I doubt any alarm company does that. Next time I do this, I think this will happen.

But installing alarm systems is not for me. It is something we could do if we wanted to be in this business. But yes, the SolarSystem works!

Saturday, 18 November 2017

Monzo current account and fast payments and their API

I thought I'd be less controversial for a change and say I am actually pretty impressed with the API link to a Monzo current account.

Just need a business account from them now. We could build services around instant fast payment to us I am sure.



P.S. I am testing BACS and DDs to see how they work, do I get a ka'ching at midnight or something. BACS is all batch date based, so will be interesting to see.

Friday, 17 November 2017

DoS attacks, sorry

I'd love to be able to blog in detail about denial of service attacks, what works, what does not, what we can or cannot do, but it would be mad to do so.

Suffice to say that anyone that wants to, and has a few bitcoins to spare (or fractions thereof) can engage one of the many botnets that exist and DoS the hell out of almost anyone, no matter how big.

It is unusual for an ISP to actually be the target, so much so that we did not cope well at all. We have all been working hard all day, and we all feel knackered. This attacker has caused the problems he wanted to cause, he has upset customers, and disrupted an ISP. Whack-a-mole does not really explain today adequately.

A customer had even ordered pizza for the staff, which is appreciated.

We still have much work to do even if the attack has now stopped, and we hope it has. Not only undoing loads of temporary hacks that cause their own issues, but planning for how to handle things better in future.

At the end of the day we are a small ISP and we try to do a good job for our customers. It is a shame when we cannot do that, and if anything we, or I, did upset anyone I am sincerely sorry for that. We also appreciate that it may have been something a customer has done to upset someone.

And, of course, I am sorry that our customers have suffered.

P.S. Increasingly looks like "not me" but a customer pissed someone off. I have had words!

OK I get it, enough is enough

Just in case someone felt my last blog post was dick waving about how good our network is, and I think it is good, I do accept that any network can be DoS'd to hell.

I get it, OK.

Thursday, 16 November 2017

Pushing the limits

We are deploying some new LNSs at A&A, three of them so increasing the eight live LNSs to eleven. This will happen over the next few days.

Why? Well, we are hitting some amazing levels of traffic - we seem to have actually exceeded 1Mb/s average peak usage per customer.

This is a metric which we discovered is used by the likes of BT Wholesale, and a few years ago, at over 100kb/s we were a really high usage ISP on their network - the highest at the time.

Basically you look at the peak usage, which for us is now in the evening, and divide by number of customers, simple as that. Bear in mind, at that peak time, pretty much every one of our business lines (which is a lot of our customers) will be idle.

Even so, that average is getting to the 1Mb/s. That is amazing. I am shocked. We have multiple 10G links to peers, and multiple 10G links to one carrier, and extra 1G links going in right now to another carrier, and more LNSs. All in aid of not being the bottleneck.

This really is the netflix generation taking off, and it is a challenge for ISPs. We are coping, and we are making sure we stick to our "not the bottleneck" aims. But it means some quick work to extend the capacity with staff on site today installing new LNS hardware.

We know that Talk Talk are working on expanding their network, no doubt seeing the same increases, and we are working with them to make the best use of that new network as soon as possible. Thanks to those customers testing things with us.

It seems not long ago the maximum a customer could manage, a far cry from average, was 64k ISDN. How times change.

With new links to BT going in, we expect to make tariff changes over the coming weeks to make "terabyte" just another level on all of the Home and SoHo packages, with easy regrades, and balancing of traffic over multiple lines. (Not for 20CN, sorry)

If you want Internet access that works, and keeps up with the ever increasing usage, you know where to come :-)