Thursday, 23 November 2017

Summer is coming?!?

I have mentioned this before, but it always amuses and annoys us that BT seem unable to handle summer time.

The usual areas of issues are XML, where a time ending in a Z is Zulu, UTC, not "local time". A time without a Z is "local time" (but local to which end? hence to be avoided). A time with ±HH:MM on the end is a specific time zone. We often see BT see a time and assume UTC when no Z, or local time when a Z it used and sent. It causes no end of issues. To be fair Talk Talk have similar issues on XML. We also see XML from BT with the Z explicitly but the time itself is local time and so everything ends up an hour out. Customers may have seen on clueless that target completion of work in summer is often 00:59:59 the next day (local time) as BT have sent a date/time ending in T23:59:59Z.

It amazes me that this is remotely difficult or not just fixed.

The other classic for a long time was the Microsoft Office invites that would always say meeting at time GMT, e.g. 15:00 GMT, when they meant 15:00 UK local time. So we would not be ready at 15:00 local time, expecting the conference call or whatever at 16:00 (i.e. 15:00 GMT).

This was made worse when Microsoft added a note along the lines of "Times in GMT do not reflect local daylight savings times" or some such, which to me explicitly states they really do mean GMT (UTC) and not BST in such cases. Madness.

Today I have someone in ops suitably annoyed as they normally finish by now, when BT wanted to co-ordinate a simple certificate change at 18:00 BST. At least, today, they finally confirmed the right time, eventually. This is not hard, honest.

Them:

We do not perform any cutover during business hours.
We have the slot at 18 BST on Thursday , we can have the cutover if its fine for you.
Me:

18:00 BST (so therefore 17:00 local time for us here in the UK) on
Thursday is fine for us although isn't that in business hours too?

We can also do 18:00 UTC.
Them:

We can do at 18 BST today. I will send the invite shortly.
Me:

Ah, OK, I don't have Exchange so that invitation doesn't mean a lot to
me, but parsing the text by hand it says:

DTSTART;TZID=GMT Standard Time:20171123T180000

So, OK, it's 18:00 GMT. I can still do 18:00 GMT but it is an hour later
than 18:00 BST. You have caused some confusion here. We are no longer in
daylight savings and 18:00 BST and 18:00 GMT are an hour different from
each other.
Them:

I am so sorry James.
Yes that’s 18:00 GMT.

Apologies for the inconvenience caused.

Wednesday, 22 November 2017

Openness

As an ISP we (A&A) try to be very open about what we do.

I worry that sometimes my own staff can be concerned that we are too open. But I appreciate that some times we are just "asking for it" if we are too open, and that is why I have had to be so coy with all of the work we are doing on DoS attacks. It is frustrating that we are doing a shit load behind the scenes but we cannot really say anything about it. It would just open up other ways to attack things. I am really sorry.

I would say to some of the dealers and peers we work with that I am more than happy to discuss more detail off the record, ideally when shit has stopped hitting fan.

But, in the interests of openness I just want to say I am sorry that we cannot say much.

As both an ISP (A&A) and equipment developer (FireBrick) we take these issues very seriously. I wish I understood the motives and psychology of such attacks.

It may be worth saying a few things about my background, as I was using computers long before the Computer Misuse Act and the illegality of these sorts of attacks. I was, what kids may call leet, or 1337, long before that was a term. I fully understand the fun of proper hacks - the thrill of basically solving impossible puzzles. It was a game. The game is the same for any coder. And anyone trying to debug someone else's code, especially "black box" style remains. Working out how and why some system will be vulnerable is a game.

Even now I see behaviour in equipment we are trying to work with and I can imagine or picture in my mind the bad coding that must exist for that particular bug to happen. It is so frustrating.

When I am working on code, I now have to think how someone may exploit it, how they could send a packet that is not quite right just to find a weakness in my code. The very things I may have once done.

I remember the days of unix /etc/passwd files where hashes had no salt, and just comparing them to other passwords allowed one to find matches. I remember simple setuid shell scripts being allowed and making them as root to allow access later when root password changed. I remember then being employed to work on sorting such issues on systems.

In many ways it was fun, but these days it has lost a lot of appeal and not just because of the illegality of it all. Yes, expecting and finding (and reporting) bugs in systems we work with is still an important skill, as it coding systems to expect the "attack" vectors, but just not as "fun".

Anyway, I may not be leet any more. For now I won't be posting any details regarding DoS attacks.

Moving on

Handling DoS attacks is hard work, takes up a lot of staff time, and involves a lot of changes to code, and config, and policy and procedures and involves meetings and continues to do so, sadly. We are, of course, continuing to work on issues in a lot of ways and improving. There are likely to be even more short notice planned works over the next few days, so watch this space.

However, I hope we can move on to more pressing matters, which are that the Talk Talk backhaul is starting to struggle again, and we are really not happy. We set high standards for the back-haul carriers. We know we have high per user peak loads as I mentioned before, and they have to cope.


This is not all lines, indeed, I am not sure it is a lot of lines, but it is a concern that we have even one line like this. It is not acceptable, and we have been working with Talk Talk for some time on this sort of thing. It looks like, for now, some bits of their old network are suffering and that is a graph that shows one such line.

The good news is their new network is making progress. There have been a lot of significant snags, and Juniper have been working with Talk Talk to address these. I have to say I am shocked that a system like L2TP, which has been stable as a protocol for like 10 years, is hard work for the likes of Juniper, but there you go...

The new platform is under test and we appreciate everyone's help with this. We hope it will be live soon and the issues on the TT back-haul put to bed for a long time.

Thanks for your patience.

Monday, 20 November 2017

This is why I will never be an alarm installer

Today, with some help from Jamie (thanks), I installed an alarm system for someone. They are guinea pigs in this, using the SolarSystem alarm system I created (on GitHub). They know what they are getting in to and have paid cost price for it. It actually seems to work well, but let's give it a few months. My system at home has worked for 5 months now with no issues.

Whilst I am capable of doing this, no way I would do this every day. Yes, maybe, I could run a company that does, but seriously, I have managed to avoid this "hard work" thing quite well, and my knees are now killing me. All I did was spend half the day standing on a step!

And what can I say without swearing? Fucking screw connectors. They are evil. Really, horrid things.

TBH I am thinking that I need a shitload of simple crimps for wires to then use in screw terminals. Better than solder tinning them, as I understand it. Better than twisting them together. And something you can do on a panel at ceiling height whilst standing on a step. It has to be done, to do it right, and I doubt any alarm company does that. Next time I do this, I think this will happen.

But installing alarm systems is not for me. It is something we could do if we wanted to be in this business. But yes, the SolarSystem works!

Saturday, 18 November 2017

Monzo current account and fast payments and their API

I thought I'd be less controversial for a change and say I am actually pretty impressed with the API link to a Monzo current account.

Just need a business account from them now. We could build services around instant fast payment to us I am sure.



P.S. I am testing BACS and DDs to see how they work, do I get a ka'ching at midnight or something. BACS is all batch date based, so will be interesting to see.

Friday, 17 November 2017

DoS attacks, sorry

I'd love to be able to blog in detail about denial of service attacks, what works, what does not, what we can or cannot do, but it would be mad to do so.

Suffice to say that anyone that wants to, and has a few bitcoins to spare (or fractions thereof) can engage one of the many botnets that exist and DoS the hell out of almost anyone, no matter how big.

It is unusual for an ISP to actually be the target, so much so that we did not cope well at all. We have all been working hard all day, and we all feel knackered. This attacker has caused the problems he wanted to cause, he has upset customers, and disrupted an ISP. Whack-a-mole does not really explain today adequately.

A customer had even ordered pizza for the staff, which is appreciated.

We still have much work to do even if the attack has now stopped, and we hope it has. Not only undoing loads of temporary hacks that cause their own issues, but planning for how to handle things better in future.

At the end of the day we are a small ISP and we try to do a good job for our customers. It is a shame when we cannot do that, and if anything we, or I, did upset anyone I am sincerely sorry for that. We also appreciate that it may have been something a customer has done to upset someone.

And, of course, I am sorry that our customers have suffered.

P.S. Increasingly looks like "not me" but a customer pissed someone off. I have had words!

OK I get it, enough is enough

Just in case someone felt my last blog post was dick waving about how good our network is, and I think it is good, I do accept that any network can be DoS'd to hell.

I get it, OK.