RevK®'s ramblings

2016-12-30

SendNotificationResult

For anyone trying to work out why Microsoft Exchange Push notification message responses are not being accepted by the server, it has taken me a while, but it seems to be that it does not accept a "chunked" response.

We were sending a response from a CGI script from apache, and that is normally chunked.

But there is no way to guess what is wrong. Lots of examples on the Internet, but none worked.

We were sending text/xml with :-

<?xml version="1.0" encoding="UTF-8"?>
<Envelope xmlns="http://schemas.xmlsoap.org/soap/envelope/">
<Body>
<SendNotificationResult xmlns="http://schemas.microsoft.com/exchange/services/2006/messages">
<SubscriptionStatus>OK</SubscriptionStatus>
</SendNotificationResult>
</Body>
</Envelope>"

I tried every combination of xmlns tagging and all sorts, eventually solved the problem by sending with a Content-Length rather than chunked. That is what it wanted.

Arrrrg!

But I guess it is in line with the rest of the documentation, which is pretty crap (in my opinion), e.g. one page listing a field as URL in one place and Url in another, and we had the wrong one. You just have to guess what is wrong half the time.

Anyway, this blog is for those looking for Wisdom of the Ancients thanks to XKCD

2016-12-28

BT and their wifi adverts

BT have made some interesting claims regarding their WiFi, with the latest being that it is the "most powerful".

Now this is a rather odd claim as "power" is not something that is all that relevant - power (measured in Watts) is not that helpful as a measure of WiFi, indeed many smaller APs with lower power can (I believe) provide a better coverage and performance. Saying you have the most powerful WiFi is like saying your house has the brightest street light outside it. The main impact being it will make other WiFi nearby worse.

BT have made all sorts of claims before, all of them (in my view) rather suspect. The claim of most powerful WiFi, when WiFi is a radio data system to strict international and national agreed standards, is rather odd. The WiFi will have the power within the standard and legislation, like any other WiFi. It cannot in practice be more powerful.

They even published a document to justify the claim (here).

First issue: "The BT Smart Hub has superior specifications than the routers of all major broadband providers". So only most "powerful" if you ignore the smaller providers. They only look at "major" providers. AAISP have been offering Unifi APs and packs of multiple Unifi APs for some years now, but that does not count as not a "major" provider.

Second issue: The comparison compared many things but not one of them was in fact "power"! They state: "The most important aspect of wi-fi for customers is their Transmission Control Protocol (TCP) throughput.". Whilst this is actually a quite good metric, it has absolutely nothing to do with the claim of being most powerful. Power is in Watts and is not a measurement of download speed.

Third issue: They actually tested the WiFi. This is good as that is what they are claiming is most powerful, but they are selling an "Internet Access Service" using this. The tests are nothing to do with Internet access (you can tell from the speeds they measured) and for most people any speed on the WiFi that is over the speed of their Internet Access is irrelevant, so no help. Yet the advert is to sell Internet Access, not simply WiFi APs.

Basically, they are simply claiming they have a good 3x3 antennae single AP WiFi system they sell/provide with the Internet access system they sell and that it is somehow more "powerful" than other ISPs. They ignore the other (smaller) ISPs selling systems just as good. They ignore those selling multiple access point solutions which are better. They ignore all of the non ISPs also selling this equipment. And they ignore that the actual "power" is the same on these devices and their claim of most "powerful" is not actually about "power" at all but TCP throughput.

Anyway, yes, consumers want an Internet access service that is good. If BT are "most powerful" in that, why are many ISPs (including AAISP) way higher in ispreview's list? (here)

2016-12-22

Connecting to @AAISP Fast?

Over the last week or so we have had a lot of custumers that were with "Fast" wanting to sign up with us urgently because they, rather unceremoniously, dropped all broadband customers overnight!

The good news is that most of their customers are using a back-haul provider we use, and that means that using our login on their existing line "just works".

There may be exceptions, but we have connected loads of ex-Fast customers same day, once they order we allocate a username and password and allow them in. Some are on-line within minutes.

We had to tweak the systems we have to allow this. Normally people can only log in once installed. But with some complicated work on the RADIUS server, we managed it.

Once the day comes to actually migrate there may be some downtime - some re-jumpering, etc. But overall this is a bonus for all those stranded customers. Getting on-line before Christmas matters.

It did rather put a strain on the staff just before Christmas, and even managed to upset my system for attack management that thought someone was pushing automated invalid orders at us! Soon sorted.

But, as ever, we'll aim to offer an excellent service to those customers moving to us, and ensure it all works over the holiday period.

Merry Christmas to all our new customers.

--
Adrian
Director of A&A

2016-12-21

Good news for privacy - Investigatory Powers Act vs CJEU

As reported by the BBC, the European Court of Justice has made a ruling that could seriously impact the powers in the Investigatory Powers Act to collect data on everyone in the UK.

The IP Act has provisions, much like the Data Retention and Investigatory Powers Act (DRIPA) it replaces, and the Data Retention Directive (DRD) before it, to retain data about use of communications systems.

The IP Act actually pushes this much further - previously telcos/ISPs could have been asked to retain certain data they processed (e.g. telephone itemised billing records) but could not be required to actually generate data they were not processing. The IP Act allows much more and it has been made clear that the government wish to log usage of the Internet in some detail - down to the level of recording every web site everyone has accessed. This is far more than just retention of data, and would apply to everyone, even those not suspected of any crime.

The good news is that the ruling from the CJEU is that this sort of mass retention of data is not consistent with our basic human rights and EU law. These apply regardless of whether we leave the EU or not.

The BBC article is not ideal in its analysis, and Open Rights Group have a much better analysis (here).

Retention is an invasion of privacy

The key point of argument here is that the UK Government considered that indiscriminate retaining of data should be allowed as long as access to that data was restricted and controlled in a suitable way. However, that is not the case. The court ruled that indiscriminate retaining of data was simply not acceptable. You have to be much more specific about whose data is to be collected to target suspects in a crime.

Only to be used for serious crime

The court also looked at the issue of controls over access to the retained data. Again, this did not go well as the access has to be restricted to only serious crime. The IP Act tries to even redefine serious crime to include things that are not serious, so that will have to change too.

Proper independent authorisation of requests for data

On top of that - the access to the retained data should be approved by an independent body, such as a court, and not simply by the current system of a Designated Senior Officer. This could finally mean we see proper court warrants for access to retained data.

No more secrecy

As I have long said, the secrecy around data retention and collection of data is not really acceptable. The ruling says subjects of access should be told about it once there is no longer a risk of prejudice to the investigation.

We can still catch criminals

None of this stops wire taps (or the Internet equivalent) on suspects in serious crime, set up and accessed with the proper controls. All it stops is the indiscriminate logging of everything we all do on the Internet - and that is a good thing - we are all meant to be innocent until proven guilty, after all.

Read more

Read the ORG article for a lot more useful insight in to this ruling.

2016-12-20

IEC18004 QR Codes

I said I had my mojo back :-) Yesterday afternoon I decided to have a bash at writing a QR code encoding library, from scratch.

Yes, this is re-inventing the wheel as there are QR encoding libraries out there. It was fun, and it is always nice to have source code that is ours, especially if we may put it in the FireBrick (I am looking at making the TOTP logic in the FireBrick a lot easier to use).

Thankfully Cliff had already written a Reed/Soloman ECC generation function for me, and has made me a very simple BCH coding function. Whilst I understand Error Correction Code, it really is just beyond me in terms of the maths.

I found a copy of IEC18004 on-line. You normally have to pay for a spec, and I may do so at some point, but the court ruling on reading stuff on-line using your browser makes clear that I am not breaking copyright simply be reading it in my browser - whoever is hosting it is. It is 118 pages long!

What really annoys me about this whole specification is the tables of numbers. Instead of saying that the alignment marks will be evenly spaced with spacing between 16 and 20 units starting on unit 6, or something like that, they have a table that states the positioning for each. I played around and worked out a simple algorithm to work out the table and so did not have to use the table - yay. I double checked my calculations only to find one barcode size does not follow the same logic and is a special case for no apparent reason. Why not just make it a simple algorithm?

You then have the same for the level of ECC coding - rather than say "medium ECC uses X% of the space for ECC words" and work that out for each size, there is a table, for four different ECC levels for 40 different sizes of barcode. Then the number of blocks used for ECC is not something simple like "use more blocks when data encoding size is 32 bytes or more" or something simple, no, again a table, for all four ECC levels and all 40 sizes. It drives me round the bend. It could be one line of C rather than typing and double checking and testing hundreds of numbers in to a table.

Anyway, in the end, I have myself a nice little library that codes in 8 bit, Alphanumeric, or Numeric (not Kanji, but I could add that I guess). It codes the input all in one format only - I may, later, make something to work out optimal coding of the string changing coding in the middle as needed, like I did for the IEC16022 barcode library I wrote years ago, but I suspect there is no point.

It is very useful having QR readers on my phone to test it, and the reference coding in the specification was really useful too. I like specs that do a worked example like that.

All in all a fun little project for a Monday afternoon.

This was published at the time on the A&A site but is now a GitHub project (here). And yes, we did put it in the FireBrick for TOTP.

2016-12-18

Boiling a frog, and old age

We know the story of boiling a frog - you start with cold water and gradually make it warmer, that way the frog does not notice and jump out. [who would do that?!]

Well, I have noticed that getting old is like that. Several times now I have discovered a change in my life that only strikes me when fixed. Being diagnosed diabetic was scary as only when I was on medication did I realise how much all the symptoms crept up on me over the year before. These are symptoms I knew to look for and had drummed in to me by my mother since I was a child, and still they eluded me. Mostly tired and thirsty. I had got used to taking a glass of water to bed - which stopped being necessary as soon as I was on medication. Now I am on insulin, and my diabetes is well under control, or so I thought. Indeed, the annual reviews and HbA1c tests are all good.

The latest example is one where, over time, I have realised that whilst generally feeling reasonably heathy, I was going to bed tired sooner, and feeling much more apathetic and doing less work. If I was up at 9pm there was a joke in my family that it was past my bed time. That should have been a clue. I would sleep for like 9 hours a night, and not do a lot of work during the day.

Then I was put on indapamide as my blood pressure was getting higher, as I blogged recently. The 2.5mg dose was too high and I felt like crap, but now on 1.25mg, I feel better. I realise that since I started on the indapamide I am feeling "better". Over the last few weeks I have designed, coded, and deployed the whole 2FA systems for A&A (four separate systems), whilst also coding a load of other stuff including a Monzo API library and a few other things - documenting it all, and testing it all.

To my surprise, I look back at last week, and realise a couple of days ago I was up at 5:30 am and working solidly until 11:30 at night with no problem, only to be up at 6:30 the next day. I am finding I am bored just watching TV or going to bed, and instead am doing stuff. All last night I was designing in my head new code for a feature on the FireBrick which I ended up getting up and documenting first thing this morning. I feel like I have my mojo back.

The issue is not, as I see it, the blood pressure, which is what the indapamide is for, but it has changed the way my diabetes is working - I am having to take more insulin, about twice what I was, but I am much more stable now. It will be interesting to see my HbA1c in a few weeks time. Indapamide is not listed as a treatment in any way for diabetes, just that it can impact blood glucose levels. What is encouraging is that, having mentioned on a recent blog, I am not alone. Others with diabetes found they were "revived" (which I feel is a really good description) once on indapamide. So maybe it should be a diabetes related treatment?

Now I wonder what the next thing will be - something that will creep up on me over many months before I realise.

2016-12-16

Change Freeze

Tricky subject, and the very fact the subject has come up means something for the size of A&A now.

We have a change freeze, started this afternoon, and going on until after new year bank holiday.

The principle is fine - we have a lot of staff off, especially some of the senior technical staff, and none of us want major issues whilst we are at home with family if it can be avoided.

So the idea is we don't make major changes or deploy new systems over the change freeze. Nice idea.

There are, however, a few problems, and it is a change for the way I work for a start.

I am very keen to do a job and finish and deploy it - I hate having any job interrupted by a big gap - I lose track of what I am doing and spend a lot of time catching up and things can be missed. So this means that where there is a job in progress before xmas, I have been rushing to make sure it is all deployed before the change freeze. This is not to say taking short cuts as such, but rushing. I don't want a half finished job not deployed. And no, finishing on a dev system and deploying next year is not good - I like to deploy things as I go and pick up issues and fix them whilst still fresh in my mind. If I did the work and did not deploy for two weeks that would be horrid.

We also have the fact that xmas can be a quiet period from a technical point of view - it is (was) an ideal time to deploy and test changes with lower than usual impact. For a start, a whole bunch of customers are not even there - businesses shutting down. And whilst I don't mean to say business customers are more important than residential, there is a difference for a business customer disrupted in their business for a few minutes, or a home customer disrupted whilst eating mince pies. So traditionally the xmas break has been a good time to work on some major projects and iron out the bugs before everyone is back to work or doing anything serious with what we sell.

Ironically, whilst a few months ago, I would almost be happy to sit around doing nothing all xmas break, we now finally have me on medication for my blood pressure which has had some sort of impact on my diabetes which means for last few weeks I feel much more like I was in my 20's, bored of watching TV, and coding from dawn to dusk (well much later with it being winter). Seriously, this is great, even if it won't last (5mg perindopril + 1.25mg indapamide, FTW).

We had only one snag with stuff rushed through yesterday, and it was not actually due to rushing at all, it was a VoIP issue, which is a complicated set of issues where a recent change, which had been tested on several boxes, was deployed as part of an urgent update to address a customer issue. Sadly, when load got to a certain magic level on the live VoIP servers we go drop outs. Our normal testing on several other boxes did not pick it up, and would not have had we not had the change freeze and hence done the update next week instead. Sorry for the inconvenience on that - the VoIP servers are a pain as reloading means dropping calls but waiting means people with dropouts in calls until we do - we managed to move calls and so only drop a few to get the new code deployed during the afternoon.

But overall I feel rushed by the change freeze and not entirely convinced it will help with issues cropping up or not. I guess we'll see over next couple of weeks, if I go crazy, and/or make a huge list of changes all done on Jan 3rd and consequences of that.

If I do have my mojo back, I am damn well going to do stuff, but maybe not A&A stuff. My son has a load of web/app sites that could be tidied up, and my mate Mike has loads of stuff he wants re-inventing from scratch (probably including "the wheel", knowing him). I may find stuff to do.

So, happy freeze everyone.

RevK^®'s ramblings