What the hash?

OK, this may take a little explaining, but I think that all the huge hassle we have had to sort new BACS systems may have been totally and utterly pointless crap.

How it works
We do have to send HMRC details of payroll each month (well, on or before each payment made under PAYE). That is the new system. Previously we did this once at the end of the year.

It uses XML and stuff to send the data, and on its own this is not too bad. We have to do all the work for PAYE and make pay slips and so on - the extra work to effectively make an XML payslip and send that to HMRC is not big. Indeed, I got most of it sorted in a few hours at the start of the week. As ever I have spent a lot of time polishing up the old code for PAYE (see other blog post of code wearing out).

But yes, on its own, simply sending an XML copy of payslips as we make them to HMRC is not a huge task.

What is hard work is that every BACS submission software package and BACS bureau has also had to change to allow "BACS field 7" to be populated. A previously unused 4 character sub-reference. This even led to Lloydslink shutting down their BACS bureau. It has taken a lot of time and money for us to do this all.

The reason is that, when we send the data to HMRC, we can prove we paid the employee. This is done using a hash.

The way it works is that a one-way hash is made of some data in the BACS payment we make. BACS do this, when they get the payments, and they send the hashes to HMRC. The theory is that HMRC do not get the whole of the bank details (privacy issues?) just the hash. We do the same calculation and send the same hash in the submission to HMRC. HMRC can match them up, proving we paid the employee.

The hash is made from fields in the BACS payment. These include our sort code, the recipient sort code, and the amount we are paying. They also include this sub-reference, which has to have a "/" at the start (so BACS know it is for HMRC to get the hash) and there random characters. The random characters mean the same person paid the same amount each month does not get the same hash. They could, of course, have simply included the date to avoid that, but maybe that was too risky as BACS could be run early or late for lots of reasons.

Clever stuff, you may think. Including the amount in the hash means HMRC know we paid the right amount to someone for payroll, doesn't it? The theory is that if they cannot match the hashes then they consider that more of a risk in their risk management.

The problem
There is, of course, a huge hole in this. All the hash actually proves is that we made a payment. It is not likely we could make a matching hash for some payment we did not make. It could be any payment to anybody and any amount, as long as we made it, and and therefore we can work out the hash for the payment. There is no way for HMRC to check the hash we send them is right as we don't tell them sort codes, or the sub-reference. All they can check is that it matches some payment that went through BACS.

Wouldn't BACS tell them the amount? Well maybe, but if that is the case why include the amount in the hash?

Wouldn't BACS tell them all the details, sort code, account, amount, etc? Well maybe, but in that case why have a hash at all - why not make the submission to HMRC include the sort code and account of the employee? That would have meant we would not have had to change BACS at all.

In fact, HMRC state that they will not have any access to the source information from BACS (such as bank account details). But to check I have FoI'd them. I'll FoI how much all of this has cost later!

Basically, when making the submission to HMRC, as long as we have a means to make a payment of some sort via BACS, even if it is 1p (or indeed, the correct amount) that is paid back in to the same account it left, and we can work out a hash and include that in the submission, it will match up on HMRC's side. The actual payments to staff could be completely separate, different amounts and times and anything else. The very issue one assumes they are trying to address by matching up payments.

Basically: It is trivial to fake the BACS payments with matching hash, independent of actual payroll payments.

So why the hell did they bother?
That is the real question. Why make a half hearted system that has caused a lot of people a lot of work for no real gain?

Either don't try and check payments, or check them properly by requiring the actual payment details in the submission (sort code and account). This faffing with hashes is a total waste of time.

PAYE coding

I have to say, as I work through this nightmare of PAYE coding crap for HMRC's new RTI stuff, I am consoled by the daily Dilbert. It shows me what I am up against.

SIP progress

Well, I am finally coding again, and have been bogged down with BACS stuff and HMRC stuff, and even some FireBrick stuff, and I really have to make some progress on the FireBrick SIP code now.

We have been using in the office since last year some time, and it works really well. The features so far make up a good, scalable, PABX system. We can expand on those features over time, but we already have hunt groups and time profiles and fallback groups and busy lamp fields and call steal and transfer and so on. A good set of PABX features.

One small point has been integration testing, and I am pleased to say that it is going well. We have customers starting to use the FireBrick as a PABX now, and getting useful feedback. Even the entry level FB2500 has the PABX features.

We know it works well with a range of handsets including Snom (on which busy lamp field works too), grandstream, gigaset, and I think a few others. Obviously if anyone has anything they can test, let us know. We have tested on a few IPv6 handsets now too and that works well and maps the media to/from IPv4 as needed.

The server side has not had a lot of testing until now as people have been using with A&A VoIP, and we know that works. Turns out we were missing some of the Record-Route logic for use with proxies. I was hoping to avoid a lot of that by being an endpoint only and not a proxy. We had included the main areas of this but not all. One of SIPs most annoying features is the route set logic which has to be stored on the dialogue. Well, we have added that now, with some careful reading of the RFC and testing.

The result is we have it working with sipgate and gradwell nicely. What is fun is that we even have it working on BT with credentials that normally go on a home hub. Sadly this is a slightly bodged config with a faked DNS reply added as BT talk to a proxy but use a different hostname in the actual messages, one that has no A or SRV records. They could have done this right so simply, and just need a DNS entry. I may try and make the FireBrick config cater for this though, as it seems that some other devices will do the same (talk to named proxy but use different name within the payload). I am sure we can do the same if that is what devices usually do.

The plan is to test more handsets and SIP services - apply any fixes if needed - and write up configuration examples on the web site. We'll also look at any changes to make things more intuitive in the config (always a challenge for SIP).

I then need to work on the second phase of development, starting with a re-write of the RADIUS client code so that it can be used for SIP as well as L2TP, then doing RADIUS based user registration and call routing and CDRs... We then have various fun features, like call recording, voicemail, and so on, to bolt on. Maybe even conferencing if I feel adventurous.

Watch this space, as they say.

HMRC and summer time

So trying to access HMRC DPS service - this is basically a system to get the notifications (e.g. tax codings, and so on) from them.

Looks OK, SOAP based, XML (D'uh) so simple enough.

But for authentication they need you to get a "token", which is a chunk of raw XML sent in a CDATA response. Eeeeewwww

Well, I tried that, and parsed it, and included it in the request (with namespaces and all) and it failed. Given the way XML canonicalization works I expected to have to send character for character "as is". That is a horrid bodge to my XML library, but what the hell?

But still not working.

So why?

Well, not sure yet, but maybe HMRC do not understand summer time.

A request for a token at 17:05 BST gets a reply that includes...

IssueInstant="2013-04-06T16:05:00.185Z"

Well, given that I just pressed the button when my watch said 17:05, that is pretty good, only 185 ms out :-)

For those that do not know, the Z on the end means Zulu time, i.e. GMT or UTC, not BST, so 17:05 being 16:05:00.185 is good.

But hang on ...

AuthenticationInstant="2013-04-06T17:05:00.185Z"

Thats an hour out... Worrying...

Then this bit...

<conditions notbefore="2013-04-06T17:05:00.185Z" notonorafter="2013-04-06T21:05:00.185Z">

That is not good. It looks like we can't use the token for an hour.

How hard is it to understand the basics of British Summer Time, FFS.

Update: Even waiting an hour did not help, and my error was using wrong servers and SOAP action in wrong places. They did not document simply what they are but provided an XML definition. They then use test system URIs in the namespaces on the live system which fooled me, but turns out to be what they want for it to work. Anyway, we now have a working DPS system, and all ready to do an Employer Alignment Summary this week ready for our first RTI submissions this month. Overall it has been probably less than a day's coding, and getting replies from HMRC test team on Sunday morning is impressive. I am sure there are several more hurdles though...

Nationwide and DDs bouncing

So, anyone else with a Nationwide account had this.

You pay someone by DD, and some of the DD collections are bounced by the bank with "AMOUNT NOT DUE".

The DD rules are clear - that error is for when the collection is within 2 days of the DD being set up by AUDDIS (the paperless system for DD instructions) or because the payer (you) say you did not get the right notice (i.e. collected too soon).

As a DD collector we have had this a few times recently with customers with Nationwide accounts. We are trying to reverse engineer there rules. We tested with a member of staff after a couple of customers had issues.

It looks like they bounce any DD that is 2 or 3 working days after the first collection. Well, that is the guess so far. It could be some "rate limiting" thing, e.g. more than 3 DDs in 3 days or some such. We have more tests to do.

The problem is that we launched a new tariff with very strict credit control - bounce a DD and you are cut off until sorted. We have had to somewhat relax that logic as a customer with a Nationwide account apparently bounced a DD when in fact it was all done by their bank.

Having some sort of logic for rate limiting is not totally daft, although the DD guarantee makes it a tad redundant - but we have people who sign up, are invoiced a connection, and a few days later invoiced an ongoing bill, so two DDs close together. In some cases they buy another service like VoIP, and bingo - three DDs in a short space of time. All notified as per the DD rules.

The real problem we have is the bullshit from Nationwide. Sorry, but that is what it seems like to me. It is bullshit - and does not stand up to simplest of checks.

1. They tell customer that A&A did not send the collection, our fault. We have proof they rejected the DD, so that is a lie.
2. When confronted with the XML of the bounce from BACS they say we only "set up" the DD on the first payment, and not weeks before. That is also a lie.
3. When confronted with a screen shot of their own on-line banking showing that is a lie, and DD was set up before, they invent new problems. They invent the idea of a DD instruction having an "end date".

They are totally unable to explain their actions. The last straw was this imaginary final collection date ("end date"). We have worked it out, it is on their screen as "last collection", we suspect. We even asked for the date for a load of other DDs set up on the account, and all were in the past. Clearly "last" collection is not a "final" date but just the date the last collection was made. But it shows they will invent bullshit to excuse themselves. They also blamed "keying errors" by A&A or our sponsoring bank, but could not explain the two payments that did go through. They seem adamant that the sponsoring bank are involved in AUDDIS submissions even though we are "direct submitters" to BACS/VOCA.

So how to fix - we are working through channels, via BACS (who confirm that error is only valid where payer rejects or within 2 days of AUDDIS), and via sponsoring bank. No idea if it will work.

However, we think we may have a simpler approach. It is a bank error, so covered by DD guarantee for consequential losses. So I am tempted to say that the admin fee for a bounced DD with that error is £1000+VAT. That way, anyone suffering this gets to claim from their bank, involving the financial ombudsman if needed. If we get any that are actually paid, we'll split it with you as a £500 credit off future service bills. Anyone wanting to try this is welcome to talk to me on irc - we'll formally agree the charge and then submit the fully notified DD collections.

Wunch of....

P.S. it is probably not valid to make up an admin fee to share with our customer, sadly, but I think we should at least charge our standard admin fee if they don't sort it.

HMRC RTI Hash up

So, just started looking at the HMRC RTI stuff. I mean, how hard can it be...

The first step is that to send the data to HMRC you have to go via the government gateway. The whole lot is XML, which is, itself, not an issue. XML is fine, it is the way people use XML that drives one mad.

So, gov gateway uses a SOAP style envelope posted via https - simple stuff. The envelope includes login credentials which can be plain text, MD5 or proper certificate signed. MD5 is not much better than plain text as it can be replayed, but does not expose the original password. In any case it is all over https, so not too bad either way.

The MD5 is interesting as the steps are (a) convert to lower case and UTF-8, (b) Make MD5, (c) encode as base64. So I wonder, why make the password case insensitive? Who does that? Then I pondered more - UTF-8 means they understand it could be non ASCII. Do I have to convert an upper case Omega to lower case? What of accented latin characters, or latin characters with accent modifier characters. Doing all that would be a lot of work. I decided not to bother asking and just use tolower() and hope for the best :-)

Then we get to the actual payload, a single XML object within the in the envelope. Again, standard stuff, but for HMRC this needs a hash. This time it is an SHA1 and base64 coded. This would not be too bad if it was an SHA1 of the data as sent, but there are some gotchas.

For a start, the hash is within the message, so you have to make the message, then make a hash, then remake the message with the hash within it. You have to be sure of making the object the same both times (which is fine unless your code uses counts of namespaces usage to control local tagging, etc, and adding one more item could change things).

What is worse though is the output has to be canonical XML. This would be fine if it was just textual normalisation, but it is not - you have to order the attributes in the elements alphabetically. Well, again, not too bad until you realise the order has to use the original URI from the namespace prefixes to order. This means full XML parsing in order to normalise the output. This is fine except you have to retain all the extraneous spacing which my XML parser does not do, as spacing between elements is used in the hash. Anyway, as we are only generating this we can generate normalised canonical XML in the first place and so make a hash.

So far so good. You make a hash of the HMRC message, embed in the message, and then pass to the gov gateway handling code to include in an envelope and send. Well, no! The hash is not over the HMRC message (i.e. from start of <IRenvelope ...> to end of </IRenvelope>) it is over the encompassing element in the sending envelope. This means the embedded IRmark hash has to be worked out in light of the sending protocol. It is just adding <Body> to start and </Body> to end, in theory, except it has to know the spacing that the wrapper will use between <Body> and <IRenvelope>. Also, just for fun, the <Body> is not as sent, but with the xmlns needed from its parent envelope. In short you can only add the IRmark in the envelope handling code so mixing the logic between layers - messy.

Anyway, once you get passed all that, there is another hash. This time for the BACS payment, but not MD5, or SHA1, no, this time SHA256. And not base64 coded, but lower case hex coded. Just to add more to the mix.

Anyway, now to integrate with payroll...

Update: This is all simple stuff, and HMRC were keen for us to be ready by 6th April, but they are closed! Their systems are down for maintenance! Arrrg.

April fool date command

Why does the linux date command not understand tomorrow or yesterday properly.

A script run at 23:00 on 30th March checked date -dtomorrow +%k was 01 to confirm it is running on last day of the month, but the damn date command just adds 24 hours, instead of getting to same time the next day, resulting in just after 00:00 on 1st April. The same script then ran 23:00 on 31st and ran again...

A script ran around 00:30 this morning that creates a directory based on date -dyesterday again got the wrong day...

Both scripts being fixed. But really... Arrrrrrg!