Thursday, 31 December 2015

Royal Mail Shipping API - finally working

Since Royal Mail are pulling the plug on Smart Stamp, tomorrow, we have been rushing to get an alternative postage system in place. The obvious choice was the Royal Mail Shipping API, which they document on their web site.

This allows us to print postage, which is purchased "on account" rather than pre-paid on card like Smart Stamp.

The API uses XML and SOAP, and seems to be documented in the technical specification on the web site. There is an on-boarding platform to allow testing, and a check list of tests done before you can go live.

Frustratingly, in spite of a couple of months notice it took until 3pm the day before Christmas Eve before we got any credentials to test. All of my testing failed with "Authorisation Failure". Looking at the the documentation and the information they sent over this is clearly a common problem. There is a lot in the technical specification and they have code samples for generating the authentication strings in the XML in php, python, exe, and others.

However, today (yes, very last minute) we finally have this working on their live platform, so here is a list of the issues which I hope they can fix, and others can learn from.

Client certificate

Even though the https uses a self signed certificate their end, they want you to use a specific client certificate they issue. This means we'll have to faff with renewing it every year or so.

Why the certificate is not the only authentication is beyond me, or why, when using https, they do not simply use plain text http authentication or plain text SOAP authentication, is also a mystery.

Authentication

The authentication uses SOAP WSSE Password Digest authentication. This is well document, and involves sending a Created date/time, a Nonce random value to avoid replay attacks, and a Password which is a password digest. The digest is defined for WSSE as SHA1(Created+Nonce+Password) and both nonce and the digest are sent in Base64 in the XML.

What Royal Mail expect is: (a) Nonce is 16 byte binary value (b) "Password" as used in the WSSE hash is actually Base64(SHA1(PlainTextPassword)). This means you send Base64(SHA1(Created+None+Base64(SHA1(PlainTextPassword))))

Their documentation does not exactly make this clear at all, and contradicts itself,. Their example code does not do this, it uses just SHA-1(PlainTextPassword) as the password in the WSSE algorithm, so generates credentials that do not work. Their help desk would not actually make an XML file that works to the actual on-boarding system. They confirmed the examples I made using their own code fragments were "right" when there were not, and only used some SOAPUI test platform rather than using the actual on-boarding system. This wasted a LOT of my time.

I note that at least one example used a timestamp that was local time with a Z suffix (meaning UTC), so I half expect this to all go horribly wrong at the end of March.

ApplicationId

The application ID looks to be sort of your account number, a 10 digit code. But for some reason both the on-boarding and live systems cannot actually work with the right application ID, and helpfully just say authorisation failed. They have provided a different one to use for now. This just compounds the confusion over the authentication hash as trying either way of doing the hash still fails when using the right application ID.

Also, they quoted the application ID wrongly in the first place in their email, just to add to the fun.

Schema Errors

The XML you send is checked against the schema, and a single character out of place just says Schema Validation Failed. No clue where it got to. This is a nightmare to debug.

This is made worse by the specification actually quoting at least one object name incorrectly, so if you follow the spec you will get a schema validation error. For example, object unitofMeasure is actually unitOfMeasure.

XML namespace

Anyone that uses an XML library or understands XML namespaces will know how they work - each object and even attribute can have an associated namespace, which is itself usually a long URI.

These are then typically abbreviated to an arbitrary prefix. So soap:Envelope is saying that the Envelope object is in the soap namespace, but there is an xmlns:soap attribute somewhere at or above that object saying what the namespace actually is.

The tags you use are normally totally arbitrary, however, when I tried using different tags to those they used it failed validation! I don't know if this is only the on boarding system, but it means they are not doing XML right. Thankfully, using the tags they expect is not that hard, but it a requirement they should make clear.

It gets worse though, and this may only be on-boarding again, but the location of the namespace matters! Not for the schema validation, no, that would be too easy, but for authentication. My XML library places the namespace declaration as close to the outer most usage of that namespace. But doing that means you get the ever unhelpful Authorisation Failed error. This again compounds the confusion of the hash used. Thankfully my XML library has a mode that puts all namespace declarations on the top level, and that was good enough for the on boarding system (albeit not quite matching their examples).

Rather annoyingly they mix some objects using V1 of their API with some using V2. They also have subordinate objects that are in the default XML namespace (so untagged), which means you cannot simply make the whole message have an xmlns at the top level and avoid using tags. It is just messy.

Service Matrix

They have a service matrix - the services they offer are a combination of a service type, service offering, service format and enhancements, and then separate options for signature. It is actually really messy, e.g. for recorded delivery you use a service enhancement (a number) on a normal service list 1st class post, and do not set the "signature" field! Basically, we had to make a table of what things to set for each service we wanted to use. But we don't have a clear list of options from Royal Mail, i.e. other than just the title of each so not clear initially that CRL is Royal Mail 24 and Royal Mail 48 which are account (cheaper) versions of STL 1st and 2nd class! We hope to find out more next week, and something about the tracked but not signed for options which we want to use for routers.


I hope this is useful to anyone else doing this... P.S. I have C code for this - ask me on irc.

37 comments:

  1. If you have been reading some of the other threads around this you may be aware that I have struggled through this as well. To follow on for RevK's post I also have it working in VB.net with the application ID assigned to our account. I will put this up on github in the next couple of days once I have finished the library, ill pop back and post the link.

    ReplyDelete
    Replies
    1. Hi Dave, I'm curious, have you uploaded this to GitHub yet? I'm reviewing the API documentation, planning to write integration code, however, if someone else has already done the heavy lifting, I'd be happy to use it.

      Delete
    2. I would also be very interested! - I've been banging my head against a wall for so long trying to figure this out. I'm getting schema issues whereby my request doesn't quite match theirs. I think it's to do with the prefixes they are using, and my WSDL/soap output isn't using them the same.

      Delete
    3. Hi Dave, as another frustrated customer of RM (the API documentation is a joke), if you could post the VB code it would me much appreciated. I'm starting to dream of E0007 Authorisation failure!

      Delete
    4. This comment has been removed by the author.

      Delete
    5. Hi Dave, i have wasted many hours trying to make a call to royal mail using the shipping API without success, i would greatly appreciate any VB code you can share to help me proceed.
      Cheers

      Delete
    6. Hi Dave,
      Looks like you have not got the code to github yet. Did you have problems with it? Does anyone know anywhere where there is some vb.net or c# example code that I could start with?
      Cheers

      Delete
    7. Dave, Thank you for your insights which have helped. Doing this API is VERY PAINFUL, RM do not make it easy, as of June 2016 there is STILL things not in the Spec.

      Delete
    8. Dave, Thank you for your insights which have helped. Doing this API is VERY PAINFUL, RM do not make it easy, as of June 2016 there is STILL things not in the Spec.

      Delete
  2. Hi all - I got code working to the point of being able to communicate with them - getting schema validation error, but I managed to figure out what I was doing wrong with digital certificate

    So 1 step forward :-)

    ReplyDelete
  3. Programs depending on the namespace tags, rather than the URI, is so common that you might as well start out by assuming that is what's happening in any new system you see. This tends to mean that their underlying processing doesn't support namespaces at all -- they've just got hardwired tag names that happen to have colons in them (talk about missing the point...)

    ReplyDelete
  4. So now I am onto altering namespace - grrrr

    I am overriding EndpointBehavior etc etc

    but the class that I was using to disply the xml sent stops working correctly now

    would that be expected?

    will I have to use something like fiddler?



    ReplyDelete
    Replies
    1. Sounds like VB or NET and I have no idea on that, sorry. Our stuff is all C code on linux, which I am happy to share.

      Delete
    2. Hi - can you supply a link please. I am really finding this difficult.

      Delete
  5. Thank you for that offer - I eventually worked out why

    the BeforeSendRequest is called before the 3 new class overrides start to deal with the message and do replaces

    So I downloaded Fiddler - struggled with it
    So I downloaded Charles - and hey presto

    I can see the XML outgoing soap (which really does help when trying to get such exact xml)

    Now that I am getting there, I feel I can keep going, and I am now down right determined to get it to work

    So thank you for offer of code - if I get utterly stuck I may take you up on that, but right now I am enjoying the glow of small triumphs

    Royal Mail phoned today and my partner passed on my unhappiness
    Sadly I was at work, I would truly have loved to pass on some messages in person

    They actually want to send someone to show us how to use DMO

    thanks but no thanks.....think I can figure out the website just fine

    ReplyDelete
  6. Big Grin

    Got it all working :-) kinda chuffed since it's been 15 years since I did any vb.net

    have even managed to automate sending the PDF direct to the zebra printer

    So, just bought Dymo USB postal scales - next little challenge - to pull weights directly into application


    :-)

    ReplyDelete
    Replies
    1. Hi, as someone getting nowhere fast with the API would you be able post the VB.net code? It would be much appreciated, I'm starting to dream of E0007 Authorisation Failure!!

      Delete
    2. Hi
      I have got to the stage of the dreaded namespaces using VB.net, I have created the 3 classes but now seem to be stuck again, where is the BeforeSendRequest located that you mentioned above? maybe this will get me moving on again :)

      Delete
    3. This comment has been removed by the author.

      Delete
  7. Have 1 question

    Digital certificate prompts for password - is it possible or advisable to provide this in code to remove need to enter password?

    ReplyDelete
    Replies
    1. That is one to consider as security policy - if the password is in the code, or command, or whatever is that any more secure than having no password on the certificate? If the whole thing is pretty safe, locked down to what user runs it, etc, then maybe no need for password. But I cannot really say for your situation.

      Delete
  8. Is there any experience here with the new V2 of the API which supposedly does away with the need for a certificate and instead uses a ClientId and ClientSecret which are, according to the documentation, to be sent as HTTP Headers of X-IBM-Client-Id and X-IBM-Client-Secret.
    I am using a basic WCF client created from the wsdl but whatever settings I try I cannot get through the error "The HTTP request is unauthorized with client authentication scheme 'Anonymous'. The authentication header received from the server was 'default'."
    Has anyone had similar experience with the new API ?

    ReplyDelete
    Replies
    1. Not heard of a new API. Sounds almost as bad!

      Delete
    2. Did you ever resolve this issue Roger?
      i'm having the same issue at the moment and would be grateful for any help you can provide if you have already solved it

      Delete
    3. Full C# Royal Mail Shipping v2 solution available here: http://stackoverflow.com/questions/38896117/royal-mail-shipping-api-v2-soap-authentication-c-sharp

      Delete
    4. I just solved the unauthorized issue. You would need include X-IBM-Client-Id and X-IBM-Client-Secret in the request header. I used the method in this link solved the problem: http://stackoverflow.com/questions/38896117/royal-mail-shipping-api-v2-soap-authentication-c-sharp

      Delete
  9. Hi Rev,

    This has helped me out a lot! did you ever come across what the wsu:id was for in the usernameToken tag with the ws-security?

    ReplyDelete
    Replies
    1. I think we made shit up for that!

      Delete
  10. Good articles. Struggled with this and C# for days back in July. I posted some of my solution on SO a few months back, which might be of help to people reading this blog.

    The namespace issue was the biggest problem for me which was solved using the following: http://stackoverflow.com/questions/31595770/c-sharp-wcf-namespaces-move-to-header-use-ns-prefix/31597758#31597758

    I then posted some C# code in response to someone else struggling with this which should be enough to point anyone in the right direction: http://stackoverflow.com/questions/34508811/consume-wcf-royal-mail-api-in-c-sharp-console-application

    ReplyDelete
  11. It looks like this API is for parcels only - did you ever come across an API for standard letters (1st / 2nd class)?

    So for all I've found is their online web print service (https://www.royalmail.com/discounts-payment/online-postage/home) which has no API. Really want to print labels and postage directly if possible.

    ReplyDelete
    Replies
    1. No, does letters at franking rates as well!

      Delete
    2. Awesome! They should make that more obvious on their website! Franking rates without the need for a franking machine, contract, headache!

      Let the integration commence!

      Delete
    3. Wow, would even do the bloomin' Christmas cards with this. Excellent!

      Delete
  12. Hi RevK - do you have a git available of your code? or details to contact you on IRC? We are setting up our store back end and require the API for our team to use.

    ReplyDelete
    Replies
    1. I can email it if you need - though they are talking of changing the API for some reason.

      Delete
    2. That would be great thanks, if you could send it to ewan at relchron dot com that would be much appreciated. The Royal Mail are a law unto themselves ;)

      Delete
    3. Hi RevK,

      I'd be grateful for the code as well at scott at canuckconsulting dot com scott@canuckconsulting.com. Having an absolute nightmare getting this working!

      Thanks for taking the time to post about it.

      Scott

      Delete