Wednesday, 22 March 2017

Keeping Customer Informed

We are starting on a fun project at A&A. Well, to be fair there are a few major projects going on, but this is likely to be one of mine rather than the ops team, at least to start with...

There is this horrid term BT use, "KCI", which is Keeping Customer Informed. They have these stages KCI-1, KCI-2, and so on.

But, in spite of the annoying term, the principle is reasonably good, and we are working on a system for A&A.

The basic concept is that there are a lot of cases where automated systems (usually) need to update a customer on some progress of something - whether an order or a fault or something else. At the moment we have a lot of systems, some of which get KCIs from BT or TT, and trigger events, some from our own systems, some as a result of an action by staff or a customer, etc. There are some consistent systems for some subsets of what we do, but nothing as a whole.

So the plan is to make a new system, a general purpose system, that can easily be bolted in to all of the systems in place of what we do now, and be consistent and helpful to customers.

The first issue we identified is there are two main grades of notice to customers. The simple "short message" type thing such as "order accepted by BT", "appointment booked for the 3rd, 8am-1pm", etc. The second type is more detailed long messages we currently send by email, such as the detailed order confirmations, or notices about open DNS servers, etc.

The short messages can be sent in many ways, and we currently, for some parts of our network, have messages with a choice of SMS, Twitter or email. These are all ideal for the simple short message type notifications.

So the plan is to allow customers to define, at various levels, e.g. a control page login, where they want notifications sent, and maybe even more than one place at once. Also, especially with text messages, time windows such as (8am-8pm Mon-Fri), etc, so not woken by unimportant messages. We may in future be able to extend to Signal, or WhatsApp, or whatever, where there are APIs.

We have to allow for message to be time sensitive, e.g. no point sending a message about an appointment after the appointment has happened. We may have to delay some messages, e.g. if a line is flapping, the line up/down messages get delayed (for both cost and annoyance reasons). We may also have to pair and cancel messages, e.g. if you have texts 8am-8pm but at 3am your line drops and reconnects a minute later then those two messages can cancel out and not be sent at 8am. We have to also consider load and rate limits on things like texts.

Now, when we get to emails it is also a bit fun. These can be used for the simple texts and for longer notices we send. We already try to sign most emails with an automation signature, but we are considering encrypting emails. We have been asked about this by a few customers, and we need a central system to handle this (makes no sense for everyone to have their own keyring). Our ticketing system could do it for us even.

So how would customers register a public key. Well, the plan is they email it to us, and we send them an encrypted email with a link to confirm the email address. Once done, we have a database of customer public keys and email addresses to use for sending email.

That is pretty simple, and the wonders of GPGME library have been impressing me for the last two days.

The huge problem is turning it off. Technically simple, and we can have have a staff interface for that, but the issue is policy. If someone wants encryption and has any risk of emails being read in transit, they do not want someone to just be able to phone up and turn off encrypted emails from us. Indeed, we cannot even sent a link to confirm which is not in fact encrypted to be safe. Sending an encrypted link will work for someone simply wanting to turn it off, but what of when someone loses a key??

Indeed, even accepting a replacement public key is tricky as it could be sent by someone that has means to intercept email, and they can then extract the confirmation link from our reply as they made the key.

Obviously the traditional face to face key signing is not practical on scale.

We could use customer login, 2FA, and so on, but how do we know the email they are using is really them. They could use their account but with someone else's email address which they have means to intercept, even if temporarily.

I am slightly at a loss on best practice on this at the moment. Comments welcome.

I suspect the best we will do is create policy and a good practice which minimises risk, but can never be bullet proof.

No guarantees on timescales yet, and it will be a gradual deployment, but watch this space...

P.S. PGP/MIME is a pile of shit, IMHO, so far.

P.S. One issue is that we are not expecting to register public keys "per account" or "per login" but "per target email address". We would send an encrypted reply with confirmation link. This makes validating changes or removal more complex.

P.S. After a lot of work on the library, I am the PGP/MIME king - all working as expected.


  1. Perhaps a silly idea, but might you encourage someone to add a photo to their public key (of a not too massive size, but not terrible resolution either) which they give to you and, if they want to change their public key, they need to use a video feed (Skype, FaceTime, or whatever else you support) to make the request to change, so there's some degree of visual validation?

    Or could you offer different levels of security (as with 2FA)? So someone could agree a lesser level of security for key changes, if they wished, and revoke the key through control/clueless or whatever it might be, or a heightened level which is less convenient for them (such as needing to post you a certified original copy of their driver's licence, or something bearing their address, or other credential which they have previously specified to you)?

  2. Where I work, all of our users use two-step authentication (via Duo), but for provisioning the whole process requires just a regular login (as two-step isn't set up yet). Afterwards, two-step is required to make changes.

    For key validation, I think it's a question of how much money you're willing to spend, but there is technology that can be leveraged.

    For example, if a customer has an A&A SIM that can receive texts, maybe you could encrypted-email the client a code, and ask the client to text the decrypted code to a number, with the client using their A&A SIM. But then you'd have to trust their phone, and the network.

    You could also use the mail: Email the client an encrypted number, whilst mailing the client another number. Have the client add the two numbers and send in the result. In that case, you're trusting the postal system and the client's mailbox.

    The common factor among all of that is, as email itself can't be trusted, supplement it with something else (or somethings else). You're in a position where (I assume), during the key enrollment process, you could simply cut a customer off from the Internet, giving them access to (proxied) DNS and the key enrollment site. I'd say that gives you lots of options!

  3. For broadband customers, some sort of requirement to submit the request via the web from the account in question? (I.e. now you've limited your possible attackers to those who live with you and people that have already compromised your computers)

    Not a solution alone and still needs to be verified by say, you calling the telephone number on the account to verbally verify the request with the account holder. All the better if on a mobile phone.

  4. Pushover would be a good notification system to support. The API is very easy to use and you can define a priority to the notification anything from just indicating on the badge icon right up to critical alerts which can be set to bypass any time restrictions.

  5. Side issue: you mentioned whatsapp: any chance of supporting iMessage? - for contacting AA staff too, as well as notifications. It has numerous advantages over IRC and SMS for me. I don't even know if there is a public API to it, mind you. (One staff member was kind enough to give me permission to iMessage him at an Apple ID associated with an AA email address. That was very helpful.)

    I can imagine that a lot of people would find the option of being able to use Facebook Messenger very useful indeed.

    1. We had iMessage for ages and nobody used it - I don't believe we can automate it though, so an app like Pushover would be simpler.

  6. You've been asked about encrypting mails by a few customers. Is there a business or operational need for you encrypting such mails (even with customers who haven't asked)? Or is it a case of a few geeks wanting to play secret decoder rings because why not? No harm in that but then they are capable of going through pgp best practices for managing keys, subkeys and revocation processes, because unless everyone is playing the pgp trust game it's a pain in the arse to operate.

    For normal people it's not clear to me what problem this is meant to resolve and I think that's where the uncertainty over the best way forward is coming from. You can do your KCI stuff regardless, and let the people who want pgp encryption work out their own contingency. Eg a pre-shared code between you.

    1. +1 It's technowankery.

      People who don't want supersecret info sent in emails could instead be sent a link to a web-page to read them, which authed though one of the existing 14 different ways of logging into A&A.

  7. For me, an important part of these kind of challenges (lost keys / lost 2FA device / etc) is they essentially can't be instant. There must be a couple of ways that you contact me to tell me the change will be made, the notifications must be repeated after a few days, and must leave at least a couple of days before the change happens.

    If it's possible to phone you, and get my encryption key reset there and then in the phone call, it is always going to attackable via social engineering. As I'd guess you know, several large companies have fallen foul to that kind of social engineering, eg.

    You can perhaps mitigate this to some extent by having multiple factors, e.g. confirm by calling back to user on their registered phone number, then royal mail them a code to their registered address they need to complete the process. The problem is that people change phone number and address and forget to tell you at the time... then you're into things like Neil's suggestion of certified copy of id.

    It's a very hard problem.