Amazon done wrong

We sell on Amazon, and for that I'll apologise at the start, but it does provide some convenience to some customers and mostly works.

A few weeks ago we got an email saying we have to upload dome documents or they will close the account, and several ongoing reminders. Oh shit...

The problem is that the staff, including me, have a login to the seller account and can do 99% of things on it, as needed, but we could not do this as it had to be the main seller account. OK, so log in to that. Easy.

Except it wants to SMS a number to one of two numbers. Both were set up (and so worked at the time), but are 01344 (i.e. landline) numbers, and between when set up and now Amazon have obviously fucked up their SMS stuff and cannot SMS either.

The interworking of SMS really is a shit show and needs OFCOM to actually do its job and ensure it works properly.

But they also for calling them with a code - yay - except that also does not work - no clue why. Both numbers valid for SMS and calls, and they do neither.

So I tried the "remove 2FA" and sent pictures of passport, or something. My accounts manager did the same several times. No joy.

Every single chat was just crazy, and got to being daily. I mean I am tempted to publish them - they are so mad. One of them insisted I wait 24 hours after trying before contacting them - I said I had done that - he asked why I did not contact them after waiting 24 hours - and I explained I did - this is me contacting them!! The chats were like that. I don't like to judge people as "stupid", honest, people are only sensible in the areas they know, and that applies to all of us. But really, these chats were "special".

Just to be clear - the only reason I could "chat" at all is I had a working amazon account from which to chat. If not I would be screwed.

I have my case handled and referred to these, all of which did not reply at all!

• Internal escalation team
• Account recovery team
• External escalation team

I finally got someone that insisted that another "remove 2FA" should work, and I tried it, and, well, it did. But the instructions were clear - once done I could log in with just a password.

This was a lie. I logged in, and it said "Enter verification code. For your security, we've sent the code to your phone ***-***-**23" which basically means 2FA, so not removed, but also a number I don't know. Not either of the two numbers before.

Many more chats. No way to progress this. Doing the "remove 2FA" again stalled as it needs the code sent to *23 to get as far as uploading a driving licence even. I kept on, emailed in again, and finally it changed.

Now I have option for OTP by email or by SMS to *23, yay, but then it again does "Enter verification code. For your security, we've sent the code to your phone ***-***-**23" as well. So email and SMS now! (yes, when I got SMS working it wanted email too, so yes, both now needed!).

Had there not been an actual mobile *23 I really think we would be screwed. They have zero way to sort this from what I can see. Removing 2FA is "still do 2FA using a *main* mobile number" which you cannot change until you log in!

Turns out, after some research, checking staff lists, and pestering someone on leave and at a party (really, thank you Mike), I found someone that had a number ending in 23, and they were indeed what was on Amazon. So they let me know the code. Yes, that is being fixed, they should not have been on the main login, really. History from when first set up.

So finally I am in - it took a couple of attempts, but now the 2FA is sorted.

Phew.

So back to step 1. What is it they need.

Well this...

Yep, it is a sort of Farage test I think!

Obviously I click Agree and submit.

And, well, predictable...

So, err...

So another chat, and really, I am at a loss on this chat...

Ok, long winded, but basically no clue why this was an issue "not understandable" somehow, and actually, no, there is no way to upload a screenshot.

Eventually got an email and replied with a screenshot. It is a "don't reply to this" email address, FFS.

What we did in the end for A&A 2FA

The system is OATH/TOTP 6 digit 30 second authenticator codes, set up by QR code. We have TRNGs we use for seeds that are 320 bits long.

On the accounts system we have gone for some flexibility. Option to SMS codes instead, but configurable, and configurable trust level to decide when to ask for a code. It is also a seed we hold so staff can ask for a code to check you are who you say you are (a useful feature on phone, irc, web chart, etc).

On the control pages (and the internal staff A&A systems) we have gone for encrypted TOTP seed and no SMS option. The seed is binary data, XOR'd with a stretched Argon2 hash of the password and a seed set for that purpose (i.e. the seed also has a random seed for its encryption), so no way to check you have right answer other than doing the Argon2 hash and checking an authenticator code, so not a shortcut to crack the password hash.

This means that on control pages the password change needs old password if you have 2FA set up, and expects an authenticator code as well. Some staff can override, but they will also look at account settings as part of deciding you are you!

I think, overall, we are doing well. Hashed passwords and 2FA with encrypted 2FA seeds.

There is always more to do, and more security to add, but this is an ongoing process.

Customers can now set up 2FA on A&A accounts and control pages if they wish - have fun.

The TOTP seed storage dilemma

When making any sort of login system you typically used a username and password. One of the key things you should do, if at all possible, is hash the password.

This means that we do not know the password people have used! We can check a password attempt against the hash (and then forget the password), but we never store the actual password.

If the user database was to be compromised in any way the attackers would not get real passwords, and so could not use them. More importantly, given so many people re-use passwords, it does not give then the passwords people are using on other systems.

So far, so good.

In additional to passwords we are now using two factor authentication using an authenticator app that provides a new 6 digit code every 30 seconds (Timed One Time Passwords). This uses a seed that is a long random number which is stored in the app and which we have to know as well - that way we can generate the same code and check it matches. We actually generate the codes for the last few minutes and check it matches any of them.

This creates two factors - one is something you know, being the username and password, and the other is something you have being the smartphone app or authenticator device. There are, of course, issues of ensuring the two have the same seed, but using a QR code on an https page seems a good compromise.

The problem is that we have to be able to see the seed to check the code, so it is not hidden. If that seed gets out then the authenticator is compromised as someone else can always generate the same codes. So in an ideal world we do not want to be storing this seed in the clear.

The small revelation I had this morning was that we could simply encrypt the seed with the plain text password. This means that when you provide the passwords and authenticator code, we can check the password, and knowing it is right we can use it to decrypt the seed and check the code, all in one go.

This is great, and I nearly rushed off an implemented it before realising a significant number of shortcomings with this.

One big problem is lost passwords. When changing password you always have to know old and new passwords so as to decrypt the seed with the old one and re-encrypt with the new one. This is fine for a change password form, but not for any sort of lost password reset process.

Indeed, at present, we use the authenticator code to validate the reset password request that is sent by email (so working email as well as using the authenticator code as two factors). However, if the authenticator code cannot be validated without the password, you cannot do that. You either have to trust just the email working, which is not ideal, or you have to find some other validation process as well. Also, when resetting a password you have no choice but to also issue a new authenticator seed and reset up the app with that new seed.

You also cannot use the code as a validator when talking to staff as they could only check the code unless they also have the password, and we would never want to ask a customer for their password.

So this creates a trade off - transparent storage of the seed on our systems and added convenience and some extra security on password reset, or encrypted seed on our system and some much more constrained processes and less convenience.

I can see we may end up with the underlying libraries allowing both options and using for different systems as appropriate.

Isn't security fun some times :-)

P.S. Read the comments - at least one important point I had not realised.

2FA on A&A control pages

I think we have 2FA sorted nicely on our accounts pages for A&A.

We have taken on board some constructive comments, and done things like "you can't set paranoid mode until you have confirmed that the app has been installed and used a code", and also not showing the QR code or seed again, once installed.

The trick to both of these was to use a different seed for codes sent by SMS than codes from the authenticator - that way we can tell if the authenticator has in fact been used and not just an SMS'd code. Obviously we did have to allow for the remote chance of code matching both seeds and ask for a new code in such cases.

I event made a video showing how to set it up!



The next step is 2FA on our control pages. But first it is worth explaining - yes, I know that really what you should do is identify a specific living individual in some way, and separately have associations of accounts or logins that they are allowed to access in various ways. The system we have does not do that, I know - we have the accounts logins and the control pages logins. The system we have also has a complex set of linkages such as dealer logins on control pages, and, of course, staff logins. Changing to a "single sign on" is a good idea, but a big step we'll tackle another day.

The plan is to use the same library and tools, and almost identical set up processes, as the accounts 2FA system. Indeed, some of the pages/scripts will simply be copied and amended for the different database structures in use on the control pages.

There is, however, one extra trick we can do, and it will be an extra button. As we have the "seed" for the 2FA on the accounts, we can simply have an option to "Copy 2FA from account" for an associated control pages login - why not? Obviously we will ask for password and a new code at the time to confirm you have the authenticator, and there is no reason to show a QR code when doing this. But this would reduce the number of authenticator entries you need installed and will be ideal for cases where it is one person handling accounts and technical issues - like most of our home customers.

This would just be an option though - customers can have separate control for accounts and technical, and have separate people and hence separate authenticators for the control pages if they want.

I hope that sounds sensible. However, I do plan to "let the dust settle" a bit, and see how the accounts 2FA works out before working on the 2FA on the control pages. Feedback welcome, as always.

Two factor authentication

I am working on some new two factor authentication for our systems.

Before I even started this, I actually updated the systems we have in place for managing password hashes to move to the password competition winner Argon2. It updates the hash on next login to our various systems.

However, a big step forward would be two factor authentication where in addition to a username and a password we ask for an extra bit of information.

From various research the way to do this is using TOTP (which is a timed OTP using OATH hashing system). Basically you have an app or device that provides a code every so often, and when you log in you have to enter the current code. We are using the default of 6 digit codes created ever 30 seconds.

There are quite a few issues with this, and a scarily large number of OTP and OATH and TOTP applications available. It is a well published standard.

The challenge is getting the "seed" or "key" in to the device, or if it is a hardware device, then from the device to us. The latter is something to tackle later as most people use a mobile app these days, so we make the seed/key and it has to get in to the app.

The answer is a QR coded URI and there is a "standard" for this. It encodes the settings in a standard format with the seed/key in BASE32 which can be read by various apps including Google Authenticator. Once read, it provides a code every 30 seconds.

At our end we need to store the seed, which ultimately has to be readable. But we have hash for password and a readable OTP seed, so two factors, which is a good start. There is no real way around storing the seed in a readable format, sadly.

But it does get quite complex, and this is what I am working through now.

1. How do you make sure setting up or resetting the TOTP is safe / authenticated. Current plan is texting a code as an alternative two factor authentication before we can disclose the seed/key as a QR code. Some actions need to be properly two factor authenticated.

2. Do we allow changes of password if not already TFA, and what of lost password - is that independent?

3. What access do staff have to reset or clear the TFA system? How do we defend against social engineering whilst not locking out genuine customers? How much staff training on social engineering can we do? Ho much staff time will this take?

4. What levels of control do we offer to customers, what degrees of paranoia do we support?

5. What of ancillary systems such as ordering or the CHAOS2 API? Current plan is ordering will require the TOTP code if that is set up at all, even if normal logins not requiring as "trusted browser".

It is never as simple as it sounds when looking purely at the technical side. Systems like this extend in to social engineering!

Anyway, we are starting with staff logins, and then moving to end user logins on our various systems offering, and even recommending, two factor authentication.

P.S. Yes I waited more than 5 minutes after taking that picture so that even if you know my username and password you cannot use the code. And yes, we also protect against replay attacks on the code.

P.P.S. After some feedback we now only show the QR code until the installation has been confirmed, then it is no longer shown. That means the SMS codes and the authenticator codes use different seeds so we can tell which was used (and check for a clash just in case). Thanks for the feedback, this helps security.