DoH and VPNs and trust

We live in a strange world - where trust is a complex issue.

Once upon a time we would all trust the "authorities", i.e. the police and our own governments, but increasingly we live in a world where a lot of people have good reason (not criminal reasons, even) not to trust people.

The Internet is an especially complicated area where international players of all sorts come in to play, with commercial and political and criminal reasons to cause you concern.

The Internet protocols have been built on a lot of trust, but now we see some new mechanisms to help, two of these being DoH and VPNs.

DoH

DNS over https is one element, with DNSSEC being another. Using DoH means you use an https request to some external server to make your DNS requests.

An https request looks much like any other, and could as easily be your accessing facebook as accessing a DoH server. It is not something that can be snooped on, or selectively blocked.

If you do not trust your ISP to provide "clean" DNS without filtering or snooping, DoH allows you to choose someone else to trust. This is the problem, you have to trust someone, but you have a choice of who you trust.

In addition to DoH, you can also use DNSSEC to validate the accuracy of the responses. Using DoH means someone in the middle cannot snoop, or easily do any selective blocking. But whoever offers the DoH service could.

VPN

A VPN provider works in much the same way - you effectively choose a different "ISP" to provide your Internet access via the ISP you use. Again, choosing who to trust.

I was surprised how popular our own (unencrypted) L2TP service has been at A&A. In time we'll be offering IPsec based virtual ISP services too, I am sure.

Browsers doing DoH

Mozilla are working on using DoH in browsers, which means someone (like an ISP) cannot snoop, or selectively block, DNS requests. It is sad that this is even necessary. Note that AAISP do not filter or block any DNS, and have no plans to.

Oddly this upset ISPA, who has considered making Mozilla their "Internet villain" this year for DoH work.

This seems odd. If an ISP has an order to block some DNS, then they cannot block DoH, but so what? they are complying still!

I was surprised ISPA took that stance, even as a joke award for Internet villain. I can only hope they do not select them as the villain.

So A&A have donated the same amount as an ISPA membership, £2,940, to Mozilla. We have not been ISPA members for some time, but this is the first time I felt ISPA were perhaps taking views I did not really agree with.

We all benefit from the work of Mozilla so much every day, this seemed well worthwhile.

The problem with SNI (and domain fronting) - Heisenberg's SNI?

HTTPS/TLS means that a browser or app or client can access a website or other resource over an encrypted link.

Now, the way this usually works is things like https://aa.net.uk/ where the browser makes a connection and as part of that connection its says it wants aa.net.uk so the server knows to serve the certificate and key for aa.net.uk.

The problem is that the name of the service (SNI) is in the clear and so some censorship systems can spot this and block it.

Now, we have some generic hosting environments. Cloud services. The nice thing is they have loads of IP addresses for all sorts, so hard to block.

Domain fronting fools this but sending a different SNI (asking for a specific certificate, in the clear) that is "OK" and then talking to a server that is not the same as the SNI.

This fools censorship systems.

Some cloud services are blocking "domain fronting" like this (why?).

OK this is hard to explain... We could do SNI after DH logic, making it so the service you want is encrypted. But as that is before you authenticate (as you cannot authenticate yet as that depends on SNI) it could be "man in the middle" but if it is then the next step of authenticating will fail.

It is like Heisenberg - you can either see what "domain name" is being requested, or you allow the connection, one or the other.

So the logic is simplified - your have two choices :-

• You can detect what someone is accessing (see the SNI)
• You can allow a connection to work

Basically no way to detect it and allow it to work. It is one or the other, only!

Does TLSv1.3 do this, I have not checked. If not then maybe TLSv1.4 will.

Clearly this is possible, and necessary, ASAP. It seems a real shame to me that the SNI was ever "in the clear".

Network appliances and https

One of the challenges we have is how we have a network appliance like the FireBrick and https from the start...

We have a really good system for setting up the FireBrick with a proper hostname on the internet and https using ACME and Let's Encrypt, but we have to take a step back here...

This is a problem for everything from WiFi APs, switches, routers, firewalls, and indeed any network devices, is how to get started, and is an issue that is going to be really hard to solve.

Most such devices need some sort of initial set up, and the way to do that is a web page. So how does that work?

http

The simplest is just http, so you may, for example, connect a FireBrick to a laptop and access http://10.0.0.1/ to talk to it. That works, but is insecure http.

Now, http works, and is a simple protocol and works when connected directly to the FireBrick via  cable with no security issues. But it is something browsers are increasingly saying is BAD BAD BAD in red letters!

So not good. What would be better if a way to do https, somehow...

https

Using https is way better, it stops the browser getting upset. The problem is that there is no way to get a certificate for https://10.0.0.1/ for obvious reasons.

So what do people do - and to be honest, I am interested in feedback here.

What I have seen is devices that provide a certificate for their domain, some-ap.com. It will not match 10.0.0.1, but you can override on the browser to carry on. Oddly I see many that go for a non standard port as well like 8443 instead of 443 - why is that?

But nothing you do can actually guarantees the security, but you want the warnings to be minimal.

FireBrick

What we are planning to do is make a simple self signed cert to match the request, i.e. one for 10.0.0.1 if that is what you used. It means a warning, as you would expect, but you can then access the FireBrick via https, and avoid passive snooping. That is a step more than just using http.

The future?

We do need something!

Maybe some flag in the self signed cert? Maybe something that means browsers are less fussy if the connected device is local subnet, TTL 1 TCP working, and the cert confirms the MAC address, or something. I think even that could use spoofed MAC to divert a switch, but harder to do.

What we need is some way to say to a browser, yes, warn this is not that secure, but don't go all alarm bells ringing because clearly it is locally directly connected somehow (e.g. TTL 1, MAC in cert). So that embedded s/w in network appliances have some way to allow that initial and direct config to work via https. Maybe we need CAs that are MAC OUI level somehow?

I can't think of a clean way to do this to be honest. It has to have warnings, but can there be a standard way to do this that minimises the scary warnings somehow?

Something to ponder.

P.S. Interesting, some browsers send the IP as the SNI, e.g. 10.0.0.1, but some send no SNI if an IP address. All adds to the fun. Latest FireBrick alpha has the self signing code for testing now.

Learning to ACME

[Technical]

I thought it may be fun to explain what I have been doing over the last week in more technical detail... I have been coding ACME (which is the protocol to get a certificate issued for a domain) for FireBricks. It is aimed at making HTTPS set up really easy. Right now you have to install keys and a certificate manually, but ACME will make it simple and seamless.

JSON

The obvious first step is that the protocol talks to the ACME server using JSON. We send JSON objects and receive JSON objects back. It is all done over https to the ACME server.

I commented on JSON recently, and even with years of experience using XML, and in some cases converting XML to/from JSON to work with Javascript, I am thinking JSON is not bad, and seems quite well suited to this job.

Not the first time I have handled JSON, but I needed a JSON library for the FireBrick, which only took a few hours.

JWS

However, the JSON that we send to the ACME server is not just a simple JSON object, oh no. It is a JSON Web Signature protocol. This means you make some JSON, you then BASE64 code the JSON, and make that a "payload" field in an new JSON object. You also make some more JSON which is various fields defining the public key you are using. These fields (e.g. "e" and "n" for RSA) are BASE64 encoded. That chunk of JSON is then BASE64 encoded, and included as a "protected" field in JSON. Then a signature is made, BASE64 encoded and added as a "signature" field. So what you post is a JSON object with three BASE64 encoded fields, two of which are BASE64 of other JSON objects. Yes, I know, complicated, but I got all that working. Thankfully the reply is just a JSON object as normal.

BASE64 but not quite normal BASE64

Another fun detail is all of the BASE64 used is not normal BASE64, which is A-Za-z0-9+/ but a URL safe BASE64 which is A-Za-z0-9-_ instead. So even simple debugging using base64 command line tools on linux often failed. Also, normal BASE64 pads with = at the end, but this is all unpadded. I never really understood why padding is used anyway, so quite on board with that one. Fortunately BASE64 is a doddle.

Not just JSON

A somewhat frustrating part of the API is that it is not just about sending and receiving JSON objects. If only!

No, some of the key data you need is in the HTTP headers. Some of this is as per HTTP spec, but they did not have to do it that way - they could simply have sent all of the data you need within the JSON objects, or even duplicated in to JSON objects if they wanted. So you cannot use a simple HTTPS client to get a response, like curl, you have to also get selected header values as well. In some cases more than one such header.

So, my client library was updated to allow selected header extraction.

Replay-Nonce

There is also a special field, a nonce, a code issued by the server which you have to send in the next message you post. This is one of those header fields, but only if you POST something, not if you just GET, so you only grab this header on some of the interactions not all (arg!). You then use this in the JSON (not as a header!!) when you post the next item. This is all good in that it stops someone capturing an interaction and replaying it for their own use, but it is annoyingly inconsistent, header one way, JSON the other, for example. It is, however, included in what is signed in the JSON to avoid tampering.

JWK Thumbprint

This is special. It is a hash, BASE64 encoded, of a chunk of JSON which holds the public key (JWK). You send exactly this JSON as part of the ACME messages (in the "protected" part). It is also part of the response the web server has to send when challenged. You have to prove you own the domain by making the web server respond to an HTTP request with a specific value.

What puzzles me why not simple send a nice random string as part of the ACME protocol and expect me to respond with that?

But, no, we have to make this Thumbprint. However, this is where it gets a tad special. First off, the JSON has to be exactly right, with the exact fields you need in exactly the right order and no whitespace. If not, then the signature does match and all you know is it does not match!

Now, this is not a question of using the same JWK you sent in the ACME messages, no. They can be fields in any order, for example, and work. No, it has to be exactly right. However, the ACME accepts  it in that format so I can use one function to make it.

But it gets worse. The public key includes the "mod" value, which is a long string of bytes BASE64 encoded. A small note mentions that any leading zero bytes must be stripped. This is not needed for the ACME messages in JWS to work, but if you don't do it, you get a different JWK Thumbprint and so nothing works. It is not even quite what you might do in ASN.1 as the next byte has to not have the top bit set else you indicate the field is negative. This case is simply strip leading zero bytes. That took me hours of testing, comparing to examples, and re-reading the spec.

I am still quite surprised it is not simply some random string provided by the ACME message for the challenge.

Certificate Signing Request and ASN.1

Having got through the challenges and got as far as an authorised order I can send a final request with a CSR and get a certificate. yay!

But I have to make a CSR. So far the FireBrick code has has to decode ASN.1 for certificates and so on, but not generate much ASN.1 (SNMP is somewhat simplified in that area).

So, another couple of hours making an ASN.1 construction library, and then working out what goes in to a CSR. Thankfully tools like openssl will parse what I make at an ASN.1 and CSR level to tell me what I have.

ASN.1 is a bit like riding a bike. Every time you work on it, it all sort of comes back to you...

I am also really impressed with the Let's Encrypt staging server in terms of the error messages it returns. They tell me exactly what I have wrong.

It turns out the certificate only needs the common name, which makes sense as LE only sign that as that is all they have proved, so no need for company and locality and all that.

I was quite chuffed that the first attempt to make a signed CSR just worked, I got the signing right. That is rare in coding.

Two key pairs

So, I finally have a valid and signed CSR, and send that, and get an error telling me the key used for the "account" (all the messages to/from the ACME server, and for the JWK Thumbprint) must be different to the key for the domain (i.e. in the CSR).

So now I have to faff with a second set of keys and make sure they are used in the right place.

Finally

Finally we get the certificate and install as normal. Actually, for Let's Encrypt it is two certificate as they have an intermediary one as well.

Testing on a new box, I added a hostname to the config, and 4 seconds later we had working https using that hostname. That is how simple it should be :-)

Next

I have a lot of tidying to do, and we need to make this a bit more polished before a release of FireBrick with this in place.

One idea is handling more than one hostname. I think this will be less common, and originally we thought we would get one certificate with "alt" names on it. However that does leak all of the other names for a brick if you access one. So plan is separately getting a certificate for each, and probably a status page showing progress, and expiry and so on.

To be fair, the host names used with Let's Encrypt are published anyway, which may be an issue for some. But ACME should work with other CAs, though we may have to add extra fields if someone wants to do that.

There are also access control issues over HTTP access during the authentication stage which needs allowing TCP port 80 automatically, even if only for a few seconds, and also being locked down to just the ACME authentication and no other access via that. Not hard, but needs doing with option to turn off.

So, maybe next week we will have alpha releases for people to test.

P.S. Some work over weekend - much more polished, and much better error reporting. Really close to an alpha for customers to test now.

FB2900 and Let's Encrypt

Well, the FB2900 is out!

The retail prices are lower than the old FB2700, £500+VAT for base, and £550+VAT for fully loaded with £35+VAT for rack mount kit. We should have the DC powered models available soon.

We have gone for lower prices to encourage more take up in the SME market. It is a bit of a gamble, but this is a really good product - not just a gateway router handling multiple ISPs, but even a VoIP switch / PABX. Perfect for most small businesses and even some large businesses.

The delay, for a week or so, was down to wanting to ensure https was working - this meant a lot of loading Windows VMs and testing on all sorts of different browsers. It needs manual loading of key pair and cert but it works well. I am really impressed with the work of my colleague, Cliff, on this, as the end result is just as fast to use as http. Very impressed.

It is timely as safari, and I am sure others, are now getting quite pushy on sending any form to a site not using https.

But we have said we expect to release more new code soon. The FireBrick s/w has always been free, and we have ensured the older models FB2500, FB2700 and the FB6000 series, all have the update for https now. But the next code issue should make it a lot cooler.

First off, I am planning some simple self signed stuff so you can use https before setting anything up. This is a bit naff, but every other idea we have come up with has flaws, and it is what everyone else does. The key thing is that it stops passive snooping as a threat, but not not proper security.

You need a proper key pair, and certificate, to do https without warnings. The FB2900 have a key pair loaded individually as part of the production process which means we just need a certificate. The FB2500, FB2700 and FB6000 series will need a key pair loading. This is partly because we are not yet confident we can make a "good" key pair. We are very cautious when it comes to security, and this is an area that has gone wrong for others, so we want to be careful. When we are happy we can, we will, but whilst FB2900 has a hardware true random number generator, the older models do not, so it will not really help for non FB2900s.

But even with a key pair loaded, which is not hard, you need a certificate. This is where we plan to do way better than most embedded systems. We plan to use ACME with Let's Encrypt as standard!

So the idea is simple, tell the FireBrick its public hostname (and if not an FB2900 then load a key pair) and it will make a CSR, apply for a certificate from Let's Encrypt and install it and renew it as needed. Proper working https with no warnings and no faffing about renewing things. That's the plan.

The same certificates and keys can then be used for IPsec, obviously.

It is not that easy as it is aimed more at a traditional machine / server, and not an embedded device, but I believe we should be able to do that within a few weeks and have a new s/w release.

In the mean time, do enjoy the new s/w release for the whole range - which will be a formal release shortly after beta testers are done with it.

P.S. (18th April) All going well, and we expect to issue alpha code any day. Test bricks with just adding public host name working on https 4 seconds later. This is "fun" coding!

FireBrick FB2900

FireBricks have been around nearly two decades now, before things like https were a consideration. Whilst we have embraced IPv6 as part of the design of the current FireBricks from the start, https was not top of the list. Why? Well, the FireBrick web interface is usually only for management of the FireBrick. The idea is that most customers would have it is on a separate management LAN, or locally connected, of even behind an IPsec tunnel (which the FireBrick can do), so https was not actually needed that much.

However, https is more and more a thing and becoming so much the normal way of working (with browsers warning if not https even), so we are including it in the new FB2900, and the existing FB2500, FB2700, and FB6000 series as a free software upgrade.

In fact, working https, with an SSL Labs score of at least "A", is pretty much the reason for the current delay on the FB2900 launch. We have finally sorted the other issues which had added months and months to the launch of the FB2900, but as https is almost ready we are going to ensure the launch has https. It is literally a matter of days away - I have working https on my test FireBrick (SSL labs score "B") even now, thanks to hard work of my colleague Cliff.

We then follow on with ssh, and the plan is ACME support to use Let's Encrypt to make https really easy to install - point a domain/hostname at the brick's IP and bingo, it will be properly certified https. It will still have all of the access controls, but with caveats for ACME certificate renewals. The ACME Let's Encrypt certificates will help with IPsec configurations as well.

Sadly, one of the things we would have loved to do is impossible. We wanted a brick "out of the box" to work with https with no warnings. We could maybe include a cert for my.firebrick.uk or some variant to do this, but any means by which a FireBrick has a private key in the code would mean someone could get a FireBrick and JTAG or some such to extract it from the flash. It would allow that key to be extracted and misused. The only real answer will be for a FireBrick to have a unique key pair and obtain a signed certificate by ACME, or similar, and that can only happen after it has a public hostname and internet connection. So the initial set up will have to be over http or with a "security exception" to talk https. Typically this is literally a laptop connected to the FireBrick, so either is acceptable, but a shame no way to avoid that. It would be interesting to consider the ways embedded devices could solve that within an https and certificate framework one day (TTL 1 and tied to MAC address or something?).

So, FB2900 really close now... Many boxes on the shelves ready to ship... Watch this space!

P.S. I won't bore you with the days of work on the outer packaging shipping label featured in the image above. Lots of svg, barcodes, and postscript and stuff with UPCs and things. All very boring I am sure... :-)

P.P.S. We may forego the "A" rating at launch for the working on all main browsers and not add more delay.

P.P.P.S testers that can load "alpha" releases should hopefully have access to play with this in next day or so.