Latest project is a time server.
So why? Well, they exist - you can buy a really nice LeoNTP server, with impressive specs. We see response times of 0.1ms, and it claims 100,000 requests a second. They also have a PPS output, and can do a calibrated 10MHz output apparently (I can't do that).
Can I get close using an ESP32? Well, sort of.
Challenges
It should be simple, in theory - a GPS module, capture CPU cycle count on PPS interrupt and use to get clock rate. Capture NMEA to get time for PPS. Capture cycle count on NTP packet and use reference cycle count, clock rate, and reference time to know exact time of day to fill in NTP reply. yay!
Of course it is never entirely that simple. I did all this and got a working system, but I could do better. My latency as measured on a FireBrick was 2.5ms. I did averaging of the PPS intervals to get a more consistent clock rate, and often the standard deviation on that was below 10ns! But not always. I then did a best of last 5 seconds in terms of a PPS interval to use as a reference - that way the odd delayed interrupt had no impact. That seems to work.
One thing I wanted was PPS interrupt at higher priority than Ethernet, but all GPIOs on an ESP32 are the same interrupt source. Bugger. Do a search and you see plenty of people pissed off about this. I found a fix, make the PPS a PCNT (Pulse count) which has an interrupt (count to 1) which can be set separately to the Ethernet interrupt. There is a trick to remember.
The hardware
The PCB design is not that hard now I have cracked Ethernet. A main PCB, with USB-C, DC input, PoE, and Ethernet.
I have since made an even more compact design.
This then connects to a GPS module, which you can hang out the window (or better, fit in a Stevenson screen). Link with 5 core cable (solid cat5 is ideal), and it can handle a few metres.
Then put both in a nice 3D printed case. See https://shop.revk.uk/ if you want to buy. £60 not £600!
Making it faster
The fact my response times where around 2.5ms was not ideal. I wanted better, but how.
First bodge was hook in to the Ethernet driver receive code and check for NTP packets as they come in, and do a direct reply. I got latency down to 1ms, yay!
But I can do better :-)
Scrap the Ethernet controller altogether. Write my own custom low latency driver. Dedicate CPU1 to PPS and Ethernet only. My custom driver can...
- Be reactive only - reply immediately to ARP, ND, and NTP.
- Being reactive, no need for mutex wrap on access to Ethernet chip.
- Work in interrupt (yes, nasty, but dedicated CPU) so not task switching.
- Know state of registers to avoid read-modify-write as used a lot in standard driver code.
- Do read fifo and write fifo with no clean-up between
And guess what - latency down to 0.1ms - bang on what I wanted. Indeed I have seen 0.077ms even.
Now, the logic is fun - it makes the Ethernet no use for anything but NTP. So I have made it (a) optional, and (b) normal Ethernet for first 2 mins so you can access it via Ethernet if needed (assuming you can control power/PoE to reset).
But once switched to low latency NTP only Ethernet, you have to use the WiFi for any access, management, MQTT, and so on.
Bugs
I found bugs in the Ethernet chip (KSZ8851SNL) which does not check IPv6 UDP checksums correctly!
I also found it almost impossible to convince the Ethernet chip to give me unicast, multicast, and broadcast packets - in spite of a lot of reading the data sheet and trial and error. It is now in promiscuous and relying on the switch to protect it. Even that makes no sense - you have to set "Allow any packet" and "Invert the input filter" which to mean means "allow no packets!". I may yet find a working setting. I have already spent a day on this.
I also found it used edge triggered interrupt with a check in task for ISR set as well on timer, I changed to level triggered and that seems to work without hanging.
I also found it nearly impossible to cleanly decommission the Ethernet driver in ESP IDF. I managed to take over interrupts and kill the task, and that was all. Anything more thorough was a nightmare. But that works.
What is it good for?
Well some people are putting in the UK NTP pool, it seems to meet the requirements well and work at a level similar to the rest of the pool.
But it is mainly aimed as being the main server in any business / office. Include with your pool NTP, but being local it will win.
I do not get close to even 10,000 requests a second, more like 5,000, but that is not needed for a typical business, even a large one.
LeoNTP is 112500 requests a second basically it can max out a 100Mb NIC and you are correct only pool servers need that sort of speeds. Nice project.
ReplyDeleteNice work, Adrian! LeoNTP serves 100,000 requests per second, not 10,000.
ReplyDeleteThanks for the correction. I’ve updated the post. That is impressive.
ReplyDeleteYou should look at IEEE1588 PTP instead of just NTP.
ReplyDeleteFor time, you'll need a GPSDO, with the oscillator needing to be OXCO or better, and then sync the PPS to a NIC via hardware. For a gigabit NIC, the NIC of choice is an Intel i210 card, which has the SDP headers that can take a PPS signal and sync PTP. For faster cards, you can also use most Mellanox cards but will require soldering the jumper. That's all the hardware you'll need for PTP and get nanosecond sync. Here's a tindie OCXO GPSDO. The same guy also has a rubidium oscillator if you want to go atomic.
The machine itself can then just run chrony and ptp4l that variously syncs the clock, runs the NTP server and behaves properly as PTP grandmaster for your network. You just need any low power PC with a PCIe slot for the NIC.
I'd be super grateful if you could serve NTP servers via A&A too.
like time.aa.net.uk already does?
DeleteWell yes, and it has devices like LeoNTP and may soon have these new boards as well.
DeleteFWIW the esp32-p4 supports 1588-2008. It's not clear to me that that hardware can help NTP ( I think it captures timestamps only on PTP packets) but maybe there's some way.
ReplyDeleteImpressed. Very impressed.
ReplyDeleteMay I make some suggestions for v2?
1 - Replace the Wago connectors on both boards with RJ45 sockets (but ideally a different colour to the Ethernet RJ45 socket) and wire for a standard Ethernet cable between the main PCB and GPS daughter board. Would look much neater and more professional. Would also allow the daughter-board connection to run through a patch cab (subject to distance - what is the maximum, I know you said "a few metres", but how many is "a few"? Is there a way of extending this distance?).
2 - Replace the power Wago with a 5.5mm barrel jack socket.
3 - Have the connectors (RJ45, power jack and USB) stick out a little (2mm?) from the PCBs so that they can lie flush with the outside of a case.
4 - Add screw holes to the PCBs so custom cases can more easily be made.
Some excellent ideas, and considered... But some issues. 1. Makes way bigger - the nice RJ45s I have recess in the PCB which won't work for the GPS module, and yes, different colour, but adds to cost. Not impossible. However, the signals are 3.3V, and I would not trust them over a couple of metres really - I have not tested. An alternative would be the GPS module bigger, with regulator so power to it is higher, and some RS485 balanced drivers for signals. That would actually make 4 pairs (power, and balanced tx/rx/pps) so fit an RJ45 nicely. Makes for a much bigger and more complex PCB both ends. 2. Maybe - they are a pain as so many different arrangements of voltage and polarity, so I try to avoid them to be honest, but I see your point. 3. The RJ45 is a nice low profile, but cant't easily go off the edge of the PCB - any more normal RJ45 means a thicker case or a bulge in the case - I don't think it is that much off an issue, but interesting feedback. 4. Not a bad idea - I may see if I can find space.
DeleteA possible idea may be any RJ12 - you can get 6 way RJ12 leads easily. Could maybe even supply with it. Still issue with size of connector, but less likely to be run more than a couple of metres.
DeleteOther than increasing the cost slightly, does size really matter? I think I'd rather have a (slightly) bigger unit than take the compromises made in making it smaller - but that's just me and I appreciate your points.
DeleteI do like the idea of a RS485 antenna box - again, size isn't too important, but the flexibility of being able to place it wherever I want at whatever distance I want is paramount.
I take your point about barrel jack sockets and I'd probably power via PoE anyway, so I'd use a custom case with just two RJ45 connectors open to the world.
Anyway, as I said, interesting and impressive project.
Indeed, and maybe even drop the DC option - PoE or USB is pretty comprehensive. I may look in to what would be needed to make an RS485 based GPS module, and yes, an RJ45 with that. Have to consider impact of accidental connection to Ethernet - probably not an issue. Also consider voltage to drive down it - e.g. if PSU on GPS board, I could send the 12V I get from PoE module (or 5V I get from USB, 12V being better). RS485 on twisted pair is no problem at all for quite long runs - 12V is probably fine as we'll be able to tolerate a drop down to 4V. Not sure if regulator switching nest to L86 GPS module is an issue. It may be a basis for an alternate design.
DeleteI also appreciate your comments on size - size has impact on cost, but I do get obsessed with small designs sometimes. My main aim for the GPS module was to be like, and the same size as, a typical GPS "puck" antenna. This is not actually a requirement, obviously. This has sent me down a new rabbit hole, so watch this space. Sort of depends on whether they are as popular as they seem when I get next batch.
Delete