Making progress. I have (subject to lots of testing) reached a major milestone at last, which is that we now manage the LSA database.
This means discovering neighbours (Hello packets), electing the designated and backup router, starting associations with other routers correctly, exchanging LSA headers, loading the database, accepting updates and sending acks, sending flooding updates, originating router and network LSAs when needed, refreshing LSAs we originate, and generally being a part of an OSPF community.
There will be bugs! I have not tried area-border stuff at all. But this is, none the less, a major milestone. Only two key steps left: Injecting routes in to OSPF, and processing routes from OSPF (i.e. doing the SPF algorithm, etc).
What I am working on this morning, and have a lot of progress on for IPv4 at least, is injecting "external" LSAs. i.e. where the FireBrick has some external routes and wants to tell the OSPF world about them. This is carefully controlled - you don't want to leak the full table of BGP in to OSPF it seems. This will mostly be used for L2TP connected routes.
When coding, you have to cover every possible outcome in the code, so you find edge cases that should never happen and you have to cater for them. Sadly I have run in to a snag already.
In OSPFv2 the LSA-ID for an external route is the IP address of the route, but it is allowed to use the "host bits" to differentiate LSAs that overlap. e.g. if you have 10.0.0.0/16 and 10.0.0.0/24 they cannot both use 10.0.0.0 as the LSA-ID. There is an algorithm to try and sort clashes, which is itself messy as it means changing previously announced routes when adding a new one that overlaps.
The snag is that this does not work for all cases. e.g. (and we'll test this on bird later), what if I have 10.0.0.0/32 and 10.0.0.1/32 and 10.0.0.0/31. There are only two LSA-IDs that could be used, 10.0.0.0 and 10.0.0.1 but I have three routes to announce.
Now, one can consider that the 10.0.0.0/31 is not needed as covered completely by the two /32s. That is fine, until one of the /32s is withdrawn. Now you have to reinstate the /31 covering route. This means every time you withdraw a route you have to check for all containing routes that may have not been announced just in case. Arrrrg!
So, once again, OSPF is piling more special cases in my code, which is more to go wrong, and more to test.
Some protocols and algorithms are "elegant" and "simple". I really do start to feel that OSPF is not either of these. Sorry.
P.S. I wonder what bird does with 0.0.0.0/32 as the LSA-ID 0.0.0.0 is reserved for 0.0.0.0/
Update: OK, coding for IPv4 (OSPFv2) working, picking an LSA-ID and handling clashes by removing the covering route where all subordinate space is occupied, and re-instating it when there is a gap. We are treating 0.0.0.0/x the same, so that means we will not use 0.0.0.0 for default route if there is a 0.0.0.0/x route as well, but that is not exactly likely. Now to originate IPv6 routes and that will be next milestone!