Wednesday, 8 November 2017

CISCO & FireBrick?

As ever this is very much my personal opinion... I hope it makes sense.

I used to be somewhat in awe of the likes of CISCO. After all they made these big routers and profession kit that everyone uses. When we started making FireBrick routers I felt a bit like we were sort of a Mickey Mouse company by comparison. Just a few developers working on hardware and software, and doing it from scratch, following RFCs, even making an operating system from scratch.

But oddly, over time, that view has changed both ways. I am actually feeling that what we make is very much "proper" in so many ways. We are out there, and A&A run an ISP on largely FireBrick kit. Many small ISPs use the same kit, and loads of end users make use of FireBrick firewalls. It works. But of late we have some CISCO switches in the A&A network. I sort of swore I would never use anything but FireBrick, but we have not got 10Gb/s FireBrick switches yet, so we had to get them. I wish we had FireBrick 10Gb/s switches to be honest :-)

They are actually pretty good. I will say one thing for CISCO, they can make some impressive hardware. Some fast switches and fast ASICs in their routers. We are not a patch on that. Our hardware is fast but is designed to do software routing very fast. So we are there at a few Gb/s, and CISCO are there with boxes that do way way more. We're getting there - the 10Gb/s+ box is on the drawing board, obviously. But we will never manage anything like CISCO's top end, well, probably not. We can hope.

But using CISCO kit has been a challenge. It has quirks and bugs and things that will drive you mad. I never realised this until we started using it. We have seen error cases, in one case a significant A&A outage, that seems largely attributable to the switches playing up in odd ways. CISCO TAC (the support side) is expensive, but a necessary cost with CISCO kit.

I do not see our hardware equalling CISCOs top end, as they have very fast kit, but I see our hardware working well at the level it is designed to work.

Do I see CISCO software on a similar level? Well, I am beginning to think so in some ways. CISCO have way more software, and way more things they do in their kit. We have focused on what we need for the products and added lots of features, but nothing like the features CISCO have. The down side is that CISCO have loads of old and legacy code and protocols to maintain. We have less code, and newer code to maintain. So this makes things a tad more equal in terms of things that can go wrong.

We are looking at a quirk in the CISCO switches today, and will be tomorrow. An odd L2 or L3 routing issue. It should not happen, and will be a lot of head scratching and may involve calling TAC. I am sure it will be resolved, but it is a good example of the sort of problems one can have. If we had the same problem on FireBrick kit we would know a lot more, but that stands to reason. We are not without issues, obviously, but I would not say we have more than CISCO.

So with all of this I was rather interested in the views of a potential customer recently, which sort of fits with how I now feel about companies like CISCO, and comparing to FireBrick.

Firstly the customer was keen to use FireBrick as he needed IPv6 that works properly. Now, I have not tried any CISCO kit with IPv6 - we are using switches at present with a small amount of IPv4 BGP to carriers, and that is it. But the view is that FireBrick have been doing IPv6 for like a decade. Our current code base was designed with IPv6 from scratch. IPv6 is in serious use in A&A using FireBrick kit. One advantage of not having all that legacy code. So actually we were seen as more "mature" in handling IPv6, which is good news. Score one for FireBrick.

Secondly, for the scale of operation the customer needs, we can do the job with the 2Gb/s capable FB6000 series, but we are more expensive than a CISCO box that can do the same. There is not quite the same second hand and used market for FireBrick as with CISCO (yet). But that is until you factor in TAC. The support for CISCO is not cheap, and suddenly we came out on top on price when considering only a few years of support. Just our policy on free s/w updates is a huge win. So score again for FireBrick.

So here I am, well down the line in terms of FireBrick as a product and a company, many years of work (almost two decades), and I realise that we are a "contender". Our code and our hardware is far from Mickey Mouse, and even the big players have their off days and their issues. I realise that we have a really good product, and it seems from the enquiries and recent sales that customers are seeing this too.

We are still going, and the new FB2900 product is due out real soon now (boards exist, EMC testing passed, stuff on order, even rack mount kits!). We are starting on the new FB9000, with is a 10Gb/s+ ISP grade router and LNS. The future is looking bright.

P.S. Sitting down with print outs of CISCO configs and marker pens, and half an hour later we found the smoking gun - actually a config error on our part. Very odd config attributed to someone that does not work for us now, but we all have a much clearer understanding of how it works now, and the problem we were chasing is fixed and not a CISCO bug - phew.

14 comments:

  1. Pity the company that I used to work for hit the wall before I got to order, play and enjoy the FB2900!

    The beauty of smaller (boutique) companies is their ability to deliver what clients want. Some of the FireBrick features were as a direct result of the usage scenarios that we had (extras token for example). Try getting the big boys to add features that your edge usage scenario demands!

    ReplyDelete
  2. Can I treat myself to a 2900 for Christmas then?

    ReplyDelete
  3. Out of interest, have you looked at putting FireBrick software atop either white box switches or Open Compute Project Networking designs?

    Basically two different ways to get access to the fast ASICs that switches are based on, but put different software on top.

    ReplyDelete
  4. Excellent hardware and a fantastic ability to configure via XML OR TTW makes it a winner for me.

    ReplyDelete
  5. Who's this Mikey Mouse of which you speak? Has Mickey got an impostor?

    ReplyDelete
    Replies
    1. Ah, thanks (edited). I think he is Mickey's brother, and lives in Wales.

      Delete
  6. Our company and customers have also had outages due to bugs in Cisco IOS, but the nature of any software is when you use it enough you will encounter all sorts of weird, relatively one-off bugs. The other day one of our customers reported an issue with DHCP. No config errors, nothing stealing addresses, no L2 or L3 issues, but the relatively low end Cisco router did have an uptime of 18 months. One reboot, problem solved.

    Who knows how Firebricks would perform if your core network was 10x the size, 50x the size. Or if most of the Internet backbone was running on it. Lack of fast ports aside which kills modern scalability requirements stone dead, I suspect you would either end up with a lot of software forks, or a lot more code being added if it was widely used in bigger networks.

    When you have widely deployed kit, you have a lot of weird edge cases, a lot more features that specific customers need, and a lot more code. Microsoft and co will be in the exact same boat with Windows.

    "No one ever got fired for buying IBM" is still a thing.

    ReplyDelete
  7. I'm a cisco engineer and so am slightly biased in my hardware choices.

    I have have to try a firebrick one of these days, see what all the fuss is about ;).

    ReplyDelete
    Replies
    1. I understand and of course I am biased as well. I hope you like it if you do try one. It is very different to Cisco.

      Delete
  8. Of course, however good you are with Cisco kit, you literally wrote and built the firebrick kit from scratch, and can bug fix yourself (if any bugs exist) and all design decisions were made by you, so it fits exactly what you need!

    Other ISPs using Firebricks is of course a ringing endorsement though. Perhaps Firebrick could be the next UK Tech success story (a-la ARM).

    I'm somewhat surprised there is not more interest in Firebricks (as UK developed etc) from the likes of UK MOD.

    ReplyDelete
  9. Very much looking forward to the FB2900... maybe, in time, even a couple of FB6000 if they drop in price to 'make way' for the FB9000 as the new top-end model ;-)

    I *do* have a request though...

    I just came back from a datacentre visit to diagnose a FireBrick FB2700 which went offline a couple of days ago and, as has happened before with an end users' FireBrick FB2700, the PSU was faulty so it was duly swapped with a replacement which I had the foresight to ask for in advance (and was shipped without question, postage paid, by your helpful staff).

    To keep slightly on topic, I would *never* have got a free out-of-warranty PSU replacement from Cisco... even if I had a support contract with them... and that is one reason, among many, why I choose to use FireBricks on my network.

    Unfortunately though, while I had the unit open, I took the time to inspect the underside of the PCB to check for 'tin whiskers' shorting out the USB pins that was causing a variety of USB-related error messages to flood the telnet console of the device (this was suggested by one of your staff as a possibility for the errors).

    I was shocked to see how many dry solder joints plagued the bottom of that PCB and I have *never* seen that many on a piece of commercially-produced hardware before; took a couple of pictures with the phone on my camera before I reassembled the unit and put it back into production (the replacement PSU solved the issue!).

    So, can you please ensure that the FB2900 models which ship to the general public will go through a more rigorous quality control procedure than the FB2700 which made it into my hands ?

    Of course, the dry solder joints may not make much difference to the reliability of the unit (as I am not an electronics engineer although I do repair Sinclair ZX Spectrum PCBs as a hobby) but I know, from both private and public conversations that I have had with you, that you have poured many years of your life into the FireBrick in order to make it what it is - most users may never see the PCB inside their unit but I would like to think that you would take the same pride and care in manufacturing the PCB as you would the rest of the product.

    Thanks for listening!

    ReplyDelete
    Replies
    1. Please email the pictures to us, with serial number, that is a concern. The quality control is very high, and I know, for example, that the first batch of 2900 have all had X-ray inspection as well. The PSU design is also completely different on the FB2900.

      Delete
    2. Just to add, the pictures show that these are not dry joints. They are unleaded solder joints that do not look as nice as leaded solder joints and can often look like dry joints. Sorry for any concern this caused and thanks for sending the pictures.

      Delete
  10. I’m very much looking forward to the 2900 being released - to try it out with a somewhat unusual Microwave/FTTC hybrid connection.

    ReplyDelete