Tuesday, 11 October 2016

To ten gigabits, and beyond!

Not quite buzz lightyear, but a challenge none the less.

I remember when we (A&A) started, turn of the century, with a 2Mb/s BT Central. We quickly upgraded, and I remember when we got our first 155Mb/s ATM BT Central link - that really made us a "proper ISP", and I think some comparison sites required it to be listed.

Times have changed and we now have a total of 6Gb/s spread over two carriers at present, we have a lot of 1Gb/s links to transit and peering. We have a large pool of 1Gb/s LNS that handle all of the traffic with plenty of space to spare and plenty more to expand.

But we have seen nearly 4Gb/s of customer traffic on occasion - not too much to handle, but growing, and time to take action.

Building our network to handle peak load, even when Apple do an iOS update, or there is football on streaming TV, or the Olympics, is hard work, but we have done well. We have done better than many other ISPs in some cases, but we always aim higher.

Our aim is not to be the bottleneck. This is not something most ISPs boast, to be honest, but we want the service to be of the highest quality all of the time. It costs a lot, which is why we are not the cheapest, but it is important to us to be the best. We also aim to be open with customers, which is why I am posting all of these details.

The next stage is 10Gb/s links. We have already upgraded our core network to use switches with 10Gb/s ports and 80Gb/s links between our main two London sites. But the next step is the 10Gb/s links out to the world.

The first step is upgrading links in to Talk Talk. We have asked Talk Talk to upgrade two links which will mean changing each from 1Gb/s to 10Gb/s. We expect it to happen soon, and I really hope that is a matter of weeks, but it is a tad out of our hands now. We have been trying to upgrade for a few months now based on our projections, but we have now seen some lines on TT back-haul showing some latency in the evenings some of the time (we've taken steps to improve this now anyway). Once this upgrade is in place we are making an order of magnitude increase in capacity which will last us some years. Even so, the switches we have allows us to add more and more 10Gb/s ports as needed.

The other important step is upgrading external links. We have a lot of 1Gb/s links, and at present this is working very well without causing signs of congestion for normal traffic. But we know that some times, such as iOS upgrade, some links can get really close to being full. So we are getting some 10Gb/s routers and upgrading some peering links soon. Again, this is expected in weeks, but depends on factors outside our control.

It is a brave new world for us, and a big step. But this means we have a massive amount of headroom for unexpected traffic bursts in future, and as always we aim to stay on top of demand and not be the bottleneck. Seeing at most 4Gb/s of traffic and having 24Gb/s of capacity will be good start!

If all goes well by Christmas, the plan is improved usage allowances on standard Home::1 - so watch this space.

Update: Current plan is week starting 7th Nov for the upgrade.

31 comments:

  1. Any plans to change unit costs or limits. Seems the ::1 service always gets the perks these days.

    ReplyDelete
    Replies
    1. We'll look in to it but seen very much as "legacy" type of tariff now.

      Delete
    2. Is it? That's the first I'd heard of it.

      What would you recommend a longtime user of a line-bonding "unit"ted tarriff move to, then? As far as I can tell Home::1 doesn't have line bonding (or at least doesn't mention it) and Office::1 is at least three times what I'm paying now (so: instant automatic no).

      I suspect my ideal nowadays would be a bonded VDSL/ADSL setup, with the ADSL used as fallback only for incoming traffic and weighted for outbound (I'd rather weight for inbound in accordance with the line capacities, but your end can't do that as far as I can tell). But that would still be unit-based, as far as I can see.

      Delete
    3. @NickAlcock - I had exactly this setup, with VDSL+ADSL bonded on units - I wanted to move the VDSL to Soho::1 and keep the ADSL as a backup, but there wasn't a sensible option within A&A so I'm in the middle of porting the ADSL out to a flat rate tariff on another ISP to keep as a backup.

      The only thing A&A can offer at the moment is to split the account so you have flat rate on one line and a completely separate units tariff on the other. Remarkably, I think they may have even had a straight face when they offered it to me too...

      Delete
    4. Personally I'd be really happy if the peak daytime usage was changed to 9am till 4:30pm. The icing on the cake would be to count all public holidays as weekends. :)

      Delete
    5. Not an appealing tariff for most cases, certainly - after Sky killed off Be, I needed a new ISP for the office, but units-based and Office::1 were both out of the question. SoHo::1 wasn't available at that point, but is probably too expensive anyway, and 2 Tb is too much usage. (So we ended up with more or less "Home::1 but for business", IPv6 and all, but from a different ISP.)

      The Home::1 and other packages say at the end "However, it would, in theory, be possible to bond with another line of the same type making 2 terabytes in total" - the underlying mechanisms at A&A should be capable of doing this, if you wanted to order it, and the traffic sent to you *is* balanced based on line speed, as you'd want, as I understand it.

      Delete
    6. Home::1 has been bondable for ages (since inception?); I've been doing 2 years now

      http://aa.net.uk/broadband-home1-info.html doesn't show any information about "additional login" now... It certainly did previously

      I guess it's http://www.aa.net.uk/broadband-extraline.html

      I pay:
      45/month for Home::1 with 300GB
      2 x 10/month for 2 copper lines
      10/month for "extra broadband service on same login (no extra usage allowance)"

      Delete
    7. David, we did the public holiday thing years ago.

      Delete
    8. Also hadn't realised that units tariffs were considered legacy, or that migrating my units-based TT-ADSL2+ circuit to VDSL would mean that it also switched back to BT backhaul :-(

      My only requirements are
      * a VDSL circuit at my house, which works well
      * billing with a VAT invoice to my company.

      Since the Home::1 product is specifically not for business use, and Office::1 is totally OTT at £250pcm I didn't think there was any alternative but to go for units-based. Did I miss something? Usage is only around 100GB/mo, 80% off-peak.

      Delete
    9. The ordering page defines "not for business use" as meaning "residential address and not VAT registered". This suggests that it *is* suitable for home-office workers, even if they're working for huge soulless multinationals, as long as they don't actually try to run the business out of the premises the Home::1 is bought from.

      It's not like it's physically possible for me to divide my network usage into 'work' and 'non-work', not even by VPN: some of my work VPN usage is 'non-work' and some of my non-VPN usage is 'work'.

      The info page states "It is possible to buy multiple Home::1 packages at the same address and have them work as a bonded set of lines", though if the lines are wildly different speeds, with corresponding weighting, it further says that the quota balances out, which is just perfect.

      Plus, VDSL + ADSL backup comes to cheaper than my current 3-unit ADSL, for, oh, about ten times as much bandwidth.

      The only worry is IPv4 space: I can't abandon my netblock and retain a functioning network. One hopes I could come to an arrangement to retain my current allocation: the info page says that this is possible "in some cases" which is the sort of weasel wording that worries me a bit. I suspect that since I'm crossgrading from A&A, retention of allocation and VoIP will be more or less automatic...

      This regrade is looking distinctly tempting!

      Delete
    10. Oh damn. The SoHo::1 page says

      "
      Bonding option for Home::1T and SoHo::1T is two lines at twice the cost and hence twice the allowance, not special packages with a lower price second line, no allowance balancing between lines, sorry."

      Given that units-based packages don't work when bonded with this, a minimum of £160/month is not a cost I can justify, and I'm not willing to give up the resiliency of two lines -- so I guess I'll be sticking with the expensive units package indefinitely, after all.

      A shame.

      Delete
  2. Is there a 10G Firebrick LNS in the works?

    ReplyDelete
  3. We have lots of clients using bonding, we would love to get them onto one of the terabyte services as we always get grief about large bills. Surely you can now do something with tariff based usage?

    ReplyDelete
  4. What border routers do A&A use? Some Firebrick variant or off the shelf Cisco/Juniper/whatever?

    ReplyDelete
    Replies
    1. At present the routers are FireBrick, but there is not yet a 10G one so will probably be a Cisco on some cknnections

      Delete
  5. I really like the units-based tariff. I rely on line-bonding (three lines), and as the previous poster has said, I don't see any mention of this being offered for e.g. Home::1. Also I have no idea what the colon-colon deals would mean in terms of IPv4 addressing, as I currently use a comfortably-sized IPv4 address block and so am happily NAT-free. This is one of the big reasons why I originally chose AA. Keep up the good work and I notice you have improved allowances for 20CN daytime usage. If you could improve the peak-time units rate for units-tariff users it would be much appreciated.

    ReplyDelete
  6. Ok saying legacy is perhaps overkill - units is still sold, but relatively niche.

    ReplyDelete
    Replies
    1. I'm pleased to hear you are not abandoning the units tariff - like others I find that neither the Home::1 (no IPV4 subnet) or Office::1 (too expensive, don't really need bonded lines) are quite right.

      But compared with the top-up prices on the ::1 tariffs the units prices are starting to look pretty expensive.

      Delete
    2. I think the main problem is that there doesn't seem to be a single-line flavour of Office::1. If the old units packages aren't ever going to benefit from lower costs I would happily switch, but I can't justify the extra cost of bonded lines for a home office.

      Delete
    3. SoHo::1 is sort of that

      Delete
    4. Yeah, I spotted that just after I posted - it's quite well buried, not linked in the menu and only mentioned (somewhat in passing) on the Home::1 page!

      Still a little pricey compared to units-based, but worth keeping an eye on in case that changes :)

      Delete
    5. Were one to switch from units to Soho::1 would it be necessary to give up an IPV4 /27 or can that be kept?

      It's currently more than I pay on units so I'm "watching that space" at the moment.

      Delete
    6. Is there a non-Terabyte (i.e. non-TalkTalk) version of SoHo::1, for people running small businesses who don't need a 2TB allowance or don't have access to TT backhaul?

      I can't see any mention of one, but given the hoops you have to jump through even to find out that SoHo::1T exists (you seem to have gone out of your way to make it difficult to find -- why?), it's possible that it exists, but just isn't mentioned anywhere...

      Delete
    7. Not something we are generally doing at present. In theory we do have a system set for billing and pricing a 100G or 200G SoHo::1, but not on the order form. We may launch at some point.

      Delete
    8. OK, I'll keep an eye open; thanks.

      But you really should put an entry for SoHo::1T on the Broadband pages! I wonder how many people have assumed (not unreasonably) that you only do Home::1 and Office::1, and have gone elsewhere as a result?

      Delete
    9. I agree that units pricing is legacy if you look at the way we use the net these days. Streaming has changed the game as far as bandwidth usage is concerned. Last week I had someone house sitting for me who managed to burn through 12 units in a week (he took a Monday off to stream some shows at 1080p and I had forgotten to mention daytime vs off peak units). If you bear in mind that some people have kids and half term dates vary you're definitely better off on home::1 and frankly probably best off on 1Tbyte. Even 100 Gig offered on the original home::1 looks positively tiny once you start streaming Terabyte all the way. I think units might have a place for telemetry or office email only pc installations but even I watch videos for work these days so you can't stretch it much.

      Delete
    10. Indeed, which is why I am saying units is more niche rather than legacy to be honest. And yes, we hope to increase the 100G/200G Home::1 tariff allowances some time soon.

      Delete
  7. If it's not commercially sensitive, what's the biggest cost for an ISP for upgrading their network? The hardware, training staff on new equipment, the "usage costs" from peers/providers, OpenReach costs, increased customer support (as more users/usage)?

    ReplyDelete
    Replies
    1. It's complicated, and it depends, sorry. We have invested in nice new switches and that means it is now a lot cheaper and easier to get the 10G to carriers, but they charge a lot to upgrade the fibres. 10G to peering is not too bad in price, except we have to invest in new 10G routers for that, and that is expensive. Usage costs depend on the carrier and their pricing model and can change over time. It is not simple for me to pin it down and say one thing is the biggest issue, sorry.

      Delete
  8. Is there any chance of Home::1T on FTTP?

    ReplyDelete
    Replies
    1. Only if/when we can get TT or someone else on similar pricing to offer us FTTP. At present they are not doing so, but we have asked, several times. On BT back-haul pricing, where we can get FTTP, it is not yet viable.

      Delete