Monday, 15 August 2016

Pointless tests at BT Martlesham

We're having another rather interesting discussion with BT. This time BT plc t/a Openreach.

This is technical, so google the acronyms, sorry. But even if you are not technical you should appreciate the stupidity going on here.

BT have a process for testing modems that work on VDSL / FTTC lines. They even have contract conditions about deploying modems and expect them to pass this test. This does not seem to stop people choosing their own equipment that does not pass the test, so not an issue.

Obviously we want to sell equipment that passes the test if we can. It is getting a bit tedious trying to sort this. But one router we sell, the Zyxel VMG 1312-B10A, was submitted for testing and passed. It is not perfect and we are looking at others, but it is probably the best of the bunch for a consumer route right now, so let's not argue on that for now.

We were really pleased it passed the test, as one of the annoyances with that router is that in bridge mode it does not handle 1508 byte packets which are needed for PPPoE with 1500 byte IP packets. This is complicated, but basically the slightly larger than normal packets allow full size 1500 byte packets to be used on the internet and remove a number of problems with smaller MTUs. The BT modems supplied with VDSL/FTTC (when they supplied modems) did handle these.

Just to do the sums, the Ethernet over VDSL always has one VLAN tag, and could have another or even another. So if you take 12 bytes MAC, 2 bytes type, 4 bytes VLAN, 4 bytes second VLAN, 1508 bytes payload you get 1530 total bytes of Ethernet frame for a 1500 byte IP frame wrapped in 8 bytes of PPPoE header. We are thinking they want to support an extra VLAN tag as they have a VLAN tag as mandatory on the VDSL, so that makes it 1534 bytes total.

The test includes a requirement in SIN498 which states:-

"The modem shall support an Ethernet frame size of between 68 and 1534 bytes. For clarity, this figure includes 4 bytes for the C-VLAN, and excludes bits allocated to pre-amble, Inter-Frame Gap, and Frame Check Sequence at the user network interface (UNI). Support for frame sizes above 1534 bytes (inclusive of C-VLAN) is not guaranteed."

The 68 is the min 64 byte Ethernet with their mandatory 4 byte VLAN tag, and the maximum is the PPPoE maximum baby jumbo frame we worked out above. This spec was designed to allow PPPoE with full 1500 byte IP frames, obviously.

So far so good. I think that is pretty clear. I think, to pass that, I have to be able to send any Ethernet packet size 68 to 1534 bytes and the modem must support it. The Zyxel does not!!! It only handles 1500 byte payload. So cannot handle 1500 byte IP in PPPoE, and requires lower MTU and fragmentation, which has problems.

We know the Zyxel hardware can handle this as a script you can manually run via the CLI after boot makes it work, so this is a silly issue for Zyxel. One small edit to startup scripts and they could comply.

We asked BT why they passed!

Oddly it seems they test with all packet sizes 68 to 1534, and if any packet size works in that range they pass. They have confirmed categorically that if a modem only supported packets of 68 bytes it would pass that part of the test.

So I asked what of a modem that supported more than 1534, and they confirmed that it would, but they only test up to 1534.

Now, I hope that, even if you cannot understand all of the technical aspects here, you can see the logical flaw here.

The number 1534 plays no part in whether a typical VDSL modem passes the test or not! A model supporting only packet sizes up to 1526 passes, as does a modem that supports packet sizes up to 9000.

What is extra odd is the change notes confirm they actually reduced this spec from 1536 to 1534 at one point. They changed a number that has no bearing on the test result.

I can only assume that BT Martlesham have lost the plot and do not know how to do a simple test like this, or understand a simple requirement, or understand Ethernet or IP or PPPoE at all. I thought they were quite technical! Indeed, the test spec (elsewhere in the SIN) talks of checking the maximum supported frame size, and so does not actually test the requirements.

What I did find in the change notes is: "Requirement R.ETH.1 – Text reworded to clarify that multiple MTU sizes can be supported, one of which must be 1534 bytes". Well, "one of which must be 1534 bytes" is pretty clear.

In summary, I cannot take the testing seriously any more. We'll be reviewing modems based on our own tests and not on BTs test spec from now on.

This also goes a little to some of OFCOMs work recently, on things like compensation for faults. This spec means that the service we buy from BT, when we got a BT modem (that had the same spec) as part of the service, it does not fit with selling an Internet Access Service. If they supplied modems that happened not to work on Ethernet packet sizes of 142 bytes long, but worked for the rest, it would pass their test and meet the definition of the service. But that is not a valid thing to do when selling an Internet Access Service. If you cannot actually have components from a near monopoly supplier like BT plc that allow you to actually sell an Internet Access Service, how can you possibly have any automatic compensation?


  1. I would have liked to be surprised reading this, but sadly not. For a company that's still investing R&D money into fleecing an antique copper network when it would be better invested into fibre and national Wireless networks things like this just seen like the tip of the ice berg. For all the rhetoric about how we currently have most of the country getting faster speeds than xyz country etc I think in a few years we stand a good chance of being one of the slowest as others rapidly invest and deploy better choices of infrastructure. Numerous places around the world are now making gigabit internet an available option with prices that would compete with our current VDSL offerings and we struggle to get a minimum of 2Mbps without involvement from regulators. Shame on you BT

    1. Same old story. Why did ISDN never take off here? Because BT charged a premium for it (and kept on doing it into the 2000s because they had a warehouse full of kit to use up). In Germany in the 1990s you got ISDN unless you specifically requested analogue, same price, no messing.

    2. I have a friend who had ISDN in the mid 1990s. He worked from home a lot and used it to run a VPN to work. This required the office to have an ISDN line as well, there was no ability to tunnel in via an ISP or anything like. At least both ends were on the same BT exchange so counted as a local call.

  2. Well it's obviously stupid, but equally you can see how the testers are simply testing to the letter of the (badly written) spec.

    SIN498 doesn't state "the modem shall support ALL ethernet frame sizes between 68 and 1534" it states "the modem shall support AN ethernet frame size between 68 and 1534".

    So as long as they can make at least one value from 68 to 1534 work then it's a pass.

    1. Except that clearly the intention is to handle all frames else the 1534 would not be part of the spec and the change notes would not say what it does.

      Anyway if it send AN Etherney frame of 1534 and it does not support it, it has failed, surely.

    2. As you say the author's intention is clear (to anyone who understands how ethernet works).

      But the letter of the spec appears to say that as long as the device supports at least one frame size in the range 68 to 1534 inclusive, then it is a pass.

      So the tester's logic would be e.g. "my test shows that this modem supports a frame size of 526, and 68<=526<=1534, therefore I have shown that the modem supports a frame size between 68 and 1534, therefore the test passes".

      And by the same (silly) logic, changing 1536 to 1534 in the spec did change its meaning, because it means that a hypothetical device only capable of passing 1536-byte frames would now fail when it previously passed.

    3. Maybe it is a simple typo, a missing Y! "The modem shall support ANY Ethernet frame size of between 68 and 1534 bytes" would make sense.

  3. What code has to be run (via telnet i presume) to enable baby jumbo frames???