Tuesday, 28 November 2017

SVG all the things, now!

It started with "the icons look blurry" from a friend, and I was puzzled, as the icons look fine. We went to a lot of bother to make nice clean PNG images for the icons on the FireBrick config editor.

It turned in to a project taking me at least 4 days so far... I'll never be an artist!

Icons look blurry

Why did they look blurry? Well, the answer is simple - high resolution monitors! I have a 5k Mac, as does my friend, and on such a high resolution screen you just get used to the fact that all pixel graphics look blurry. To be fair, a photograph does not, but an icon, where things should have sharp edges, do not look right. I blogged an example of this recently. Here, I try to go in to a lot more detail.

Here are examples - the first is the original PNG, scaled up showing the square pixels. The second is the blurry version, cubic scaled up as would happen on a monitor that has much higher resolution. The third is a rasterised version of the SVG we now use.



The pixel version was good, it was the best you could do for the pixels available. The scaled version is pretty good, but is indeed blurry. What happens is the sharp transitions on the pixel version are softened. This makes a lot of sense with a photograph as it stops you seeing the harsh edges and makes for smooth transitions in colour.

The SVG though is perfect at any scale, and that is because it is vector based (the V in SVG). It involves lines, and shapes created using points and straight line segments and curves between those points. This is something you could draw at any size. A diagonal line is diagonal and not a jagged series of pixels making a "staircase" effect. Now, ultimately, this ends up on a screen that has pixels, but that conversion to pixels can happen at the last moment for the final resolution of the screen. If you actually had a vector output device, like a pen plotter, it could actually draw the vectors with no pixels.

That is what makes SVG very different. Vector graphics verses raster graphics. The age old issue. But now with computer screens, and even mobile phones, being such high resolution, we need things to be designed from the start as vector graphics. With the possible exception of Minecraft.

Changing this on the FireBrick was simple because we actually have SVG masters for all the icons. Our build process made PNGs from them. On a low level screen all three of those images would look the same in their original size. When all we had were screens that showed images 1px to 1px scale, it looked as perfect as it could.

Sol, given we had SVG, why did we use pixel based PNG. The reason is simple - when FireBrick started browsers did not support SVG, or need SVG as monitors were not so high resolution. Indeed, even javascript was not something on which one could rely. Finally we are at the stage where SVG is pretty much a given, as is javascript. So now we can use those nice vector originals we have had all along.

I can't just leave it alone, can I?

Basically, that was far too easy. It took my minutes as we had the SVG masters of the icons and could just use them.

So this led to my changing the CQM graphs... These are the loss/latency graphs we use for monitoring links, and A&A use for all broadband customers...

Existing graphs were pixel based in the same way, and when viewed on a same resolution screen they look good. They try to present a lot of data, and have options to only show some of the information on the graph, and to adjust scales, and colours and so on. Indeed, the A&A control pages are rather colour-blind friendly by allowing control options for customers to see the graphs in different ways or as multiple graphs even, at least for the "live" graphs.

This is what they look like as a PNG:-



Coverting this to SVG was not so simple - I had to make the raw SVG from the data. I managed to make a lot of common functions to the PNG generation, e.g. text placement and colours, but even that needed work as it made for inefficient SVG. To be honest, it would have been nicer to start from scratch, but I wanted to maintain the layout, at least for now.

A big change was the tx/rx rates which could now sensibly be lines, and not points. SVG makes that very simple.

I did this in all in one day, but have now spent three more days tinkering. It is like a piece of art - I am either fine tuning what you see - moving something half a pixel over, or I am fine tuning the SVG, making it more compact. It is potentially never ending, which is frustrating. I would never make a good artist a I would never finish any piece of work. I even ended up making some generic SVG path optimisation logic to make the paths shorter textually by use of relative points, the "h" and "v" functions, and even not making two segments that could be one segment in a line.

Now, to show you the SVG, but blogspot does not allow me to - WTF?
Fortunately, hosting it and editing the HTML works... So here is the same graph as an SVG:-



This is not the latest version, I will have that on LNSs later this week. I have, for example, moved it 0.5 pixels to the right as the line on the left is one pixel wide at 0 pixels so only shows half a pixel wide! I have fin tuned the size and placement of text (most of which you don't see on this example as it is customer line specific), and optimised a lot of the SVG for things like the "hour" numbers, and so on. This is from at least a day ago... So much has happened since then.

There is one more place...

We present a QR code for Two Factor Authentication TOTP set up on the FireBrick web page. This was generated as a QR code as a simple bit map and then made in to a PNG.

The logic for this already had to make it bigger than it should be, with each QR code pixel being a 4x4 image pixel as simply scaling up in the browser made it "blurry"...

So for the SVG I had to create code to "trace" around the pixels, making an outline for the black parts. Rather than simply making it lots of small squares that butt up against each other I decided to make it actually "trace" the outline of multiple black pixels. It has to trace the "inside" white parts the other way (e.g. counter clockwise), and so on. One thing I was unsure of was whether to follow a "diagonally" joined pixel as part of the same outline - it turned out to save a few bytes as it meant longer paths of "h" and "v" relative points rather than "M" absolutely points for each small part, but it was literally like a dozen characters saved. Even so, the result has no nasty "joins" between adjacent pixels which I have seem break things when scaled up.

The result looks identical to the PNG, e.g. (screenshot):-
SVG is the future of all icons on the web!

The fact we have so many high resolution devices, and scalable screens (e.g. phones) means we need to abandon use of pixel based icons on web pages pretty much. It looks "blurry" normally and just bad when scaled up.

This has turned in to quite a long and fun project, and something we really need to extend to more places, the main web site, control pages and much more.

For now the challenge over the next few weeks is simply switching the A&A control pages to show the new SVG graphs as we update the LNSs. They are already proving very popular. As we have so many LNSs now, this will take a couple of weeks, sorry.

26 comments:

  1. I don't think the new graph is as clear, to be honest - in those periodic spikes of intermittent traffic you used to be able to see upload, download and latency as separate things whereas now there's just dirty green triangles. Even though you can now zoom in (though the user-experience of browser zoom is pretty horrible) you can still no longer see the latency separate from the download.

    ReplyDelete
    Replies
    1. maybe more customization would help, like checkboxes to hide/display some of the traces

      Delete
    2. We offer that - you can already control what appears on the current day's graph, or have it on several graphs and different colours. Another point about SVG is that it would not be hard to make the "key" interactive, allowing mouse over to show only that part of the graph, or moving it to the front. Something for us to consider.

      Delete
  2. Great work on the graphs, they look very nice. Will you enable it in a way which allows toggling between both the new SVG graph and the current graphs, at least for the data items (down, up, latency, loss, etc) please? Obviously the text might as well remain the new cleaner look.

    The reason being both views have information not present in the other. The line graphs clearly show movement in a way which the dots don't. You can see that in the 1800-2100 parts of your example. This is very useful to see, it makes sense of the scatter of dots.

    But the line graphs also swamp the individual points, which you can see in the same section and more clearly in 0400-0500 for example. I've used the subtle changes in the dots many times to confirm backup traffic amongst normal traffic, that kind of thing.

    The absolute position of all the dots is lost in tighter areas, overwritten by the up and down lines connecting adjacent higher and lower dots. It would be disappointing for that useful diagnostic detail to be lost, and on balance I'd prefer the current dots (with the nicer fonts) than line graphs for line graphs sake.

    ReplyDelete
    Replies
    1. Possibly. We may ask make tx and rx semi transparent.

      Delete
  3. A link dump, but hopefully an interesting one by way of comparision...

    My barcode outline tracer implementation (PostScript) which takes care of recursing into enclosed regions to apply the winding rule:

    https://github.com/bwipp/postscriptbarcode/blob/master/src/renmatrix.ps#L91

    A small bit of non-technical intro with examples:

    https://groups.google.com/d/msg/postscriptbarcode/xGOhs7rDZ3Q/kxsLyvmTCgAJ

    You can try out the options at https://the-burtons.xyz/barcode-generator/

    Barcode: Two-dimensional symbols -> QR Code
    Contents: ABCDEF
    Options: inkspread=0.3

    Barcode: Less-used symbols -> Ultracode
    Contents: In colour
    Options: inkspreadv=0.3 backgroundcolor=AAAAAA

    ReplyDelete
    Replies
    1. Very interesting. Did not realise there was anyone else that still writes actual code in postscript. Nice. I think my wallet may be simpler - no need to invert paths or anything they just naturally come out right. I’ll have to blog on how it works some time.

      Delete
    2. Walker, not wallet, damn you autocorrect.

      Delete
  4. I'm definitely missing something here about the 2d barcodes - why don't you just render them as a bunch of black squares?

    ReplyDelete
    Replies
    1. Well, if you just make them with one pixel per "black square" and scale, the browser actually makes it all blurry and fuzzy, so you have to faff about making black squares that are 4x4 or 5x5, and even then they can be fuzzy. Using SVG eliminates that problem as you are being clear that these "pixels" are actually meant to be square with a sharp edge, and are not part of some "picture" that needs smoothing out.

      Delete
    2. I wasn't meaning why aren't they bitmaps, I was meaning "When you make an SVG, what's wrong with just emitting a bunch of rectangles?" But according to your later post it's to do with file-size, so you've answered it anyway.

      Delete
    3. Ah I, see, yes that would be massively bigger.

      Delete
    4. I think it would be smaller. At ~43x43 blocks, that can be encoded inside 232 bytes. Some simple embedded javascript to parse each bit into a black square is surely going to mean it will be done in less that the 11k of the PNG?

      Delete
    5. https://www.bgcomp.co.uk/svg/qrcode.svg works, and can be reduced to less than 1.2k, just 300 bytes bigger than PNG. It's filled with test data...

      Delete
    6. Well, for each box I would have M xx yy h 1 v -1 h -1 or some such, which is way smaller than rect blah blah, so should compress well, but actual QR codes are not even that worst case, so would combine rows and columns of pixels to make even smaller.

      Delete
    7. If you look at that source, you'll find that each byte of code, contains 4 bits (0-9A-F), so 4 squares. And there is only 1 rect call required.

      Delete
    8. Wow, sorry, I inspected the final SVG XML as presented by browser, did not realise it was a script in SVG, wow. That is "special". Not even considered client side scripting for this.

      Delete
    9. If you look at qrcode-1.svg, it's down to 1142 bytes.

      Delete
  5. I, too, find the old point-based PNG graphs much easier to extract information from (both at a glance and on detailed examination). The line-based SVG graphs are arguably "prettier", but considerably less useful IMHO -- and isn't being useful the whole point of them? If all you want is "pretty", you can have photos of sunsets.

    Please don't let your new-found obsession with making things look pretty on very high-resolution monitors blind you to loss of utility; similarly, just because you can now do "cool" things like having lines rather than points, the apparent "coolness" is not a good reason to do them: does it improve or degrade the information content? (For my money, the latter.)

    Another observation is that the new hour/rate grid lines are more visually intrusive than before (again reducing the graphs' usefulness). Plus we've lost the distinction between the 10k/100k/1M rate lines and the others, which is a shame.

    I would like an option to retain the old-style PNG graphs rather than the less useful (to me) SVG ones.

    Sadly, this looks to me like a triumph of form over function. 8-(

    ReplyDelete
    Replies
    1. The rate lines are a bug, they are meant to be dashed in the same places, and that is coming soon. I am pondering other ways to ensure they maintain the usefulness, and the pngs are still being generated and archived, so we could have a choice of old and new even. But personally I'd like to find ways to make the new ones "better" if we can.

      Delete
    2. Just want to add my +1 to having options for PNG or SVG. Personally I prefer the existing graphs - I like the five-pixel dots for Tx and Rx lines, and I prefer the existing raster font to the vector one. (I also agree with pretty much everything st42 said.)

      Delete
    3. FWIW I mostly like the SVG flavour, but I do find the spikes unnecessarily busy - although that might be an artifact of the traffic.

      I can certainly see the wisdom in at least being able to toggle lines/dots, which presumably is possible within SVG?

      Delete
  6. My only concern involves the change in graph type rather than the move from a bitmap to a vector image format. Like some of the others above, I prefer the clarity of the data points on the scatter graphs and find the line graphs to be too busy and cluttered. I do note however, that Adrian mentions a couple of possible solutions to this that could be implemented.

    As for the image format itself I much prefer the new SVGs. On my standard PC displays (2 x 24" Dells and a 23" Iiyama) they are clear, crisp and scale beautifully. On one of my monitors, the lower spec Dell, the text did initially look rather blurry (noticeably more so in Firefox than Chromium), but a small tweak to the display's sharpness sorted that out.

    ReplyDelete
  7. Adrian, I'm sorry but I'm not happy with these new graphs. With the PNG graphs I can load up Control and immediately see all the detail in the graph while sitting back in my chair. If I want to get precise levels I can Cmd+ the page to zoom the image to look at individual data points.

    Now with the SVGs the up and down lines are obscuring the details (in places) because at the default zoom level they all merge together as blocks of pixels. Even if I lean forward in the chair the detail is not there. Now I /always/ need to zoom the page, to maximum in fact, and then scroll around and study the image to find where the points must be in order to create the peaks.

    The 'at a glance' detail has gone (in places) and the same information as before now has to be gleaned as the peaks and troughs of sloped lines. I'm sure you had fun with it but for me the new look has added the noise of connecting lines and considerable friction.

    It's also annoying that while PNGs can be natively edited on the Mac in Preview, SVGs are not recognised by Preview and only work in Safari, and will require me to find a third-party image editing tool now to annotate them.

    Can you add a toggle or a graph option or something to stick with PNGs for the live and stored graphs please, so it's like nothing changed?

    Thankyou.

    ReplyDelete
    Replies
    1. There is a PNG option on the login page with the other graph options now.

      Delete
    2. Many thanks for providing that, it works perfectly. Thanks.

      Delete