Thursday, 22 June 2017

Badly written RFCs

There are many badly written RFCs, but I encountered one annoying one today

Update: Thanks for the comments, and I concede that the next page refers to "value" being quoted. Even so, the whole idea of using BNF syntax is to be unambiguous, and this is still therefor a rather badly written RFC!

So, for those less techie, "RFC" is "Request for comment" which is a sort of passive way of pushing proposals on other people when the Internet first started. The "standards" we now follow are all RFCs. There is a slightly more formal process that can promote an RFC to an actual standard.

The idea is that the RFC says how something works, especially when it is a protocol. People they try to make their systems work to the RFC.

Now, there is always a degree of ambiguity, and so there is a really good principle which has worked well for the Internet which is that you should be tolerant of what you receive and strict in what you send. Basically, if the standard says to do something you should aim to be as accurate and correct as possible in what you send. However, when someone sends something to you, and there is some flexibility in what you accept, you should try to be flexible and work out what it means.

This has allowed the slightly flawed and imperfect implementation of many standards.

Today's issue was an RFC over MIME email, specifically RFC 2387 which apparently has status of "proposed standard", and is 19 years old.

We were sending a MIME object in an email that is multipart/related with a "type" field, specifically type=text/html which says that the "root" of that object is of a type text/html. All well and good. Indeed, a type= attribute is mandatory in RFC2387.

The problem is that the email did not work properly when emailing yahoo addresses. But other email systems did work. We experimented and found the "fix" was to send type="text/html" instead.

Now, I am not happy about this! The RFC has this to say on the type attribute :-

This defines that type is specified (after a ;) has the text type, then the character = and then the type and the character / and then the subtype.

So, unless we are saying that the type is "text and the subtype is html", sending type="text/html" is simply wrong.

The problem is the examples in the RFC (and errata), such as :-

(the errata added the missing ; on the end of the first and third line)
Where the hell did those quote marks come from?

So, yahoo are not being tolerant in what they accept, they are actually expecting non standard data, and we were being strict in what we sent, but to make it work we are now being non standard.

There really is not much worse than an RFC where its own examples do not comply with the RFC!

I have submitted and errata to the RFC editors for this.

Wednesday, 21 June 2017


It turns out that the algorithms for creating a maze are quite diverse and complicated - wikipedia has much on this.

Basically, I have done it again - I found something that looks cool on thingiverse and decided to take it to the limit and do it myself. The thing was a simple labyrinth box. It was quite cool.

So I thought I would give it a try, and the issue is not really the actual 3D artwork and OpenSCAD stuff, it is the maze itself. How to make a maze that is challenging. Instead of a fixed thing that has a maze, I wanted a random maze so each one is different. OpenSCAD cannot quite manage that. So I used the source (Luke) and wrote C to make OpenSCAD.

The first thing is that this maze inherently wraps, i.e. mostly a maze is in a simple rectangle, but this is on a cylindrical shape so the X axis wraps. Easy enough for the maze generation logic.

My initial thought, and how I have made mazes before, is you have a path from origin and a point moves randomly where it can (i.e. to an empty cell). If it finds it cannot move at all it back tracks. This is a very simple algorithm to fill the whole space of a maze with no loops. The "no loops" part is pretty common as a basic principle. Maybe I'll deliberately throw in a loop some time.

My concern was that this made a maze that was going to be too simple. Or that did not "look nice", so I added a few variations.

For a start, when moving, I made a bias to continuing in a straight line rather than left or right. 25% of the time it continues, else it is random (including continuing). This makes for nice long runs in the maze.

I then added a bias for back-tracking even when not a dead end. This did make the maze more interesting but created a lot of one unit blind dead ends which are annoying and boring. You want your blind dead ends to be "interesting" and so long.

So I started again and made it that a set of all cells in the maze with somewhere to go, are picked from at random to continue the maze. This is not following the last point and backtracking, it is saying we have a set of points that can move, and picking one and moving.

This created a lot of short blind paths, hmmm... OK next trick was to add a random bias to use the last added point and continue from there, but not all of the time (just 75%). This made much longer blind runs which is what I wanted. I actually made it wander like this 100% initially until it hit a dead end or the top layer, then 75% leaving 25% of the time it picks any node at random to continue from making a fork.

Finally the trick was to then make the exit at the top the cell where it is the longest path from the start. This helped avoid accidentally having a straight line from start to finish or something that simple.

I am actually quite pleased with the result, which you can find at thingiverse. It is interesting how slight biases and choice of algorithm can massive change the nature and appearance of a maze.

P.S. You will note that there are a number of example STL files on that thingiverse entry. Obviously, as I have been tinkering, changing the artwork slightly, and changing the maze design logic slightly, I have wanted to update the thingiverse entry. But this is a set of about 10 designs as examples, so a tad tedious to load each one in to OpenSCAD and make an STL file. Thankfully OpenSCAD (even on a Mac) has a command line option, so with a simple script (and waiting several minutes) I can make a set of files automatically!

P.P.S. I did do a bit more tinkering, not just the artwork (changing from round to polygonal outer shape), but also to the maze. The one "knob" in the lid was not that good so I made the maze mirror on to the other side so allowing two knobs - this simply means as I generate the maze I add the same to the far side, but the maze can still going around the cylinder interacting with its other self even. I made it general to allow N paths. I also tweaked the bottom (end point) of the maze to be a right angle. I even scripted rendering several random boxes to make a video :-

Tuesday, 20 June 2017


So, we have a case of someone feeling they should mow down a group of muslims praying, and why? Presumably because it was muslims that did something nasty last time.

This logic is terrible, and if you follow the same logic forward, the logic used for all race and religious hatred, you must now panic at the sight of a white van man!

On my drive right now, we have two of them (carpenter and decorator), so that must be terrifying to the neighbours!

If people can see that such a comment makes no sense, how can they not see that such comments about muslims or any other race or religious group are just as daft.

Monday, 19 June 2017

Controversy of empty London buildings

After the Grenfell Tower fire there have been interesting questions raised in the news, and these are seen as Labour vs Tory, but are not so simple I am sure.

The issue, it seems, is there are many empty houses in London for various reasons, and there have been some calls to force those to be used to house people stranded after the fire.

Before I am start on my musings on this issue it is worth pointing out that this question should not have come up, surely?
  1. It should not have happened - no way a tower block should catch like this - we have building codes, and health and safety nightmare legislation for a reason.
  2. Why can the council simply not cope - it is a building managed for the council - but there has to be some fire insurance, and anyway the council must have contingencies to take care of people in an emergency, surely? The council can pay for people to be in hotels if needed. The landlord should surely have some responsibility for this?
But, rants aside, my thoughts here are really on the issue of empty houses, and the idea of commandeering them to house people in this emergency.

Owning property is a right!

The issue comes down to basic rights to own property. If you own a house, whether you are living there or not, you have rights. We have basic concept of property rights in this country. These will always be unfair by their very nature - i.e. if I own a thing (whether an empty house in London or a Mars bar) and you need it, but cannot afford to pay for it, then that is a problem. If you are starving and I have a Mars bar, do I have a moral obligation to give it to you? Probably, unless I am starving, or I was going to give it to someone else who is starving and giving it to you would deprive them. Ethics and morals are hard, aren't they. Even so, however unfair property rights are, they do exist, and we do not want to abandon them on a whim or even an emergency like this.

So what does it mean? Well, we do have laws that curtail some property rights. There are rights over monopoly control of things, where people can set any price they like as they own all of something. There are rules for landlords even when they own the properly (rules that should make them responsible for helping tenants out in cases like this I am sure). But we only curtail these basic rights in very rare cases. Is this such a rare case where new laws should apply?

Emergencies only?

One thought is some sort of emergency measures arrangement - where something like this would be some sort of special case. But that is complicated. There are around 600 homeless after the fire. So is 600 homeless people an emergency deserving a law allowing homes to be commandeered? Seems legit, until you realise that "Shelter calculates that 170,000 people are homeless in London today" so if 600 is an emergency we would have a permanent emergency and nobody would want to have an empty house in London. Indeed, anyone investing in London property would take down the house, or make it uninhabitable, so they avoid this special emergency legislation. Such a law would not actually help...

Of course, if you do allow houses to be taken over, how far do you go. Does everyone with an empty bedroom have to take on an otherwise homeless stranger? Where would you draw the line?

But some people want to help!

This is good news, there are builders that have finished flats and houses about to go on the market but offering for free as an interim measure. Apparently this is an issue, as anyone taking such a place apparently gets taken off the council housing list to find somewhere (as they have somewhere now!). This is bad for them, and the person offering the temporary accommodation.

In fact, it is hard to offer temporary accommodation without issues with sitting tenants or squatters that you cannot get rid of. Normally this is good for tenants - having strong laws to help them. But this is probably the very reason why there are lots of empty houses. If renting them out was a "safe" thing to do, people would do it and make more. Sadly, if they do, then selling their investment property is harder - they have to find someone prepared to take over the tenant or some way to get rid of them, so simpler to keep the place empty. There is, perhaps, the real problem here.

Maybe we need some type of tenancy where you have a fixed term and an absolute right to evict at the end of that time, come what may, but not a moment before. That would allow a very clear cut arrangement, a house for 3 months, but not a day more unless you are offered an extension. Safe for landlord - no messing with court orders, etc, just evict - and safe for the tenant as he knows where they stand and has security for that fixed period.

Of course, if such tenancy agreements existed, they would be used all the time, so that is not good. Maybe an arrangement like this could work where there is zero rent. People would only offer such arrangements in emergencies like this, but would not be hindered, or stuck with squatters, if they do offer generosity like this.

To be clear here, I am not saying that people will stitch up someone being kind, but they may find themselves with no choice. If you are homeless today, offered a house for 3 months, and then find the only option is being homeless again you may choose to become a squatter if that is allowed or at least prolongs your stay. If the alternative is being homeless for those 3 months instead, surely it is better to have someone prepared to offer somewhere, even on a short term arrangement like that?

Is there a solution?

The issue is people feel it really unfair that there are so many empty houses, and so many homeless (not just because of the fire). Maybe the existing legal framework is skewed just a tad too far towards the tenant? Maybe if there were some changes that could make it sensible for people with empty houses to rent them out then we would not have empty houses. It may simply be a case where the law has pushed too far in favour of one side of a commonly adversarial contractual arrangement (tenancy) to the ultimate detriment of both parties, and hardship of potential tenants.


Needless to say, I once again think the issue is not the symptom we see (empty houses) but what legal and economic framework resulted in that being a sensible thing to come about. That is what needs fixing, not the symptom. People need to "take a step back" a little more often.

Sunday, 18 June 2017

Much better motion blur using Final Cut Pro

Thanks for the feedback on the last blog post.

I agree, it looked way too ghosty, and I have finally worked out why.

If you want to do this, you need to be a bit careful with the settings, and also a bit careful with believing what people say on the various Final Cut tutorial videos.

You use Motion to create a Final Cut Title.

You then do a few simple steps:-

  • Remove the text element - we are just showing the "background"
  • In "Render", click "Motion Blur"
  • In "Projects", "Inspector", "Properties", set motion blur samples and shutter angle.
  • Save it with a suitable name for Final Cut to see it as a title effect

Some experimentation confirms what I understood. I am applying this to a sped up video. In my case 60x normal speed. So I set 60 samples and 360° shutter. What this does is for each frame of output it merges all 60 frames of original creating a smooth motion blur effect in that one frame the same as if this was actually a 60/25 (2.4s) second exposure for that one frame. This is the effect I actually wanted.

So what did I do wrong?

Firstly the various tutorial videos say the duration of the title effect does not matter, so I made a 10 second (the default) duration effect. I placed it over my (sped up) video, and stretched to fit (in this case, 3 minutes).

What I did not realise is that stretching the effect meant that it stretched the sampling. Each output frame was indeed merged from 60 input frames, but the sampling time period had been stretched 18 times, meaning that it spread over not 2.4 seconds of original (which makes one 1/25th second output frame) but over 43 seconds of original, moving 2.4 seconds forward each frame. Hence the rather odd ghosting effect.

The solution was to make an effect that was 3 minutes long and not to stretch it!

Is there another way?

One of the tutorial videos said to set the effect to loop one frame before its end. Presumably to make a 10 second effect, that when stretched, did not stretch the sampling.

It worked for the 1st 10 seconds, but unfortunately this simply meant the video looped every 10 seconds.

Maybe there is some other setting to define the way the title effect works when stretched, I may have a play.

How did it look?

Here are the three videos again.

Saturday, 17 June 2017

Time lapse and motion blur (Final Cut)

I have a slight dilemma with my new camera. It can record for about two hours because (a) the 256G card I have records about that and (b) the battery lasts about that long.

So, if I want a time lapse I have to change battery every 2 hours or so anyway, so may as well change memory card. Hence not a lot of point in recording in "time lapse" in the first place unless I can connect some mains power to the camera. There may be cases where that is sensible and the time lapse feature in the camera makes sense.

However, for now, I figured I may as well record in real time, and then make a time lapse in Final Cut Pro. It is very simple to tell it to speed up a clip.

There is a down side, the time it takes to transfer and process 256GB of data at a time, and so on. However, I think Final Cut Pro could do a better job than the simple time lapse.

The first thing that occurred to me is that the time lapse would be good if it did a proper "motion blur" on the images. So if I am doing 60x time lapse, process 60 frames to make a motion blur for each frame.

It turns out that whilst this is not a native feature of Final Cut Pro, it is a feature of the titling functions. So by buying the "Motion" system, and creating a text title with no text in it and set to do motion blur as a feature, I can overlay that on the sped up video. Yes, it is complicate to make it (and you can google for videos of how to do it), but it works.

Down side it that it takes a while, well, especially if you do 60 samples at 60x speed like I am. It takes a very long time! I need a faster machine (roll on December, Apple).

But the result is impressive.

So first the non motion blur video - which is as you might expect - still very good :-

And now the motion blur version :-

As you can see, quite a difference. It makes for a very different time lapse, and looks ghosty - see next post for the fix!

So, when the do eventually start building the actual climbing frame, etc, I should be able to make a nice time lapse.

Wednesday, 14 June 2017

I am me

I have been pondering a bit on "identity"...

This is not a simple matter, and mostly I am looking at this through the eyes of the techie. The way in which one identifies a user of a computer, for example. But it is a bit more complex than that.


One issue is that we often play many roles. In some cases the concept of an "identity" in a computer system may sensibly apply to a role that a person plays, such as "The System Administrator". Even the role in a company, such as "Managing Director". For a lot of systems there may be a role that persists beyond changes of the person taking on that role. Ideally in a computer system you want "identity" to actually relate to a person, and have a separate linkage of that person to the role they play, currently - a linkage that can change later. A lot of systems do not quite work like that.

Even knowing my identity, I have many roles I play when interacting with systems. If all systems could have the same way to actually establish my identity first and then see which roles may apply to them, that would be "neater" in many ways. At present, each system, even those that understand roles as separate from identity, have to have identity recognition which is separate to each other system.

In an ideal world we would be able to identify ourselves in some simple and consistent way to any system. We would not then need multiple logins and passwords. Such systems are an ideal which is fraught with issues though when it comes to sharing information and trusting third parties.

Multiple realms

Even so, we each may want more than one logical identity. This is not simply different roles, but we live our lives in different circles. Some times we may have almost completely separate lives. I am not talking actual multiple personality issues as a mental disorder here (though that raises some huge issues for "identity" management), but simply the multiple roles we all have in our lives.

For most of us we have "work" and "home" at least as two disconnected sets of people. We may be known by different names even (a nick name at home and family?). With social media we may have totally different lives and groups of people that know us in a different way as they are on one platform or another. Some even have multiple "handles" on social media platforms. Some of us have very different personalities in different social circles even. When you then look at "gaming" we have people that have totally different personas and roles in a context of a different group of people unconnected with their "real" lives.

The fact we can have many distinct social groups and ways in which we interact is also an issue for "identity". We may not even want those circles to be able to overlap - i.e. we do not want someone in one circle to be able to identify us in another circle.

That concept - a social compartmentalisation privacy requirement, would be a hard thing for any "real" identify requirement on computer systems.

Such separation may be even more important for people who have requirements to operate in socially distinct groups - I am thinking largely of those with sexual preferences that may still have stigma and prejudice. We see people that are homosexual finally finding themselves able to declare that to the world, but this is not the only social grouping where distinct identities may have been required in the past and may be required in the future. There will always be prejudice, and even political viewpoints may create distinct social circles which people seek to be kept separate.

I think "social compartmentalisation privacy requirement" may be a concept that the law has yet to understand or even recognise. I do not think it even comes under privacy rights at UN or EU where a right to a private family life may come in. The right to multiple independent lives/identities may not be a thing yet!

Such rights would mean ensuring individuals can maintain multiple independent "identities" within identification systems that emerge over time. Even to the extent of multiple "identity cards" if such things become a reality in the UK ever.

Of course such systems need some what to not allow someone to operate as multiple people, e.g. when it comes to voting or legal liability for crimes.

Identity cards

One thing occasionally called for is identify cards. These serve to create a record in a database, a national database typically, that associates a person in real life with that record and then associates various attributes to that record such as name, address, date of birth, and so on.

The issue is guaranteeing that the actual physical person can uniquely be linked to the record and the record can be linked to the person in a very one to one association. No chance of one person having no ID card or having more than one ID card. This is a challenge.

It would, however, distill the person to one identity, and so break that concept of any privacy between realms.

Fingerprints and the like

There are ways to identify an actual physical person, and these are typically called biometrics.

This is where you get close to a system to ensure a record in a database has a one to one link to an actual person and ties up their "identity" to that one record.

However for the concept of identity theft, where someone claims to be you, these have a big issue that you cannot revoke them or change them. Any system that can be fooled by a fake fingerprint is a problem as you cannot change your fingerprint once compromised.

Such biometrics only work if the actual sensing technology cannot be fooled, and that is not entirely the case yet.


DNA is, in effect, the holy grail of identity - as everyone has a different DNA. There are, however two big holes in this. Firstly DNA checks usually look for markers and not your entire sequence. Even so, that is probably good enough to identify a unique individual. The second issue, and big fly in the ointment is identical twins, who share the same DNA. So close, but so far. I can almost see a world where identical twins are banned (kill all but one?) or legally treated as the same person sharing the same debts and crimes.

Also, if you believe one of the episodes of CSI (and why not?), you can get people with Chimera DNA where they have two separate sets of DNA in their body. I suspect a Chimera where one set of DNA is shared with an identical twin is almost (but not quite) impossible.

Even so, for most people, a DNA profile to tie to an identity record is pretty final - you cannot escape that identity match.

When identity is not actually simple

There will be special cases, and these have been a challenge for law I believe.

One is, of course, a proper multiple personality disorder. Should each personality have their own "identity" if they can act independently?

Another is Siamese twins, where you may have two people in one body - again, do they get distinct identities.

In both cases you have a challenge if one of the two has debts or crimes, and punishment cannot only apply to one!

Law and order

Obviously, ultimately, law has to handle identity. Your identity is you, the person (baring complications as I mention above). If you committed a crime, you pay the price, not someone else. It matters not what your name is or your address or what a record says is your date of birth.

Identity is all about matching one thing to another. Matching a real person to some record of their actions, such as committing a crime (a fingerprint at the scene, etc).

I am me

You get things like PGP signing events where people sign other people's keys.

At the end of the day they are saying this "tag", or "attribute" matches some other "attribute". Ultimately this person that has in their mind the memory of the pass phrase for this secret key is associated with this "identity" such as a name or address or DOB...

I have said before in such cases, I am me. I do not need a passport to prove I am me, all that does is associate a name, DOB and passport number with me. I am me, here, now, I will enter pass phrase for this key, so you can see this key is mine (or a gullible friend's key) - take a picture if you like and link that to the key - that is one way to identify me the person in front of you...

Identity comes down to the ways we address real people - linking that addressing (name, twitter handle, whatever) to a real person. And such things are improving all of the time.

Allowing a person to legally maintain more than one identity (even if not for crime or debt or voting) is a challenge we may not have considered fully.