In Search Of The Lost Program January 11th, 2018
Programmers just can't seem to stop making new things. You only have to look at how many different unit-test frameworks and build systems there are out there to see that. We're drawn to keep reinventing software that already exists, adding little improvements and new approaches. It's like a disease sometimes, it infects us, attaches to our brains while we sleep and whispers "Code me..."
It results in this explosion of software, all of which does exactly the same thing that all the other software does, except This One's Written In Rust, or This One's Got Python Support. They're not descendants of each other, which might build on previous code, but separate creations that begin anew. And while each may add one new idea, they tend to have forgotten at least one feature that the previous ones already did.
But why? Why can't we just make, say, the perfect build system, and then everyone could just use that? Why do we make so many programming languages and libraries all doing the same thing over and over again?
The Lost Chord
I heard there was a secret chord, that David played and it pleased the Lord.
There's this lovely myth in music of the Lost Chord. A chord better than any other, considered the Holy Grail of music. And having perhaps heard it clearly in a dream, a musician might spend his whole life circling around it trying to discover the notes that make it up, hoping to recreate the perfection he once briefly tasted.
The thing about a chord, for the non-musicians out there, is it's just a thing composed of at least 3 notes played together. So any old keyboard or guitar could play it, if only you could discover all the right notes to use. It's merely a matter of composition, so surely it would be easy to just hit all the right notes and let the chord ring out? So why haven't we found the secret chord yet?
The answer, somewhat obviously, is because it doesn't exist. At least, it doesn't exist in our world, it can't. The Moody Blues once released an entire album inspired by it, 1968's "In Search Of The Lost Chord". This snippet from there perhaps gives some insight:
The Word (The Moody Blues, 1968)
Two notes of the chord, that's our poor scope
But to reach the chord is our life's hope
And to name the chord is important to some
So they give it a word, and the word is...
The daydreaming musicians recognize something here. We get two notes, not the three needed. It's not just the we haven't found it, it's that it's unfindable by us. It's a mirage that hangs just on the horizon, but vanishes as you approach it. It's a Rubik's cube where you can get one side done, but then getting the second side done messes up the first one.
Perhaps computers are the same. Are we trying to solve impossible problems? Like a cartographer trying to produce a flat 2D map of the 3D earth, some tasks just aren't possible without at least one compromise. If you try and unpeel the Earth and flatten it out, something's going to end up looking the wrong shape. The Mercator projection, perhaps the most commonly-used world map view, makes Greenland look bigger than Australia, when in fact Australia is over 3 times bigger.
Or you could try the Peirce Quincunxial Projection, which has a lower overall distortion, but on the other hand splits Antarctica into four and makes Africa go funny.
The point is, there's no correct way to solve this particular problem, it's simply not possible. The only thing you can really do is to move into a higher dimension, where you can show a full 3D globe.
Maybe the inner truth of programming is something that can't be represented by our simpler world of variables and values, of bytes and registers. Are we just endlessly trying to find new ways to fit 9 bits into an 8-bit byte? Like the story of the blind men trying to describe an elephant, we're all just groping different parts of the perfect program we dreamt of, none of us able to get the full picture at once. Are we just circling around perfection, and never able to achieve it?
It seems to me sometimes that there will never be a One True build system, unit-test framework, or programming language. Every new programming language is destined to play just two notes of the chord, never the third. Perhaps programming is an impossible problem too, one we'll only ever truly overcome once we find a way to escape our current computing flatland and finally learn to move sideways.
Anger That Burns Without January 5th, 2018
The fire has burned a hundred years. No one knows for sure where it started. In the south they say, in the greenlands. I can't remember a day when the world wasn't burning.
My earliest memory as a child was when we had to leave for the first time. My grandfather would not get on the wagon. No, he said. I will run no more, and I will not spread the evil any further. I will let the fire take me today. I remember seeing my grandmother slap him hard on the face, and he sat on the back of the wagon for the whole day, in silence. I think about him a lot these days.
It is late in the year, but the air is still warm. The distant smoke climbs high above the horizon, forming giant trails that block the sun. We have traveled farther north than I thought possible, but still the fire drives us onward. The scrubland here is arid, with bush after bush that the fire will eventually consume. We cannot stay here.
I unpack the mule and lay the packs out on the dusty ground. Uncle Jerome hacks away at a nearby acka plant and drains the water from within, catching it in a small clay bowl. It's easier now there's just the two of us. No more fighting over who gets what.
We eat a meal of sandroot; there is nothing else. I boil it slowly over a small campfire. My uncle has said nothing all day. I stare into the flames as the pan boils. They seem almost alive as they flicker and dance in the darkness.
"You shouldn't look at it," says Uncle Jerome. His round white beard glows brightly against the night. "It'll burn you."
"Don't worry," I say, "I'm keeping my distance."
"That's not what I meant."
A small lizard runs from the rocks, attracted by the warmth. My uncle drops the bowl down on top of it and traps it, then quickly reaches underneath and grabs it by its tail. He throws the lizard onto the campfire for luck. I can still remember the rhyme my grandmother taught me long ago -- a flame fed well will cast no spell, an evil unfed will surely spread. Uncle Jerome still believes in good luck.
I finish my meal and take the pan off the flames, leaving the water to cool for tomorrow.
"You'd better put that fire out before it spreads," he says as he lies down on his blanket.
I kick the dirt across the fire and stamp it out. We sit in the darkness, the only light coming from the distant glow behind us on the horizon, where the smoke has turned the sky a strange reddish. I sit looking southwards to where we have come from. The many places we had fleetingly called home are now but a distant memory, all consumed by the flames.
"Uncle Jerome..." I ask.
"How did the Great Fire start?"
The old man lifts himself up onto his elbows and lies in thought. With a great effort and a greater grunt he sits upright.
"The way I heard it," he begins, "there was a man whose heart was consumed by anger. They say a woman rejected him I think, chose someone else over him. Well that kind of thing can really get to you, you know? Well of course you're too young for all that, but you'll understand one day."
"Anyway, the anger built inside him, until it could be contained no more. Then one day he lit a match, and stared into it until it became one with him. And so he set fire to her house, and let the evil escape from within himself. Of course, with that much rage there was no stopping it, and the monster was released."
We sit in silence for a while. The wind blows with us from the southeast, carrying the smell of ash with it.
"Do you really think there's a land to the west?" I ask. "I mean that's what you said, right? Across the sea, a land where the Great Fire cannot reach?"
He strokes his beard in thought. "There must be," he says. "Old Tommeth used to tell tales about it, where I were a lad. Used to tell tales of his grandfather being a fisherman on a sailing boat, back when there were plenty."
He takes a pipe out of his pack and start filling it, poking around in the campfire's ashes to find an ember to light it with.
"Tommeth had always planned to take a boat to sea years ago. Before the fire caught up with him, of course." He stares into the night while taking a deep puff on the pipe.
"What's a boat?" I ask.
"It's... it's a kind of floating wagon I suppose," he says. "Like a big wooden duck."
He offers me a drag on his pipe, but I shake my head. I wish he'd stop smoking that thing. He's getting crazier every day.
I sit and look up at the stars. Long ago, back when there were ten of us, we traveled with a man who could read books. I had asked him about the stars. He'd told me they were lands like ours, only further away. He said they burned forever, like a fire that never goes out. Perhaps our land is becoming a star, I consider, once the Great Fire finally takes us all.
Uncle Jerome lies flat, but not asleep for I hear no snoring yet. I ask why the fire doesn't go out. Why does it keep driving us further northwest, I ask, and why won't it stop?
It does not drive us, he tells me. We bring it with us. It follows with the evil in our hearts. But we are not evil, I say. Yes we are, he replies, and tells me to go to sleep.
He is a stupid old man and knows nothing. I hear a crackle from the campfire, for some small embers still remain. I sleep. In my dreams I hear the fire calling to me. I will follow you, it says. We are one.
We reach the coast on the first day of winter. I have never seen so much water before. I want to settle here. No, says the uncle. The fire will reach here within the season. I want to drink the water but the stupid old fool won't let me.
We find an old wooden boat left abandoned, and westward we sail. The wind does all the work, and we sit and wait. The boat floats like a duck; the old man was right about that at least.
It is strange out here. The land falls far behind, and for the first time in my life I can see no glow on the horizon chasing us. We have left the fire behind. It feels quiet. My uncle will not allow me to light a fire in the boat, so we eat little. And yet I feel strangely happy, like a weight has been lifted.
On the third evening we make landfall, on a new world of wispy trees and the scent of olive. Deep lush forests of pine cover the hills, untouched by the hand of the Great Fire. We could settle here, I say to my uncle. We could stop running. Perhaps, he says.
Yet there is a fire ahead; I can feel it calling. We explore forwards and come across a village of thatch buildings, surrounding a clearing where a bonfire burns brightly.
They stare at us as we enter. A broad tall man approaches and pulls a large knife from his belt.
"That's far enough, newcomer," he says to Uncle Jerome. The accent is new and unfamiliar. He waves the knife at us, and the villagers crowd behind him.
"Have no fear," Jerome replies with his hands half-raised. "We are no enemy of you."
Don't trust them, I hear a strange voice whisper. They are evil.
A haggard old woman pulls at him from behind. "Don't trust them," the old woman says. "Look at their clothes. They are evil. They've been burned by the fire."
My uncle takes a step forward to say something, but the tall man is quick and dashes at him with the knife. I cry out but it is too late. The stranger stabs my uncle through the stomach, then again through the heart, and he falls to his knees.
I stand with my mouth open.
"Let the fire have him," the stranger says. "It needs something fresh to feast on."
They drag my uncle's still body across the ground and haul it up onto the bonfire. I follow, tugging at them to stop them, but one pushes me aside and I fall to the ground. The tall man is laughing.
"We should be thanking you," the tall man says. "Your friend will keep us all safe for weeks. Didn't your mother ever teach you how to feed a fire?"
A rage burns inside of me. The bonfire calls through the darkness, and finds me waiting. Release me, it says. Release me, and we will avenge him. I hear it so clearly.
I look up and pull a burning stick from the fire and throw it, as far as I can. It catches in the timbers of a thatch roof. The villagers scramble to put it out, but the thatch takes it swiftly. The tall man tries to start a bucket chain from the beach, but the fire is quicker. Half the village is alight now.
A warm wind blows from the east, across the sea. The new fire is spreading fiercely, and takes into the nearby forest. The oil from the trees burns well. I hope this fire lasts a hundred years.
I pull my uncle from the bonfire, but the life has already left him. I lay him on the ground and I tell him he was right, even though he cannot hear. I sit with him and I laugh, and I watch the new world burn.
Something Rotten In The Core October 24th, 2017
There's a key thought of UNIX philosophy which centers around the idea of linking programs together. You know, piping the output from
sedand then into
sort, that kind of thing. It kinda works well, I guess. For text at least.
But one of the reasons it can work OK is because you, as the end-user writing this little script or command, have full knowledge of the pieces you're building it from. You understand grep, you understand sed, and if any of those pieces suddenly stop working then you can pick the piped command-line apart again and see why. It's a system made up of pieces that you have control over, and most importantly: all the pieces are exposed to you.
This idea spread throughout UNIX, but in ways it should never have. I'm referring here to the misfortune that is debugging.
You see, on UNIX there's GDB. That's the debugger. That's the only debugger. It's very old and has had a lot of work put into it, and as a result it usually works pretty well, at least in terms of functionality. But on every other metric you measure software by, it kinda sucks.
Despite every computer made in the past 40 years having a graphical display, GDB lives in a parallel universe where the framebuffer was never invented and we all still use teletype printers. Teletypes work OK enough for getting output, but for interactive programs it just falls apart.
And so people who didn't want to deal with the pain of GDB invented the "GDB Wrapper" -- a separate piece of software that would show a nice user interface, but internally would call GDB to do the work.
We're not talking about calling out to a library here. We're talking about actually launching an instance of GDB, passing it commands, and parsing the results it prints out. And this is where we get led down a dangerous path.
APIs are hard to begin with. Good API design is very much an art, and it takes a lot of experience to come up with good ones. And the reason so many APIs are bad isn't because someone designed a bad API -- it's that they didn't even realize they were designing an API to begin with.
So much of our software world now is filled with wrappers -- programs that don't actually do the thing themselves, but 'outsource' their work to other programs. It's a stack of layers, and it's not a nice clean stack. I remember something Jeff Roberts once said to me -- the layers grind against each other, and you can feel each one chipping bits away as they collide.
I had the utter delight a few months back of trying to debug something using Qt Creator one day, except I couldn't. It just suddenly one morning refused to start debugging programs. It wouldn't say why of course, it would just sit there doing nothing.
Nothing had changed, at least so I thought. So why the failure? It turned out, after some experimentation, to be because Microsoft's symbol servers were down. That's right, a remote failure on someone else's part meant I couldn't debug locally.
Now of course errors happen in life, and are to be expected. But I didn't get an error. I didn't get a warning. I didn't get anything, except an unresponsive UI. And the reason for this, I think, is precisely because the wrapper wasn't fully aware of all the facts.
Jeff Goldblum said it best in that famous scene from Jurassic Park:
The problem with the scientific power you've used is it didn't require any discipline to attain it. You read what others had done and you took the next step. You didn't earn the knowledge yourselves, so you don't take the responsibility for it. You stood on the shoulders of geniuses to accomplish something as fast as you could, and before you knew what you had, you patented it, packages it, slapped in on a plastic lunch box, and now you want to sell it.
We've seen it hundreds of times in all kinds of software. Functions that return
boolinstead of an error code. Where did the precise error vanish to? Poof, it's gone! What used to be a useful error message became
false, and if you're lucky you'll get a generic "Unexpected error" appearing on screen. And that's if your program is using a library. If it's calling out to a command-line worker, the most likely case is it won't get checked at all and will just get printed out into a log file you'll never find, and then never seen again.
Or networking software that just sits there spinning a cursor when something went wrong. So much user-facing network software is built on top of other programs, like
rsync, and when those things fail they just don't know what to do. And so much of the problem is precisely because they're not using them as libraries, they're using them as command-line utilities. They're using these things that have ill-defined interfaces to begin with, and because it's all based on outputting text, the programmers think they can just look at examples of the output and figure out an API from that.
There's a quote from the great Douglas Adams which I'm sure I've used many times before, but it's just so incredibly apt for most of today's software:
The major difference between a thing that might go wrong and a thing that cannot possibly go wrong is that when a thing that cannot possibly go wrong goes wrong it usually turns out to be impossible to get at and repair.
You've all seen those cheap Chinese toys that look like a PlayStation, but inside its just a 6502 and 50 NES games. It's all fake, it's an illusion. It's a nice plastic finish on the outside, but if you were to open it up there's nothing in there. It's a rotten core, wrapped in layers of opaque complexity.
We're making systems that are fragile, because they're just glued on rather than bolted together. We're wrapping complex things up a wrappers that don't take the same responsibilities as the things they rely on. Like Homer's pecking bird in the Simpsons, they work just fine when everything is as expected, but when the slightest change in situation happens then everything breaks.
Written by Richard Mitton,
software engineer and travelling wizard.
Follow me on twitter: http://twitter.com/grumpygiant