Kris Jenkins' Blog

Why fsyncing a File Isn't Safe Enough

Thu, 03 Sep 2026 00:00:00 GMT

Recently I learnt something interesting about writing files to disk. Something that connects my first ever low-level database PR with the reason you can’t pull a USB disk out of your laptop without ejecting it first.

As you may know, when you write a file on a computer it doesn’t actually get written to the disk straight away. In reality it gets written into some cache in memory, with a promise that the information will be permanently stored on disk “soon”. It’s the operating system’s job to fulfil that promise. To prioritise getting your data safely stored on disk along with the dozens of other tasks it’s trying to do at the same time. Usually that’s quite fast, but it depends on a lot of factors. In practice, writing to disk “soon” may mean milliseconds or minutes.

There is a risk to that though. What if you save your file, then the machine immediately crashes? Was the file saved to disk in time, or was it still only in memory? Does a crashed operating system mean lost data?

The answer depends on fsync. Now this part I’ve known for years - if you want to write a file to disk, and actually be certain it’s on the disk, you have to fsync it. You have to tell the operating system that you want your data written, and you want to wait until it’s actually guaranteed on-disk. To do that, you write the data then call fsync. When fsync returns, the OS has completed its promise and your data is durable.

Databases use fsync all the time to make sure data is permanent. It’s also why you can’t pull out a USB disk without pressing eject first. The eject button is the user-facing version of fsync - it makes sure that everything you think you wrote to the disk is actually physically stored before you disconnect it.

But here’s the thing I didn’t know about fsync¹. If you call fsync on a file, it only makes the contents of the file permanent - not the things that file depends on. So imagine this scenario:

Create a new file.
Write some data to it.
fsync the file.
The computer crashes.

What happens? In the worst-case scenario you’ve lost your data, even though you fsync’d it! The file itself was written permanently, but the new directory entry pointing to the file was not. So the data’s on disk as promised, but there’s no way to access it. You can’t see the new file entry in the directory. A forensic specialist could track the data down, but you and I have lost the route in. And that’s the important thing I learnt: fsync doesn’t save all your work to disk. If you modified the parent directory by creating a new file, you have to fsync the parent too.

Doubtless the writers of the USB disk’s eject button knew this. I learnt it while I was looking through the code of SpacetimeDB. There’s a case where it’s building up the database’s write log, and it has to create a new segment file. The code correctly fsync’d the file, but not the parent directory. An unlucky crash could have led to data loss. Of course, it’s an easy fix - you just make the extra fsync call. I patched it and it got released this week in v2.9.0.

And that small-but-important improvement is my first truly low-level database contribution. Never have I patched so close to the hardware before. 😊

Perhaps I should have known. Perhaps when I first learnt about fsync it worked differently. Certainly there are some filesystem configurations where a single fsync is enough. But no one wants to leave these things to chance. ↩

Vibe Management

Mon, 22 Jun 2026 00:00:00 GMT

Lots of people are worried about Vibe Coding, and rightly so, but the thing that worries me far more is Vibe Management.

Vibe Management goes like this: Someone in middle management is told to come up with a strategy document for the next quarter. That sounds like an awful lot of work, so they start feeding documents into Claude. They tell themselves they’re getting AI to “do the first pass”, but the reality is Claude’s first pass comes out looking pretty good, and thinking of something better would be hard work, so they copy & paste the output into Google Docs. They change the odd sentence here and there, sure. Tell themselves they’re polishing the result. But pretty soon they call it a day. The strategy document has been created, the box has been ticked.

Has the strategy had any thought put into it? Don’t ask difficult questions! A document was asked for, a document has been written, and it’s filled with strategy words like ‘objective’ and ‘leverage’. What more do you want?

Document complete, they send it to their boss, who’s receiving similar documents from 5-50 other direct reports. Reviewing all those sounds like an awful lot of work, so they too shove it into Claude - thank God for that 1M-token context window - and Claude synthesises all those hallucinated strategies into an uber-hallucination they can pass up the chain. Claude will also write some comments on each source-document, feedback which will also be read by Claude. The digital echo chamber reaches perfection.

We now have a whole middle-management layer of strategies that have never had time, thought or effort put into them. At no point have they observed reality. At no point have they sought insight. But you can be sure of two things - there will be lots & lots of Google Docs to back the ‘strategy’ up, and the people on the front line will be expected to execute this ‘strategy’ no matter how detached from reality it may be.

We are in an age where making something that seems fine has never been easier. You want a website that seems to work? I can turn it around in an hour. You want a Q3 strategy that seems smart? I can crank that out while I get coffee. But the reason the website worries me less is, if it doesn’t work you’ll probably find out pretty soon. The feedback loop for Vibe Coding bugs is usually measured in days, and usually not that expensive. But the feedback loop for Vibe Management is measured in quarters, and three or four quarters spent pursuing the wrong strategy can sink a company. We’re going to see companies spend a year, and a department’s worth of salaries, trying to execute on a strategy that never had an ounce of thought put into it. And it’s going to do damage to vastly more people than the managers who were looking for a shortcut.

In the end, Vibe Coding and Vibe Management are symptoms of the same problem - letting AI do all your thinking for you. But for the most part, Vibe Management has a vastly larger org chart to inflict its damage on.

The AI Bubble Will And Won't Burst

Sun, 22 Mar 2026 00:00:00 GMT

I’m going to record this prediction now so that it can warm me in the winter months: The AI bubble will burst and the AI bubble will not burst.

The AI bubble will burst. I predict that because I’ve seen it in hype cycles ever since the dot-com era. Astonishing sums of money get spent building infrastructure for massively overvalued companies, and the profits just aren’t there to support it. Later, when the fickle winds of the markets change, the companies that paid for all that infrastructure collapse. But the infrastructure remains. The GPU farms exist now and they’ll still exist when the bubble bursts; they’ll just move to a clean-slate balance sheet. The financial circle of life continues. As long as you weren’t invested in those companies, it’s probably a good thing - you get to benefit from the roads someone else built and paid for.

Meanwhile, the AI bubble will not burst. Financial shenanigans aside, AI is here to stay. Who owns the models, who trained the models, which computers they run on — that’s a shell game that’s going to be played for a few years yet. But the technology itself? That’s here to stay. It’s changed things in a way that’s not going to be undone. Shut down Claude and half the people I know will just write spec files until it’s back up. Pandora’s box is open.

In short, I predict AI will sink and swim. Since the combustion engine rocked the world, the car makers changed, fortunes rose and fell, but none of it sent us back to riding horses.

The Shelf Life of an AI Study Is About Six Months

Wed, 11 Feb 2026 00:00:00 GMT

There’s a much-referenced study that says if you’re using LLMs for programming, you only think it’s making you faster. Programmers claim they’re more productive with AI tools, but when studied they actually take longer to get the job done. It’s a proper report — all the hard facts you’d need to justify a long discussion of AI hype & AI slop, setting up an essay on why the author won’t be changing their habits any time soon.

But there’s a massive problem with that study — it’s ancient. The sample data was gathered in early 2025. A year can be a long time in technology, but this particular year the AI world has moved at a staggering pace. Last winter, AI coding tools felt like very fancy autocomplete — they worked well provided your problem had already been roughly answered on StackOverflow. Today, they tackle difficult, multi-commit changes to large codebases fully autonomously, while you check your email. As I write this, I have one agent migrating an Obsidian plugin to Svelte 5 and another researching networking models from the early 2000s¹. I’ll check in on them both later, while an agent proofreads this post for me. You simply can’t talk about AI in February 2026 by referencing a study from February 2025. They’re twelve months and a lifetime apart.

Don’t get me wrong — objectively studying these models is important, valuable research. But at this stage, citing “Becker et al. says AI makes you 19% slower” isn’t a counter-argument. It’s a history essay.

Don’t ask why, but it’s Ben Gamble’s fault. ↩

Making Your Agent See What It's Missing

Fri, 16 Jan 2026 00:00:00 GMT

Coding agents like Claude or Warp are pretty good at basic web design. Nowhere near as good as a graphic designer, but honestly much better at graphic design than an average programmer. With a bit of nudging I can get results in ten minutes that are at least as good as I could manage after several hours of sweating through CSS files. But only if they have a good feedback loop.

The best tip I’ve discovered for turning them into a Minimum Viable Graphic Designer, is you want to teach them how to take screenshots. Remember, coding agents are only as good as the feedback they can gather. They become better programmers when you give them a compiler or a test suite, because they can see where they’ve gone wrong. And they become better designers when they can see what impact the changes had on the way the page renders.

So my top tip for working with AI and websites is, teach them how to take screenshots and insist they do! For this, I usually put a rule in the project along the lines of:

markdown

You can take screenshots of a webpage with `playwright screenshot --full-page $URL $OUTPUT_FILE`. For example:

```bash
playwright screenshot --full-page http://localhost:8000 screenshot.png
```

You must take a screenshot whenever you're finished making design changes. Take the screenshot and check it looks good, and matches what the user asked for.

(You’ll need to have Playwright installed of course.)

It’s a pretty simple instruction. The key is, it enables a feedback loop, and feedback loops make all the difference in the world.

Fun parting tip: You can also say, “Make it look like Apple’s website,” for example. With the ability to make screenshots, it can actually look at Apple’s website and use it as a design reference.

Configuring Neovim & Nix to use Java's LSP Server

Mon, 30 Jun 2025 00:00:00 GMT

I’ve had some problems setting up an LSP server with Neovim. I think maybe some of my problems came from path clashes with nix. Now I’ve got it working, here’s a note of the config.

Hopefully it will help some future visitor, and there’s a very good chance one of those future visitors is me. 😅

Neovim’s LSP Config

I’m using the lspconfig project. Here’s the Java-specific part of the config:

lua

lspconfig.jdtls.setup({
    cmd = {
        'nix',
        'develop',
        '--command',
        'jdtls',
    },

    init_options = {
        settings = {
            java = {
                imports = {
                    gradle = {
                        enabled = true,
                        wrapper = {
                            enabled = true,
                            -- Note the nested table here. This is really important.
                            -- `checksums` is an array of objects, which in lua translates to a table of tables.
                            checksums = {
                                {
                                    sha256 = '7d3a4ac4de1c32b59bc6a4eb8ecb8e612ccd0cf1ae1e99f66902da64df296172',
                                    allowed = true,
                                },
                            },
                        },
                    },
                },
            },
        },
    },
})

Note the cmd is running jdtls through nix. Also note that the SHA256 may need updating depending on which versions of everything you’re using. The jdtls is Eclipse’s Java language server.

Nix’s `flake.nix`

nix

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/25.05";
    utils.url = "github:numtide/flake-utils";
  };

  outputs = { self, nixpkgs, utils }:
    utils.lib.eachDefaultSystem (system:
      let
        pkgs = import nixpkgs {
          inherit system;
          overlays = [ ];
        };
      in
      {
        devShell =
          with pkgs;
          mkShell {
            buildInputs = [
              jdk21
              jdt-language-server

              (writeShellScriptBin "run" ''
                set -e
                ./gradlew bootRun
              '')
            ];

            JAVA_HOME = jdk21;
          };
      });
}

(The writeShellScriptBin part isn’t necessary for this, but I’m leaving it in because it might be an interesting bonus tip.)

Gradle’s `build.gradle.kts`

Getting the JDK versions to agree between Nix and Gradle is important:

kotlin

...

java {
	toolchain {
		languageVersion = JavaLanguageVersion.of(21)
	}
}

...

Delegated Coding Has A Bright Future

Tue, 10 Jun 2025 00:00:00 GMT

It’s a shame the term “Vibe Coding” got big, and triggered a wave of hype & counter-cynicism. It’s just muddied the water.

The way I see it, Agentic AI is going to be a lot like having a team of junior programmers. Programmers that lack deep roots but are great researchers; that work incredibly hard but lose sight of the big picture.

So the people best placed to take advantage of it aren’t those who know less about programming than a junior. It’s those who understand how to architect a cohesive whole, who are good at managing a team of juniors and great at breaking down features small enough to hand off to that team. To those who see the features that matter, break them down and delegate them, and those who see architectural problems on the horizon and delegate the work needed to avoid them. (And to those who understand why version control matters. 😁)

Vibe Coding’s going to create a lot of slop that only seems to work, but Delegated Coding’s going to standing up more quality services than we’ve ever seen before. It’s an exciting time.

Tustain's Law

Sun, 18 May 2025 00:00:00 GMT

Recently I’ve been thinking about something I affectionately call Tustain’s Law. It’s something a friend of mine was fond of saying: “If you make something twice as easy, it becomes ten times more useful.”

You can argue the numbers of course, but the insight is clear - making something easier to use doesn’t just mean your users will have a nicer time; often it means that ten people are trying to engage with what you’ve built, and nine of them will give up in frustration unless you’re actively designing with them in mind.

Ten times more useful reminds me of the first pair of Bluetooth headphones I owned. They always had connection problems. It was rare they’d connect first time, and because they were unreliable I didn’t get much value out of them. The headphones I own today are functionally identical objects - similar sound quality, similar battery life - but because they always work they slip into my unconscious and have become a permanent part of my life. I use them for at least 40 hours a week and I don’t leave the house without them.

Tustain’s Law means user-friendliness isn’t just a nice-to-have. It’s not a luxury to be tacked on if there’s time at the end of the sprint. It’s actually fundamental to how valuable what you make becomes. It changes the way people engage with technology, moving it from “if I really have no choice” to “it’s my go-to tool”. In many cases, “10x more useful” would be a conservative estimate - it’s probably closer to infinitely more useful. The difference between using it or not.

Paul Tustain, the man behind Tustain’s Law, was killed last week in a tragic motorbike accident. He was a brilliant man, easily the smartest CEO I’ve ever known, and a good friend who profoundly changed the course of my life.

Maybe in the coming days I’ll face writing up a proper obituary or something, but for now this is all I can manage. And it’s a pretty good start - if you’re following me you’re probably into technology, and knowing Tustain’s Law might help you make what you’re working on ten times more useful to people. That might help you change the world, or at least your corner of it. And if it does, then Paul has had a profound effect on your life too…

Paul Tustain, 1962-2025. Rest In Peace my friend.

Why We Podcast

Tue, 11 Apr 2023 00:00:00 GMT

I’ve just come to the end of my year-and-a-bit stint as the host of the Streaming Audio podcast, so I thought I’d take a moment to reflect on podcasting. What’s in it for the listener, the guest, the host and the company?

Over the past year, I got to record conversations with over 70 guests about their patch of the technology landscape, and what a privilege it was. There was a lot of work behind the scenes to make it happen. There were many, many hours of pacing back and forth muttering, “Yes, but what’s the story here?” And there was far more calendar juggling than I’ll ever be comfortable with. But every single time I hit the record button it became a joy. Talking with my kind of people, listening to them, and getting a glimpse into their minds and their problems. Learning.

So whether you’re a potential host, a future guest, or a company thinking of starting a podcast, here are my thoughts on what it takes, why we do it, and why it turned out to be one of the most valuable parts of my career so far.

Let’s Start With The Business Case

I was invited on the MongoDB podcast last year for International Podcast Day, and one of the questions that came up was, “Why would your company want to run a podcast?” Sure, it’s nice to have a chat and all, but is there a compelling reason for your company to pay you to put all those hours in? I don’t think the answer to that had quite crystallised in my mind back then, but it has now. In hindsight, podcasting was probably the most effective piece of Developer Advocacy I did, because it goes to the heart of what I think Developer Relations is all about.

A Tech Company Must Talk To Developers (Not At Them)

This is true of any company whose users are technical, but absolutely fundamental to a company that makes tools for developers. You can’t just build it and throw it over the wall. You have to be talking, teaching, learning and listening to the developers you hope to serve. You need a healthy feedback loop to create a product that’s always improving and a community that’s always growing.

To my mind, Developer Advocacy is simply making that conversation an official priority. Just as you have a department that prioritises finance or infrastructure, you have people dedicated to building and caring for the ongoing conversation with developers.

But how?

We know the default answers. Forums and Slack channels are great if you can build up enough momentum to be somewhat self-supporting. Conference talks and workshops are perfect for high-quality interactions, but you have to travel far and wide if you want to reach all the different enclaves. Blog posts are great, broad-reach content, but even the best writers I know would struggle to write more than a couple of non-trivial posts¹ a month.

Podcasts are another way to get involved in that developer conversation, and they combine some of the best properties of the usual formats. Like blog posts, they’ve got global reach and a long tail. Like forums, they can help build and strengthen a sense of community, especially as notable figures in that community appear as guests and add their voices to the conversation. And like conferences, podcasts are extremely high-quality interactions, having a personal, meaningful conversation, and recording it in a format where everyone can benefit. Done right, they have all the depth and value of a one-on-one conversation, reshaped into a broadcast that thousands of people can get value from. Human soul, with company-pleasing metrics.

Okay, that’s the must-pay-the-bills angle, so on to…

What’s In It For The Listener?

As a host, you’re a proxy for the audience, so to me that meant going into every conversation determined to learn something from the guest. What’s interesting is how many different shapes that education could take. Sometimes it meant asking technical questions about which configuration parameters mattered. Sometimes business questions about what this new feature buy me, and what will it cost me? And sometimes it was personal questions that brought everything together. There was a great episode with Rashmi Prabhu where we were talking about the topic of managing banks of clusters of servers. It’s a potentially dry topic, but halfway in I realised that this was 100% her jam. The real story was how much she enjoyed solving these kinds of problems, and once you see the problem through her eyes it blossomed into a source of fascinating software puzzles. That, perhaps, is the fundamental thing you can learn from a podcast with 52 different voices a year - 52 perspectives from which the world is fascinating. 52 windows we can peer through and learn what they’ve learned. Somewhere in there, there’s 52 miles’ worth of walking in someone else’s shoes.

And with all those shoes walking through our office, we should probably ask…

What’s In It For The Guest?

The initial motivation overlaps with why people speak at conferences - it’s that same urge to share knowledge. To share what you’ve learnt, to make the road easier for the people that come after you, and to maybe pick up some new friends along the way. It also never harms your career to establish yourself as an expert² No one knows how much your voice matters until you use it. Not even you.

There are a couple of less obvious benefits too. One is that it’s a lot easier to be a podcast guest than a conference speaker. Preparing a conference talk involves a huge amount of writing, rehearing, travel and nerves. Chatting to someone about your work is so easy you’re probably already doing it regularly without thinking about it. Talks can be hard, but conversation is something humans are hard-wired to do. A good host sets you up to look good without even trying. 😎

Another benefit—and one that truly surprised me—came from talking to Dom Fioravanti about the company he was helping to grow. A couple of months after the episode broadcast he told me it had been a huge help in hiring and in getting investment. Telling your story in such a casual, organic way can help investors to get a handle on a technical subject, and can add valuable context and tone when you’re talking to prospective employees. Conversations transmit in a way that slide decks can’t.

And of course, while I have no evidence for this, I hope a few authors sold a few more copies of their book, and a few builders of valuable services get a few new customers to value them. That kind of thing never harms, does it?

Okay, that’s everybody else covered, so I get to end by talking a little about me. 😉

What’s It Like Behind The Mic?

To a large degree, if there are great reasons for the audience and for the company, that’s the case closed. Greenlight that podcast and away we go. But as the host, you have to have more motivation than that. If you want to do a good job, and do it every week of the year, you have to know where the joy is and what the workload will be.

The biggest thing I got from it was the chance to learn from people. I’m not just a proxy for the audience - I’m the audience too! And I got a front-row seat to learn from some of the best. I got to pick the guests who sounded most interesting, and then direct the conversation to whatever details I most wanted to learn. It was always with the audience in mind, but I got to let my personal curiosity drive.

I also met some fascinating people along the way, from an author of technical books for children to the hosts of a podcast in Hebrew; from a sporting world I never thought I’d explore to a whole panel of experts contemplating the future of how we architect information systems. I got to travel to people, places and ideas, and take an audience with me for the ride. (I also got to chat with friends and call it work. It was work, but that didn’t stop it from being a pleasure too.)

The hardest part of it for me was always off-camera. Every podcast was preceded by a phase of assembling a scratchpad’s worth of notes, figuring out how all those clues could fit together to form a cohesive story, and then preparing enough that we could talk through that story without it feeling scripted - having a roadmap without running on rails. I always wanted it to feel like an organic conversation, without it becoming a directionless ramble. That takes some prep.

And the time after the conversation could be just as much work. How do you take an hour’s worth of conversation and boil it down into a 2-minute intro? What did we actually end up talking about? Which parts mattered most? How do you explain why it matters to an audience that might be causally dropping in for the opening minutes? I learnt a heck of a lot about storytelling and structure just from the relentless need to meet a weekly cadence.

The upside to that work was considerable. I got to pick the brains of the best and brightest. I got the chance to follow my instincts and say, “They interest me, so I’ll bet they’ll interest everyone else.” And this one may sound strange, but I got the honour of sitting in people’s cars and kitchens, on their trips to the gym and their dog walks, every week. I’m a bit of an audiophile. Not the “gold-plated hifi cables” kind, but I do get a great deal of pleasure from music, audio and the spoken word. The fact that people selected me as a sound they wanted to hear regularly brings me a great deal of pride, and joy. If you were a listener, thank you. ❤️

You can get a long way on quick tips, listicles, and more prosaic versions of documentation, but writing with real depth takes time. ↩
Some of your guests won’t consider themselves “experts”. Don’t let their modesty fool you. Some guests have literally written the book on a topic, while others have the battle scars from living with a tool in production. Both kinds of expertise are invaluable. ↩

Setting Up A kcat Config

Tue, 11 Oct 2022 00:00:00 GMT

If you’re using Kafka, kcat is an essential tool to have in your toolbox. As the name suggests, it’s cat for Kafka. It gives you fast, command-line access to simple tasks. Like, say you want to tail a topic called orders:

kcat -Ct orders

(-C is ‘consume’. -t sets the topic name. This is unix, so you can collapse those flags into -Ct.)

Easy huh? Well, not quite. Because there’s usually more to type than that. For a local Kafka instance you need to set the broker:

kcat -Ct orders -b localhost:9092

That’s a pain. And if you want to access a remote machine it gets much worse:

kcat -Ct orders -b localhost:9092 -X security.protocol=sasl_ssl -X sasl.mechanism=PLAIN ...ugh...

But you don’t have to do that. You can put it all in a kcat config file. It’s a little bit of a tricky one to get right though, so for your reference and mine, here’s how you set that up once and for all.

Example kcat Config Files

The file you need to edit is ~/.config/kcat.conf.

For a local instance, it should contain something like:

plaintext

metadata.broker.list=localhost:9092

For a remote server, you’ll need to include the authentication details, so your kcat.conf will look more like:

plaintext

metadata.broker.list=<Bootstrap Server>
security.protocol=sasl_ssl
sasl.mechanism=PLAIN
sasl.username=<API Key>
sasl.password=<API Secret>

With a kcat.conf correctly set up, all the connection information gets picked up every time you call kcat, so that first, simpler command kcat -Ct orders should Just Work.

Supporting Schema Registry

Update: After I wrote this up I found myself wishing I could do the same trick to connect to a Schema Registry. I’d like to be able decode some Avro values by simply saying:

kcat -Ct orders -s value=avro

Again, that’s not too hard. You “just” need to cram the location of the schema registry, and your authentication details, in the right format under the right key. Here’s the magic incantation to add to your ~/.config/kcat.conf:

plaintext

schema.registry.url=https://<SR API Key>:<SR API Key>@<SR URL>

With that, kcat will attempt to fetch schemas from the registry whenever it needs a codec.

MicroServices - Check Size Before Ordering

Mon, 09 Jul 2018 00:00:00 GMT

In recent years I’ve seen several startups go for a microservices-first approach. This is my two cents, distilled from watching those projects unfold.

If you’re Netflix or Amazon, one of your big problems will be coordinating different teams and projects. Sooner or later, you’ll hit a situation where you need to coordinate the release dates of five different projects to get something major launched.

On that day, you’re hosed.

One sub-project will always slip, so five teams sit near-idle for a month, burning time and money. Fast forward a year or two and you wake up to six-month drop-dead release cycle with a month-five code-freeze and a month-four crunch window. It’s horrible. And it never improves, because coordinating groups of people is hard. [Citation: Theresa May, 2016-2018]

If you’re in that situation, microservices are brilliant. They allow you to turn a human-coordination problem into a system-coordination problem. System coordination is still hard - supporting multiple versions, dealing with concurrency issues, building distributed, fault-tolerant systems - these are all properly, Computer Science hard. But they’re tractable. Computer concurrency problems are a pain, but you can solve them. People concurrency problems are the nightmare scenario second only to herd-of-cats concurrency problems.

Microservices turn people coordination problems into software coordination problems. Microservices are the way out of a quagmire.

But what if you don’t have those problems? What if you only have twenty-odd developers, and you can coordinate just by talking to them? What if your project coordination problems boil down to, “asking Andy and Catherine if they’re ready?” Then by choosing microservices you’ve created API versioning problems, created concurrency problems and created distributed, fault-tolerant problems, all for no benefit.

In the absence of the relevant people problems, microservices create software problems. Microservices are the way into a quagmire.

Update (1/4/22) Lately I’ve been exposed to a different style of building microservices which I’m still thinking through. The first approach - the one I spoke about in this post - looks a lot like a zoomed-out version of Object Orientation, in which you build systems that are independent, stateful islands talking to each other through messages.

The second approach to microservices looks a lot like a zoomed-out version of Functional Programming, which you build systems that look like streams of immutable data, with independent transformations on those streams. I’m still thinking through the implications, and it deserves a its own new blog post, but for now I’ve put some of my thinking into this talk: Side Effects Are The Complexity Iceberg.

Emacs Feature Branch Tip

Sat, 29 Jul 2017 00:00:00 GMT

Like many of us, I like to work on feature branches. Branch, commit-commit-commit, push, pull-request. It’s clean and easy.

But sometimes I forget a step and screw my workflow up. I get to the end of a round of commits and realise I forgot to git branch before I started coding. Now all my work is on local master.

That’s fixable, but it’s a pain. You’ve probably been there, and there are two solutions:

Create a new branch, stash, reset master to origin/master, checkout the new branch and pop the stash.
Or git push origin master:feature, and hope that origin/master doesn’t move forward before the feature gets merged.

The former’s a pain and the latter relies on luck. Neither is ideal.

Step forward Magit, the premium Emacs git client. magit-branch-spinoff automatically puts you back where you want to be. It checks out a new branch based on your current HEAD, and resets master back to origin/master¹. So, so easy.

Unsurprisingly, it’s available from a M-x near you, or from Magit’s branch menu.

You won’t ben surprised to learn it’s a bit more flexible than that, if you need it to be. See the docs. ↩

There's Immutable And Then There's Immutable

Wed, 25 Jan 2017 00:00:00 GMT

Following a brainwave, a scientist writes something important on a blackboard. Then in large, friendly letters he writes above it, “PLEASE DO NOT ERASE THIS!”
He leaves the room to grab a coffee.
Five minutes later he returns, and another scientist is cheerfully cleaning the blackboard, removing all trace of that world-changing idea.
“B-b-but - didn’t you see the note???” yelps the first.
“The one that said, ‘don’t rub this out?’” replies the second.
“Yes!”
“Oh…I thought it meant don’t get rid of the blackboard. I didn’t realise the contents were important too.”

So it may be with immutability. Take Scala for instance. Put an object into a scala.collection.immutable.List and the list is guaranteed never to change. It will always be a one-element list with that object in it. But the object itself? No special guarantees there - it’s still mutable as it was before.

In this case, the list is the wall and the object is the blackboard - one is firmly nailed to the other. But just like the blackboard, the object can change its meaning at will.

Similarly, in recent JavaScript specs we have Object.freeze() which many will say makes a JavaScript object immutable. And technically it does - the object’s keys will not change, and the pointers¹ associated with those keys will not change, but the things they point to might. JavaScript’s Object.freeze() only freezes top-level properties. In which case it might be better named Object.frosting(). Again, the blackboard is fixed, but what’s written on it is a movable feast.

All this seems odd coming from languages that have immutability as a core concept, like Clojure & Haskell. In those languages, if I give you access to my immutable list, you can do anything you like with it and it won’t affect me. I don’t need to know what you did; I don’t need to trust you to behave yourself; I can rely on guarantees from the language itself. If you try to write on my blackboard, you’ll find yourself transported to another room, with an identical blackboard that you’re free to do whatever you like with. Mine remains untouched.

And so sadly it seems that, like “isomorphic”², “immutable” now has two meanings depending on where you first heard about it. In the new meaning, you need to know some implementation details before you can say what part of the data is fixed, and what is subject to change. That doesn’t sound very immutable to me…

(I confess I struggle to see the value of immutable frosting. It seems like, “immutability made easy for the language implementor,” not, “made valuable for the programmer.”)

Still, for the record, here’s a definition of immutability that I think’s unambiguous and useful:

If you call a pure function with the same immutable argument, you always get the same result.³

Sounds obvious, but up until a few days ago I thought that was the only meaningful meaning ‘immutable’ could have…

I’m still bothered by JavaScript being a language where there are no C-style pointers, but pointer-based semantics are everywhere. ↩
Meaning, “you can convert a value from one type to another and back without losing information,” or, “the same HTML can be rendered on the backend or the frontend.” ↩
You really can cross the same river twice. ↩

Type Bombs in Elm

Tue, 08 Nov 2016 00:00:00 GMT

If you want to make anyone on the core Elm team roll their eyes, just pick a feature that Haskell has and say, “Elm should have it too.” ‘Typeclasses’ is the most common way to play the game, but you’ll get extra hipster points if you say ‘Higher-Kinded Types’.

So, I apologise in advance for this post, because it’s a bit, “Haskell has X, Elm should have X too.” And I’ll try to justify why. But really this post is about how Elm already has this $BLESSED_FEATURE, if you squint your eyes…

Get to the point Kris - what’s the feature?

Type holes. They’re brilliant. I use them. I should use them more. At CodeMesh last week I saw University Professors popping them like Pez. But crucially for Elm, they are huge-mongously useful for beginners. They make typed functional programming easier at every level because they’re all about getting fast, precise help when you don’t know what to do next.

Let’s have a quick look at what they are, for the uninitiated, and then I’ll outline a filthy little trick that gives you type holes in Elm today.

The Hole Truth

If you’ve done any programming in Elm (or Haskell, or PureScript, or …) you’ll know that the compiler always has an opinion on what the type of a function should be. If you write this function in Elm:

elm

tally xs =
    List.foldl
        (\x ( trues, falses ) ->
            if x then
                ( trues + 1, falses )
            else
                ( trues, falses + 1 )
        )
        ( 0, 0 )
        xs

…the compiler knows what this function’s type is, even if you don’t. And with the --warn flag, it will happily tell you:

markdown

Top-level value `tally` does not have a type annotation.

67| tally xs =
^^^^^
I inferred the type annotation so you can copy it into your code:

tally : List Bool -> ( number, number' )

Maybe you could have figured that type out yourself. But why bother? The compiler knows about types, and is supposed to be your assistant. It’s happy to help.

Type holes use the same mechanism at a more granular level. If the compiler can assist you in figuring out the type of a function, why can’t it help you figure out the type of an argument? Why can’t we say:

elm

tally xs =
    List.foldl
        (\x ( trues, falses ) ->
            if x then
                ( trues + 1, falses )
            else
                ( trues, falses + 1 )
        )
        HEY_COMPILER_WHAT_GOES_HERE_MY_FRIEND?
        xs

…and the compiler would say:

markdown

Thank you for asking. I looked at the way the rest of the function
uses that argument, and I think it should have the type:

        `( number, number' )`

Or somesuch thing. Tally-ho.

And you’d say, “Oh yeah, a pair of numbers - that makes sense.” And your ho would be well and truly tallied.

Any time you get stuck on what type of argument you need, you can stick in a marker to the compiler - a type hole - and it gives you its best guess. It tells you what you’re looking for.

(Actually, as far as I’ve seen, the experts don’t even wait to get stuck. They have more holes than Trump’s tax return.)

Type holes are simple, useful and universal. They save time & head-scratching. They help.

One More Example

Before we get to the Elm-today part, there’s one other place we can use holes: in the type signature. We’ve already seen that Elm can tell you the type of the tally function. But what if we were certain about part of the signature? What if we were sure we wanted it to return a pair of numbers, but weren’t sure what the input type should be? Type holes to the rescue:

elm

tally : HEY_COMPILER_WHAT_SAY_YOU_MY_FINE_FELLOW? -> ( number, number' )

markdown

Thank you for asking. Of course, I could fill in the whole type
signature, but I appreciate you doing some of the work. The missing
piece looks like a:

    `List Bool`

...to me. Making the whole signature:

tally : List Bool -> ( number, number' )

Now, why don't you stick that in and make a cup of tea with the time
saved, what?

Having type holes available in the signature makes sense - it means that whether you’re stuck on the implementation or the design, you can get some advice.

Half A Hole Is As Good As A Whole One

So that’s type holes. And Elm doesn’t have them. (Yet.) But the point of this post is not to argue for their inclusion¹. No the point of this post is, I use type holes in Elm already and I want to tell you how. It’s a dirty, dirty trick, but it works and only God can judge me.

Here’s the secret: To get type holes in Elm, pick a primitive type that you know is wrong, and stick that in instead:

elm

tally xs =
    List.foldl
        (\x ( trues, falses ) ->
            if x then
                ( trues + 1, falses )
            else
                ( trues, falses + 1 )
        )
        "LOLWAT"
        xs

Elm will react with suitable horror:

markdown

...
74| )
75|> "LOLWAT"
76| xs

Function `foldl` is expecting the 2nd argument to be:

    ( number, number' )

But it is:

    String

There - I stuck in a String knowing it was wrong. Elm borks and tells me the discrepancy. It says, there’s a conflict between ( number, number' ) and String. Since I know that String is wrong, I now know that ( number, number' ) is right. I have fooled Elm into giving up the goods.

Let’s quickly do the same at the type level. I’m unsure of the type of the first argument, but I’m certain it’s not Date, so we’ll use that:

elm

tally : Date -> ( number, number' )
tally xs =
    ...

Elm complains, but in doing so we’ve tricked it into giving us the right answer:

markdown

-- TYPE MISMATCH ---------------------------------------- ./src/Contact/Rest.elm

The type annotation for `tally` does not match its definition.

67| tally : Date -> ( number, number' )
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The type annotation is saying:

    Date -> ( number, number' )

But I am inferring that the definition has this type:

    List Bool -> ( number, number' )

Again there’s a disagreement - Date isn’t right so List Bool must be.

So that’s it. Stick in a known-wrong answer and it will complain with the right one.

In the absence of a proper name I’m calling this trick a Type Bomb, because if you don’t have a hole, a controlled explosion will do. Go out there and make some shapes…

Actually I can’t finish this note without recounting a conversation I had with @jessitron last week:

plaintext

Me: You know what feature Elm should have?
Jessica: Type holes.
Me: Type hol-. Wait, what did you say?
Jessica: Type holes.
Me: Um, yeah. That.

So she totally stole my thunder, but at least great minds think alike.

I’m quite certain Evan knows what a type hole is, just as he knows what typeclasses and higher-kinded types are, and he’s either going to put them in or he’s not. I shan’t be telling him what to do. (That said, if Evan happens to be reading, I will say that type holes help beginners as much as experts. More perhaps.) ↩

Functional Mumbo Jumbo - ADTs

Fri, 10 Jun 2016 00:00:00 GMT

What are “ADTs”?

Algebraic Data Types.

Lovely. What the heck does that mean?

It means you can mix product and sum types together.

Okay. I have one more question, and please start answering in English: What are product and sum types?

Product types you’ll be deeply familiar with - it’s when you make a new type by combining others. For example, you could define a Person type as the combination of a String for their name, and an Int for their age. Like this:

elm

type Person
    = Person String Int

…or like this:

elm

type Person
    = Person
        { name : String
        , age : Int
        }

In C that’d be a struct. In Java you’d probably call it a POJO. In Clojure they say, “Just use a map.”

Why’s it called a ‘product’ type?

It might help to think of how many possible values there are for Person now.

It’s all the possible values for String, multiplied by all the possible values for Int. It’s the multiple, or product, of the two types.

Okay, product types combine types together. What’s a sum type?

Well, product types are a bit like an AND operation, aren’t they? They AND types together. If you think of it that way, you’re instantly going to start to wonder what an OR would look like. For that, we can pull out another old chestnut from the Big Book of Type Examples. Chapter two, playing cards:

elm

type Suit
  = Heart
  | Spade
  | Club
  | Diamond

A suit is a heart, or a spade, or a club, or a diamond. Any one of those four values, but only one. It’s an exclusive-or, as a type.

So a sum type is just an enum?

Well…yeah I suppose it is. But that’s not what makes this interesting. An ADT isn’t just structs and enums, it’s the free mixing of them to make sophisticated data models.

Give me a concrete example.

Glad you asked. I’m going to make a credit card payment. When the answer comes back from my payment provider, there are lots of things that could happen. Hopefully it goes well and I get a receipt ID back as a string. But maybe the card on file has expired, and I get a few details about the card to prompt the user with. Or it just blows up with different kinds of error. I can model all possible responses neatly with an ADT, like so:

elm

type PaymentResponse
    = Paid String
    | CardExpired Int Date
    | HttpError Int String
    | AuthError

Here we have Paid, which is a wrapper around a String; CardExpired and HttpError which are product types; and AuthError which is a “traditional” enum value. All mixed together in a sum type to model all possible payment responses.

Most languages don’t have support for this. They might have enums, but not ones that carry data. You can’t have an enum of structs or an enum of POJOs¹.

Clojure’s an interesting case where it does have ADTs, but they’re possibly too flexible. You can mix together maps, lists and primitives to make any shape of data you like. But you can suffer with the exclusion problem. While you can say all these things are valid responses:

clojure

{:paid "eefa9112"}

{:card-expired [8783 #date "03/19"]}

{:http-error [500 "Server just exploded"]}

:auth-error

…you have a harder time saying what isn’t a valid response, like:

clojure

{:card-expired "we don't know why"}

That’s where Plumatic schema and spec become essential.

So is that it? Is that all an ADT is?

No. You’re at the entrance to a delightful rabbit hole. ADTs can have type variables, and can be further generalised, and all sorts of useful stuff. But if you have the core idea - mixing and matching ANDs and ORs to build up data models - you’re on your way.

So why is it useful?

Because data-modelling is the most important thing we do, so tools for modelling data are fundamentally interesting. But let me give you a concrete example of something cool to go away with. Once we’ve defined our rich data type, a suitably-savvy compiler will be able to help us use it properly. Here’s some pseudo-code to handle payment responses:

elm

handle paymentResponse =
    case paymentResponse of
        Paid id ->
            insert db "receipts" db id

        CardExpired fourDigits date ->
            notifyUser ("Your card " ++ fourDigits ++ " expired on " ++ date ++ ".")

        HttpError code msg ->
            logError code msg

Did you spot the error in that code? You probably did, but the compiler is guaranteed to notice:

-- MISSING PATTERNS ---------------------------------------------- src/Main.elm

This `case` does not have branches for all possibilities.

...

You need to account for the following values:

    PaymentResponse.AuthError

Add a branch to cover this pattern!

The compiler knows which values PaymentResponse can have, and so can check our case statements for completeness. In this case, it’s instantly noticed that we’ve ignored the authorization failure.

Sweet. Okay, last question. If there’s AND and OR…where’s NOT?

You mean a type that can be any type except something?

Yeah.

Interesting question. Yup, that’d still be part of an Algebra of Data Types, so it’d count. I’ve never seen it though. Let me know if you do…

And nearly every language has a way to get around it. In Java you’d use reflection on subclasses, perhaps. But first-class support? No. ↩

How Elm Slays a UI Antipattern

Fri, 03 Jun 2016 00:00:00 GMT

The Problem

There’s a bug in several places in the OSX Twitter client, and in at least one place in the Slack client, and in loads of other apps. Here’s Twitter:

That’s not the correct data. When I go to this tweet, it shows “0 Retweets 0 Likes” for a few moments, until the real data loads in.

Now here’s Slack:

Again, it’s not correct. when I go to the view it says, “no messages,” as a placeholder until the actual data loads in. A few moments later I get a big long list.

It’s one of those UX patterns that crops up everywhere once you start noticing it: showing zero or empty when you should be showing a loading message. It’s incredibly common.

Now I don’t have access to their source code, of course, but I’m willing to bet that in most cases, it boils down to this broken data model:

javascript

var data = {
  loading: true,
  things: [],
};

Store a loading flag and an empty list/count of things you’re loading. Then in the UI, you’re supposed to check the loading flag before you display the things.

Inevitably someone forgets, and so on a slow connection the UI immediately displays the, “No Things Found,” message until the response comes back.

We’ve all done it. Especially when deadlines are tight and people start planning whole sprints around, “the happy path.” But it’s poor UX, and it’s a shame to spoil a quality experience on such a small detail.

Now, how can we fix this? Discipline? Linters? More unit tests? A QA army? No! Let’s fix the data model, because fixing the data model is fastest, cheapest, most reliable fix there is.

To Elm!

Let’s move over to Elm, because we’ll need a language with explicit data models if we’re going to talk about this properly. So here’s an absolutely basic version of this data model, and it’s a wrong ’un:

elm

type alias Model =
    { things : List Thing
    }

If you think about it, this broken from the outset. It’s saying that you will always have a list of things available, which isn’t true when we startup. And that causes the display bug.

The Solution

A much better model would be:

elm

type alias Model =
    { things : Maybe (List Thing)
    }

Now we can represent, “we haven’t loaded things yet,” naturally. On startup the list of things is Nothing. When the load completes it becomes Just [thing1, thing2, ...].

If the server replies that there genuinely were no things, then that’s Just []. Now the empty list actually means the empty list, it’s no longer doing double-duty as the “not loaded” case.

You can do this in JavaScript of course, but you won’t. Long experience will have taught you that setting a property to null may be correct, but it’s just asking for runtime exceptions. Hence the separate loading flag, which gets forgotten.

In Elm, you can not only handle the nothing case safely, but the compiler will actively nag you to remember to write the “Loading” message:

elm

module View exposing (..)

...

someView model =
  case model.things of
    Just things -> ...

…and when we compile:

-- MISSING PATTERNS ---------------------------------------------- src/View.elm

This `case` does not have branches for all possibilities.

...

You need to account for the following values:

    Maybe.Nothing

Add a branch to cover this pattern!

We get an error message about missing out the loading case, exactly where the solution needs to go.

On the surface it seems like we’ve made a simple data-model change, but by reshaping the data it becomes much easier to ensure an improved user experience. Win.

The Leveling-Up

I like this, but I reckon we can be much more sophisticated. And Elm will still make it easy when we do. In my mental model, REST requests have one of four states:

We haven’t asked yet.
We’ve asked, but we haven’t got a response yet.
We got a response, but it was an error.
We got a response, and it was the data we wanted.

In a language with nice data-modelling facilities, we can easily create something that represents these four states:

elm

type RemoteData e a
    = NotAsked
    | Loading
    | Failure e
    | Success a

This is like a Result or Either type, where we can have parameters for the Failure and Success types, plus tracking for the NotAsked and Loading states.

Now in our data model, we can properly represent the list of things as something we fetch from an external source:

elm

  type alias Model =
      { things : RemoteData Http.Error (List Thing)
      }

Let’s make a quick refinement to that - most of our external requests are REST requests, so the error type is nearly always an Http.Error. A quick alias tidies that up:

elm

  type alias WebData a =
      RemoteData Http.Error a


  type alias Model =
      { things : WebData (List Thing)
      }

The nice thing about this data model in Elm is, the compiler will now force you to write the correct UI code. It will keep track of the possibility of “things not loaded” and errors, and force you to handle them all in the UI.

Personally, I love it. A bit of thought about the data model, a couple of utterly-reusable definitions, and for the rest of the project the compiler will be at our backs, pushing us towards better UX. Happier users at bargain prices.

The Download

If you want to play with this, I’ve published it as a standalone package which you can add to your project with:

$ elm package install krisajenkins/remotedata

Here are the docs, and happy modelling!

Clojure-Fu with Juxt

Wed, 01 Jun 2016 00:00:00 GMT

I love juxt¹. But I was reminded yesterday that not everyone has made friends with it. So, to give it a little of the limelight it deserves, I thought I’d write down My Favourite Juxt Trick.

The Recap

First, let’s recap what juxt does: it lets you pass the same arguments to several functions in a row. So if you want to find out a person’s name & height, you can do this:

clojure

(def name-and-height
  (juxt :name :height))

name-and-height is now a single function that returns both results as in a vector:

clojure

(def person {:name "Kris"
             :height 172
             :devastating-handsomeness-factor 2})

(println
 (name-and-height person))

plaintext

;=> [Kris 172]

Nothing special there. It’s just calling each function, :name and :height, with the same arguments, and gathering up the results.

The Trick

Now let’s do something cool with it. I’ve got a list of people working in different departments:

clojure

(def people [{:name "Brad"     :salary 27000 :department "hats"}
             {:name "Janet"    :salary 54000 :department "capes"}
             {:name "Eddie"    :salary 19500 :department "hats"}
             {:name "Frank"    :salary 98000 :department "capes"}
             {:name "Rocky"    :salary 18000 :department "shoes"}])

I’d like to roll those up by department and largesse. First let’s define what counts as highly-paid, using a number that’s entirely fictional in London:

clojure

  (defn highly-paid? [x]
    (< 30000 (:salary x)))

Now we have two functions, and we can group by both of them:

clojure

(pprint
 (group-by (juxt :department highly-paid?)
           people))

plaintext

{["hats" false]
 [{:name "Brad", :salary 27000, :department "hats"}
  {:name "Eddie", :salary 19500, :department "hats"}],
 ["capes" true]
 [{:name "Janet", :salary 54000, :department "capes"}
  {:name "Frank", :salary 98000, :department "capes"}],
 ["shoes" false] [{:name "Rocky", :salary 18000, :department "shoes"}]}

There - we’ve just built an on-the-fly, composite-key index of our data, for free.

Any time you need to filter some data by multiple criteria, juxt comes in mighty handy.

Sorry Juxt. I love you too, but this one is about your namesake. ↩

How I Structure Elm Apps

Sat, 16 Apr 2016 00:00:00 GMT

Updated: 1st Aug 2016 for Elm 0.17.

With the rise of people getting to grips with Elm, I’m hear several folks wondering how to make the transition from play project to a larger app. So, in the hope that it helps ease that transition, here’s my guide to structuring large Elm apps. Follow along and you should have an easier time.

Startup

First, there are two packages that begin every Elm app, so let’s install them:

elm package install --yes elm-lang/core
elm package install --yes elm-lang/html

Then I’ll edit the elm-package.json file that’s just been created and set source-directories:

json

    "source-directories": [
        "src",
        "test"
    ]

With that in place, we can talk about file structure. There’s one special file, and a then a pattern that repeats throughout the app.

`src/App.elm`

The special file is the app’s entry point. I typically call it src/App.elm, though some choose src/Main.elm. If you were building several different Elm apps from the same code base, I might name the entry points more specifically, like src/Public.elm and src/Admin.elm. But for now, let’s just talk about a single entry point.

App contains as little as possible to get the app started, which is usually just:

elm

module App exposing (main)

import Html.App
import State
import View


app : Program Never
app =
    Html.App.program
        { init = State.initialState
        , update = State.update
        , subscriptions = State.subscriptions
        , view = View.rootView
        }

This code is pretty much lifted from the docs. The only difference is this is /all/ just wiring. The real code has been moved out into a repeating pattern of four files.

The Featureful Four

Every feature, subfeature, page, widget and gizmo will have its own directory, and in that directory there will almost certainly be these four files:

`.../feature/Types.elm`

Types houses the type definitions for this feature. At a minimum, it will contain Model and Msg:

elm

type alias Model = {...}

type Msg = ...

…plus any other types that are unique to this feature.

It will also house generic functions on those types. Not application logic, but library functions to make working with those types easier.

Types will typically export everything, and use anything from subfeatures’ Types, so the top of the file will look like:

elm

module Feature.Types exposing (..)

import Widget.Types
import Grommit.Types

`.../feature/State.elm`

This is the brain of each feature. Shaped by Html.App, this will contain a minimum of three declarations, The init, update and subscriptions functions. First, init:

elm

init : ( Model, Cmd Msg )
init = ...

This is what must be set up to start using this feature. Any parent that makes use of this feature agrees to pull this in when it starts using it.

It’s perfectly fine for init to be a function instead of a value. For example, you might have a reservation widget that expires after 15 minutes, so needs to know its startup time:

elm

init : Time -> ( Model, Cmd Msg )
init startTime = ...

With the starting state in place, next we need the update function:

elm

update : Msg -> Model -> ( Model, Cmd Msg )
update action model = ...

This is the function that reacts to events, triggers external effects and steps the app forward in time. If you’re changing or debugging application logic, you’ll start working here.

Last we need subscriptions, which may be as simple as:

elm

subscriptions : Model -> Sub Msg
subscriptions _ = ...

Or may actually do its own work:

elm

subscriptions : Model -> Sub Msg
subscriptions model =
  Websocket.listen "ws://echo.websocket.org" Echo

Or even mix in the subscriptions of children:

elm

subscriptions : Model -> Sub Msg
subscriptions model =
    Sub.batch
        [ Websocket.listen "ws://echo.websocket.org" Echo
        , Child.subscriptions model.child
            |> Sub.map ChildMsg
        ]

State will usually just export these three key components, and will expect them of its sub-components:

elm

module Feature.State exposing (init,update,subscriptions)

import Feature.Types exposing (..)
import Widget.Types
import Widget.State
import Grommit.Types
import Grommit.State
...

`.../feature/View.elm`

View contains rendering code. Things with the type signature ... -> Html Msg, and rarely anything else. I name the entry point root:

elm

root : Model -> Html Msg
root model = ..

View will usually only export root, and will use subfeatures’ Types model & View.root function:

elm

module Feature.View exposing (root)

import Html.App as Html
import Feature.Types exposing (..)
import Widget.Types
import Widget.View
import Grommit.Types
import Grommit.View

...
grommitList model =
    Grommit.View.root model.grommits
        |> Html.map GrommitMsg
...

`.../feature/Rest.elm`

Last is RESTful code (HTTP and JSON-handling), which I name Rest. Not every feature needs such code, of course, but it’s so common that it’s worth describing.

The Rest namespace contains HTTP calls, and JSON encoders/decoders, and little else.

Rest will usually export everything, and may use subfeatures’ Types & Rest modules:

elm

module Feature.Rest exposing (..)

import Feature.Types exposing (..)
import Widget.Types
import Widget.Rest
import Grommit.Types
import Grommit.Rest

Wrap Up

Along with the top-level App.elm, that pattern-of-four describes the vast majority of my directories in all of my Elm projects. As a layout it works well. The pattern makes everything easier to find and modify, as it answers a lot of useful questions without thinking:

Q. Where’s the code for registration?
A. Look for a directory called Registration.
Q. Where’s the code for feature X?
A. Look for a directory called X.
Q. X doesn’t look right.
A. Look in X/View.elm.
Q. X doesn’t behave right.
A. Look in X/State.elm.
Q. X needs a new feature.
A. Start by changing the data-model in X/Types.elm, and then follow the compiler messages through.
Q. Our API is changing.
A. Start by changing X/Rest.elm, and then follow the compiler messages through.

There’s very little surprise and a good default organization. The really nice thing about it is that it works recursively. You can use this layout in every feature and sub-feature and sub-sub-features.

src
├── App.elm
├── Types.elm
├── State.elm
├── View.elm
├── FrontPage
│   ├── Rest.elm
│   ├── State.elm
│   ├── Types.elm
│   └── View.elm
├── Login
│   ├── Rest.elm
│   ├── State.elm
│   ├── Types.elm
│   └── View.elm
└── Registration
    ├── Rest.elm
    ├── State.elm
    ├── Types.elm
    └── View.elm

Caveats

Take this pattern as a default, and remember to think for yourself. In my codebases you’ll find features that are so tiny that I occasionally stick the whole thing in one file. (Not often, but sometimes.) And you’ll certainly files other than these four, for special use-cases. But this simple approach covers a lot of ground, neatly.

Git for Static Sites

Tue, 09 Feb 2016 00:00:00 GMT

I often find myself creating static websites, where I want to push the source to Github, and publish the result somewhere, quickly and cheaply.

One example is presentations, which I write in org-mode and then publish using reveal.js. Another is Elm sites, when I build the site in Elm and need to host the generated HTML/JavaScript, without necessarily wanting a custom backend.

So, here’s how I do it. Follow along and you’ll learn a practical trick for publishing, and some git-fu to boot…

Publish on Github

The first thing to know is that those generous souls at Github will publish your static sites for free. If you create any repo with a branch called gh-pages, they’ll publish it that branch on <your-username>.github.io/<repo-name> a few minutes later.

So you could just create a brand new repository, commit your compiled-down site to it, and rename master to gh-pages.

That would do the job, but we can improve things.

Orphan Branches

Keeping two separate-but-related repos, one for the source and one for the output, gets old quickly. Especially for small projects. Ideally I’d like two separate project trees managed in the same .git directory, to keep everything together. And there’s something that feels correct about managing the two facets of the project in one place. Doing exactly that is very easy:

git checkout --orphan gh-pages

When you checkout an ‘orphan’ branch, git creates an entirely new branch without any parent commits (hence the name, ‘orphan’). You can add anything you want to that branch, and it’s separate from the main history, but stored in the same .git directory.

After creating an orphan branch you can clear out the current directory - safe in the knowledge that it’s all still saved on master - and add any new content you want. The orphan branch has its own timeline and own files, but it’s housed within the same git repo.

Run git push -u origin gh-pages the first time you want to push, and from then on git push will send master, publishing your source, and gh-pages, publishing your site.

We just need one more piece to make this work really smoothly.

Worktrees

The only problem with the orphan branch approach is that the workflow typically goes:

Checkout master.
Build the site to /dist.
Checkout gh-pages
Copy the contents of /dist up to the top level.
Commit and push.

It’s fine, but the file-copying is a little unwieldy. The solution, that makes this whole approach flow smoothly, is using git worktrees - a new feature in git 2.5.

With worktrees, you can have two completely separate working directories, backed by the same git repo. It’s a feature I use when I want to flip between active-branches without stashing all the time. (For example, when you’re blocked halfway through feature A, so you want to crack on with feature B, but you know you’ll need to flip back and forth.)

For our purposes though, worktrees allow us to build directly to our gh-pages branch. We start a worktree off like so:

$ pwd
/.../my-project
$ git worktree add ../my-project-dist gh-pages

Now we have two directories side-by-side:

. - /my-project       # Checked out as master
|
. - /my-project-dist  # Checked out as gh-pages

Now we just alter our build script to build into ../my-project-dist and the setup’s complete. Any commits in either directory get stored on the same repo. And pushes, pulls or fetches will synchronise exactly as you’d hope.

To Recap

Why?

You want to manage a source and a release repo.
With easy publishing of the release repo.
With the minimum of admin.
With a guarantee that if you’ve got one, you’ve got the other.
With a guarantee that if you’ve backed-up one, you’ve backed-up the other.

How

# Create the orphan gh-pages branch
$ git checkout --orphan gh-pages

# Create at least one commit.
...

# Connect local gh-pages to the remote version.
$ git push -u origin gh-pages

# Head back to master.
$ git checkout master

# Check out the orphan gh-pages branch in its own directory.
$ git worktree add ../my-project-dist gh-pages


# Finally, from now on, to publish both the source and the release it's just.
$ git push

What Is Functional Programming?

Tue, 29 Dec 2015 00:00:00 GMT

This is my take on what functional programming really is, in a way that will make sense to a jobbing programmer just trying to Get Stuff Done.

I put it to you that every function you write has two sets of inputs and two sets of outputs.

Two? Only one, surely?

No, two. Definitely two. Let’s take a look at the first pair with this example:

java

public int square(int x) {
    return x * x;
}

// NOTE: The language doesn't matter, but I've picked one with
// explicit input & output types, for emphasis.

Here, the input you’re used to thinking about is int x, and the output you’re used to is also an int.

That’s the first set of inputs & outputs. The traditional set, if you will. Now let’s see an example of the second set of inputs and outputs:

public void processNext() {
    Message message = InboxQueue.popMessage();

    if (message != null) {
        process(message);
    }
}

According to the syntax, this function takes no inputs and returns no output, and yet it’s obviously depending on something, and it’s obviously doing something. The fact is, it has a hidden set of inputs and outputs. The hidden input is the state of the InboxQueue before the popMessage() call, and the hidden outputs are whatever process causes, plus the state of InboxQueue after we’re done.

Make no mistake - the state of InboxQueue is a genuine input of this function. The behaviour of processNext cannot be known without knowing that value. And it’s a genuine output too - the result of calling processNext cannot be fully understood without considering the new state of InboxQueue.

So the second piece of code has hidden inputs and outputs. It requires things, and causes things, but you could never guess what just by looking at the API.

These hidden inputs and outputs have an official name: “side-effects”. There are many kinds of side-effects, but they all come together under the same concept: “when we call this function, what does it need that isn’t in the argument list, and what does it do that isn’t part of the return value?”

(Actually I think we need two terms: “side-effects” for the hidden outputs, and “side-causes” for the hidden inputs. For most of the rest of this post I’ll use “side-effects” for brevity, but I’m definitely talking about side-causes too. I’m talking about all hidden inputs and outputs.)

Side-Effects are the Complexity Iceberg

When functions have side-effects (and side-causes), you can look at a function like this:

java

public boolean processMessage(Channel channel) {...}

…and think you’ve got an idea of what it’s doing, and be totally wrong. There’s no way to know what it requires or what it will do without looking inside. Does it take a message off the channel and process it? Probably. Does it close your channel if some condition is true? Maybe. Does it update a count in the database somewhere? Perhaps. Does it explode if it can’t find the logging directory path it was expecting? It might.

Side-effects are the complexity iceberg. You look at the function signature, and the name, and think you’ve got a sense of what you’re looking at. But hidden beneath the surface of the function signature could be absolutely anything. Any hidden requirement, any hidden change. Without looking at the implementation, you’ve no way of knowing what’s really involved. Beneath the surface of the API is a potentially vast block of extra complexity. To grasp it, you’ll only have three options: dive down into the function definition, bring the complexity to the surface, or ignore it and hope for the best. And in the end, ignoring it is usually a titanic mistake.

Isn’t This What Encapsulation’s About?

No.

Encapsulation is about hiding implementation details. About hiding the innards of the code so the caller doesn’t need to worry about them. That remains a good principle, but it’s not what we’re talking about.

Side-effects aren’t about “hiding implementation details” - they’re about hiding the code’s relationship with the outside world. A function with side-causes has undocumented assumptions about what external factors it’s depending on. A function with side-effects has undocumented assumptions about what external factors it’s going to change.

Are Side-Effects Bad?

When they work exactly as the original programmer expected, no, they’re probably fine. But there’s the rub: we have to trust that the the hidden expectations of the original programmer were correct, and will remain correct as time marches on.

Have we set up the state of the world the way this function expected when it was written? Or did the world get changed somewhere? Perhaps because a seemingly-unconnected piece of code changed. Or because we’re installing the software in a new environment. Hidden assumptions about the state of the world mean we have hidden hopes that it’s similar enough to work.

Can we test this code? Not in isolation. Unlike a circuit board, we can’t just plug into its inputs and check its outputs. We have to break open the code, figure out its hidden causes and effects, and simulate the world it’s supposed to exist in. I’ve seen several TDD’ers spin in circles about whether they should do black box or white box testing. The answer is, you ought to do black box testing - you ought to be able to ignore the implementation details - but if you allow side-effects, you can’t. Side-effects close the door to black box testing, because you can’t get to the inputs & outputs without cracking the box open and learning what’s inside.

This effect is amplified for debugging. If a function doesn’t allow side-effects (or side-causes), you can understand whether it’s correct just by giving it some inputs and checking the outputs. But a function with side-effects? There’s no upper-limit to how many other parts of the system you’ll have to consider. When it’s allowed to depend on anything, and cause anything, then the bugs could be anywhere.

We Can Always Surface Side-Effects

Can we do anything about this complexity? Yes. It’s actually pretty simple to get started: If a function has something as an input, just say so. If it returns something as an output, declare it. Simple as that.

Let’s try an example. Here’s a function with a hidden input. Bonus points if you spot it quickly:

java

public Program getCurrentProgram(TVGuide guide, int channel) {
  Schedule schedule = guide.getSchedule(channel);

  Program current = schedule.programAt(new Date());

  return current;
}

This function has a hidden input of the current time (new Date()). We can surface this complexity by just being honest about this extra input:

java

public Program getProgramAt(TVGuide guide, int channel, Date when) {
  Schedule schedule = guide.getSchedule(channel);

  Program program = schedule.programAt(when);

  return program;
}

This function now has no hidden inputs (or outputs).

Let’s look at the pros and cons of this new version:

Cons

It looks more complex. It has three arguments instead of two.

Pros

It isn’t more complex. Hiding a dependency didn’t make it simpler, being honest about it doesn’t make it more complex.

It’s vastly easier to test. Testing different times of day, clock changes, leap years, will all be straightforward, because we can pass in any time we like. I’ve seen code like the first version in production, with all sorts of clever tricks to spoof the current system clock for testing’s sake. Imagine the effort, when we can just make it a parameter!

It’s easier to reason about: This function now just describes a relationship between its inputs and its outputs. If you know the inputs, you know what the result should be, and you know everything about the result. This is a big deal. We can verify this code in isolation. As long as we’ve tested the relationship between inputs and outputs, we’ve tested the whole of the function.

(And as an aside, it’s also more useful. We get, “what program starts in an hour?” code for free.)

What is a ‘Pure Function’?

Drumroll please.

Now, finally, with an awareness of hidden inputs & outputs, we can give, “a jobbing programmer’s definition of pure functions”:

A function is called ‘pure’ if all its inputs are declared as inputs - none of them are hidden - and likewise all its outputs are declared as outputs.

In contrast, if it has hidden inputs or outputs, it’s ‘impure’, and the contract we think the function offers is only half the story. The iceberg of complexity looms. We can never use impure code “in isolation”. We can never test it in isolation. It always depends on other things which we have to keep track of whenever we want to test or debug.

What is ‘Functional Programming’?

With an awareness of pure and impure functions, we can now give, “a jobbing programmer’s definition of functional programming”:

Functional programming is about writing pure functions, about removing hidden inputs and outputs as far as we can, so that as much of our code as possible just describes a relationship between inputs and outputs.

We accept that some side-effects are inevitable - most programs are run for what they do rather than what they return, but within our program we will exercise tight control. We will eliminate side-effects (and side-causes) wherever we can, and tightly control them whenever we can’t.

Or put another way: Let’s not hide what a piece of code needs, nor what results it will yield. If a piece of code needs something to run correctly, let it say so. If it does something useful, let it declare it as an output. When we do this, our code will be clearer. Complexity will come to the surface, where we can break it down and deal with it.

What is a ‘Functional Programming Language’?

Every language supports pure functions - it’s hard to make add(x, y) impure¹. And in many cases converting an impure function to a pure one is just a case of lifting all its inputs and outputs into the function signature, so that the signature totally describes its behaviour. So are all programming languages ‘functional’?

No. Because then the term would be meaningless.

So what can we give as a “jobbing programmer’s definition of a functional programming language”?

A functional programming language is one that supports and encourages programming without side-effects.

Or more specifically: A functional language actively helps you eliminate side-effects wherever possible, and tightly control them wherever it’s not.

Or more dramatically: A functional language is actively hostile to side-effects. Side-effects are complexity and complexity is bugs and bugs are the devil. A functional language will help you be hostile to side-effects too. Together you will beat them into submission.

Is That It?

Yes. There are a couple of subtleties - things you probably never thought of as a hidden input before, but that’s the essence. But start building software with the perspective of “side-effects are the first enemy” and it will change everything you know about programming. Join me for part two, in which we take an awareness of side-effects, and functional programming, and fire a scattergun over the programming landscape.

Acknowledgments

This post comes out of a couple of discussions about the nature of functional programming. Particularly a chat with Sleepyfox discussing whether JavaScript could be considered a functional programming language, with the right libraries. My answer was an instinctive no, but thinking through /why/ lead me along a very fruitful chain of thought.

Hat tip to James Henderson, which whom I have bounced around many fruitful functional ideas this year.

And thanks to Malcolm Sparks, Ivan Uemlianin, Joel Clermont, Katy Moe and my homophonic-doppleganger Chris Jenkins for proofreading & suggestions.

Although Java tries really hard. ↩

Which Programming Languages Are Functional?

Tue, 29 Dec 2015 00:00:00 GMT

In part one of this post, I defined functional programming not from an academic perspective, or a marketing one, but in a way that will make sense to a jobbing programmer. More importantly, I hope, I defined what side-effects are in a way that makes it easy for a jobbing programmer to spot them before they spiral out of control.

Now, let’s take a look at functional programming languages in the real world…

Pot Shots Across the Programming Landscape

Armed with the ability to spot side-effects, we can now look at a given function and spot hidden complexity. And armed with some real-world definitions of FP, we can now survey the programming world and fire some new insights in practically every direction…

Functional Programming is not…

It’s not `map` and `reduce`

Even though you’ll see these functions in every functional language, it’s not what makes the language functional. It’s just something that crops up nearly every time you try to take the side-effects out of processing sequences of things.

It’s not lambda functions

Again, you’ll probably see first-class functions in every FP language. But it’s something that naturally emerges when you start building a language that avoids side-effects. It’s an enabler, but not the root cause.

It’s not types

Static type checking is a very useful tool, but it’s not a pre-requisite for FP. Lisp is the oldest functional programming language, and the oldest dynamic language.

Static types can be very useful though. Haskell uses its type system beautifully in the attack on side-effects. But they aren’t the ingredient that makes or breaks a functional language.

Say it long, say it loud, functional programming is about side-effects.

(And side-causes, of course).

What Does This Mean For Languages?

JavaScript is not a Functional Programming Language

Functional languages help you eliminate side-effects where you can, and control them where you can’t. JavaScript doesn’t meet this criteria. In fact, it’s easy to spot places where JavaScript actively encourages side-effects.

The easiest target is this. The hidden input that’s in every function. What’s particularly magical about this is how freely its meaning changes. Even expert JavaScript programmers have trouble keeping track of what this currently refers to. From a functional point of view, the fact that it’s magically available at all is a design smell.

While you can certainly load FP libraries (Immutable.js, for instance) into JavaScript, and that makes programming in a functional style easier, it doesn’t change the nature of the language itself.

(By the way, if you like the functional libraries that are gaining popularity in JavaScript land, imagine how much you’d like a whole language that supported the functional style.)

Java is not a Functional Programming Language

Java is most definitely not a functional language. The addition of lambdas in Java 1.8 does nothing to change that. Java stands in stark opposition to functional programming. It’s core design principal says that code should be organised as a series of localised side-effects - methods that depend on and change the local state of an object.

In fact, Java is hostile to functional programming. If you write Java code that has no side-effects, that doesn’t read or change the local object’s state, you’ll be called a Bad Programmer. That’s not how Java is written. Your side-effect free code will be peppered with static keywords, and they’ll frown and drive you out of town.

I’m not saying Java is wrong. (Well, okay, I am.) But the point is it takes a completely different view of side-effects. Java thinks that localised side-effects are cornerstone of good code; FP thinks they’re the devil.

You could look at that from a slightly different angle. You could see both Java and FP as responses to the problem of side-effects. Both models recognise side-effects as a problem, and respond differently. OO’s answer is, “contain them within boundaries called ‘objects’,” whereas FP’s answer is, “eliminate them”. Unfortunately, in practice Java doesn’t just try to encapsulate side-effects; it mandates them. If you’re not creating side-effects, in the form of stateful objects, then you’re a Bad Java Programmer. People actually get fired for writing static too often.

Scala Has a Big Task on its Hands

Seen in this light, Scala is a very challenging proposition. If its goal is to unify the two worlds of OO and FP, then through the lens of side-effects we see it as trying to bridge the gap between “Side-Effects Mandatory” and “Side-Effects Forbidden¹”. They’re such opposite views that I’m not sure they can be reconciled. You certainly can’t unify the two just by making objects support the map function. You’ll need to go deeper and reconcile the conflict between the two opposing stances on side-effects.

I’ll leave you to judge if Scala succeeds in such a reconciliation. But if I were in charge of Scala’s marketing I’d sell it as a gradual move away from the side-effecting world of Java, to the pure world of FP. Instead of unifying them, it could be a bridge away. Indeed, many people see it that way in practice.

Clojure

Clojure takes an interesting stance on side-effects. Its creator, Rich Hickey, has said that Clojure is about “80% functional”. I think I can clarify why that is. From the outset, Clojure was designed to deal with one specific kind of side-effect: Time.

To illustrate this, here’s a Java joke for you:

#+BEGIN_QUOTE

What’s 5 plus 2?
Correct. What’s 5 plus 3?
8
Nope. It’s 10, because we turned 5 into 7, remember? #+END_QUOTE

Okay, it’s not a great joke. But the point is, in Javaland, values don’t stay still. We might legitimately take something that represents a five, call a function, and find that it’s no longer a five. Mathematics says that five never changes - we can call a function that gives us a new value, but we can never affect the nature of five itself. Java says values change all the time, and as long as they’re wrapped in object boundaries that’s okay.

The integer case may seem trivial, but the effect is amplified when we look at larger values. Remember InboxQueue from part 1? The state of InboxQueue is a value that changes over time. We can say that Time is a side-cause to the meaning of InboxQueue.

Clojure focuses ferociously on the side-cause of Time. Rich Hickey’s insight² is that the hidden effect of time means we can’t rely on values to stay put; and if we can’t rely on that, we can’t rely on our functions’ inputs, and so we can’t rely on anything to behave predictably or repeatably. If even values have side-effects, then everything has side-effects. If values aren’t pure, nothing in our programs can be pure.

So Clojure takes a sword to time. All its values are immutable (unchanging over time) by default. If you need a changing value, Clojure provides wrappers around unchanging values, and those wrappers are subject to heavy constraints:

You must opt-in to changing (mutable) values by using a wrapper.
You cannot accidentally create a mutable value. You must always use guards in the language explicitly flag potential side-effects.
You cannot unwittingly consume a mutable-value. You must always use guards in the language explicitly acknowledge the risk of side-effects.
When you open a mutable-value wrapper, the thing you get back is immutable again. You can easily get back out of the time-dependent world and back into the pure one.

With respect to time, Clojure is a great example of a functional programming language. The language is deeply hostile to the side-effect of time. It eliminates it wherever it can, by default, and where you feel you must have the side-effect, it helps you tightly control it so it doesn’t spill into the rest of your program.

Haskell

If Clojure is hostile to time, Haskell is just plain hostile. Haskell really hates side-effects, and puts a large amount of effort into controlling them.

One of the interesting ways Haskell fights side-effects is with types. It pushes all side-effects up into the type-system. For example, imagine you’ve got a getPerson function. In Haskell it might look like:

haskell

getPerson :: UUID -> Database Person

You can read that as, “takes a UUID and returns a Person in the context of a Database”. This is interesting - you can look at the type signature of a Haskell function and know for certain which side-effects are involved. And which aren’t. You can also make guarantees like, “this function won’t access the filesystem, because it’s not declared that kind of side-effect.” Tight control³.

Equally important, you can look at a function like:

haskell

formatName :: Person -> String

…and know that this just takes a Person and returns a String. Nothing else, because if there were side-effects you’d see them locked into the type signature.

But perhaps most interesting of all, is this example:

haskell

formatName :: Person -> Database String

The signature tells us that this version of formatName involves database-related side-effects. What the hell? Why does formatName need the database? You mean I’m going to need to set-up and mock-out a database just to test a name-formatter? That’s really weird.

Just by looking at this function signature, I can see something’s wrong with the design. I don’t need to look at the code, I can smell a rat just from the overview. That’s magic.

Let’s just briefly compare that with the Java signature:

java

public String formatName(Person person) {..}

Which Haskell-version is that equivalent to? Without seeing the body of the function, you have no way of knowing. It may be the pure version, it may access the database. Or it may delete the filesystem and return, "screw you boss!". The type signature tells you very little about what’s going on, or what the surface area of the function is.

Haskell’s type signatures, in contrast, tell you a great deal about the design. And because they’re checked by the compiler, they tell you things you know to be true. And that means they make great architectural tools. They surface design-smells at a very high level, And they also surface patterns of coding too. I’m going to keep the words ‘functor’ and ‘monad’ out of this post, but I will say that high-level software patterns begin with high-level analysis, and high-level analysis is made much easier when you have a high-level notation⁴.

Perl

Perl deserves a mention here in any discussion of side-effects. It has a magic argument, $_, which means something like, “the return value of the previous call⁵.” It gets used and/or changed by many of the core library functions, implicitly. As far as I know this gives Perl the distinction of being the only language where one global side-effect is considered a core feature.

Python

Let’s take a quick look at a fundamental side-effecting pattern in Java:

java

public String getName() {
  return this.name;
}

How would we purify this call? Well, this is the hidden input, so all we have to do is lift it up to an argument:

java

public String getName(Person this) {
  return this.name;
}

Now getName is a pure function. It’s noteworthy that Python adopts this second pattern by default. In python, all object methods take this as the first argument, except by convention they call it self:

python

def getName(self):
    self.name

Explicit is better than implicit indeed.

Mocking

Mocking frameworks usually do two things.

The first is they help you set up value objects to act as inputs. The harder your language makes it to set up complex values, the more useful you’ll find this. But that’s an aside.

The second is more interesting in this discussion - they help you set up the right side-causes to the function under test, and track that the right side-effects have occurred after the test.

Seen through the lens of side-effects, mocks are a flag that your code is impure, and in the functional programmer’s eye, proof that something is wrong. Instead of downloading a library to help us check the iceberg is intact, we should be sailing around it.

A hardcore TDD/Java guy once asked me how you do mocking in Clojure. The answer is, we usually don’t. We usually see it as a sign we need to refactor our code.

Design Smells (or The Scent Of Nothingness)

If there were an I-Spy book of side-effects, the two easiest targets to spot would be functions that take no arguments, and functions that return no value.

No Arguments Signal Side-Causes

Whenever you see a function with no arguments, one of two things are true: Either it always returns exactly the same value, or it’s getting its inputs from elsewhere (ie. it has side-causes).

For example, this function must always, always return the same integer (or it has side-causes):

java

public Int foo() {}

No Return Value Signals Side-Effects

And whenever you see a function with no return value, then either it has side-effects, or there was no point calling it:

java

public void foo(...) {...}

According to that function signature, there is absolutely no reason to call this function. It doesn’t give you anything. The only reason to call it is for the magical side-effects it promises it will silently cause.

Summary / Conclusion-y Thing

A real, intuitive awareness of side-effects will change the way you look at coding. It will change everything from how you look at individual functions, right up to overall systems-architecture. It will change the way you look at programming languages, tools and techniques. It changes everything. Go kill a side-effect today…

Yes, I’m conflating OO and Java. In the context of Scala I think it’s fair to equate the two. ↩
One of them! ↩
PureScript takes this idea further, and is worth looking into. ↩
I’ve had some great Clojure design discussions where we’ve used Haskell signatures to explain ourselves, verify the consistency of the design, and to summarise our conclusions. Yes, Clojure discussions. Haskell’s notation has a value that extends far beyond the language. ↩
See man perlvar for the exact definition. ↩

Quote of the Week - Doug Crockford

Fri, 30 Oct 2015 00:00:00 GMT

I’m now thinking I’ve got two completely different types of objects: those containing only data, which are things I share with JSON; and things containing only methods, which are things I can share with applications.

Doug Crockford - The Better Parts

I think Doug Crockford’s great, but it does seem like he’s getting to Functional Programming by the absolute slowest route possible.

Geocoding With Elm and ArcGIS, Part 3

Wed, 07 Oct 2015 00:00:00 GMT

After part 1 and part 2, I thought I was finished with this feature, but Ariarooo Efe rightly comments:

[in] the online demo of the app, if there are no results the user isn’t told anything.

This feedback is trivial to add, so let’s just see what quick work Elm makes of it.

The Model

Recapping on the model of the app:

elm

type alias Model =
  {...
  ,places : Maybe (Result Error (List Place))
  ...}

Part 1 explains how places models the data we’re loading, and makes a clear, type-checked distinction between these cases:

We haven’t loaded any data yet.
We have loaded data, but that failed.
We’ve loaded data and it’s available.

What it also gives us, which in many frontend frameworks we might not get, is a very clear distinction of, “we loaded some data and it was empty”. If places is an empty list, there’s no confusion - it doesn’t mean “not asked” or “there was an error”, it only means there are zero results.

Armed with a correct data model, we’re going to find the rest easy…

The View

In the case of “no results”, the value of places will be Just (Ok []). That is:

Just ... - we’ve got some data.
Ok ... - we got it successfully.
[] - it was the empty list.

We can update our rootView to add an extra case in:

elm

rootView : Address Action -> Maybe LatLng -> Model -> Html
rootView uiChannel currentLocation model =
  div []
      [...
      ,case model.places of
        Nothing -> span [] []
        Just (Err err) -> div [class "alert alert-danger"] [text <| toString err]

        -- This is the new case:
        Just (Ok []) -> div [class "alert alert-warning"] [text "No results found."]

        Just (Ok xs) -> resultsList uiChannel currentLocation xs
      ...]

(Note that we put the more specific match Just (Ok []) before the more general one Just (Ok xs), because patterns are checked in order and the first one wins.)

We’re done.

Yup, there’s almost no work to do. Why? Because we had a good data model to begin with - one that was clearly defined and correctly represented the different states our app could be in.

Say it long, say it loud, programming begins with data-modeling.

Code

The whole app is a little larger, and I’ll be writing up other pieces. In the meantime, you can see the geocoding part in action by playing with the finished hackday app here and then read the source code to learn more.

Geocoding With Elm and ArcGIS, Part 1

Fri, 25 Sep 2015 00:00:00 GMT

I recently wrote a navigation app for a hack day, and since it’s written in Elm, and there aren’t enough blog posts about Elm, I thought I’d break down some of the pieces. Here’s a nice juicy one: letting the user search for a location, and turning that into latitude & longitude - ‘geocoding’.

The Spec

We need an input box you can type your search into.
When you submit, we fetch a list of possible matches from ArcGIS (the company that sponsored the hack day in question, and who have a Geocoding API).
We need to display those matches.
You can choose one. (A choice which will cause something else to happen. Something that’s outside the scope of this post).

Let’s see how Elm tackles it…

The Implementation

Every Elm app breaks down into a few simple parts:

A data structure representing the state of the world.
A set of possible events that can occur.
An event handler: a function that takes the current state of the world, and one event, and returns the new state of the world, plus any side-effects that must occur.
A function that can display the current state of the world.

Let’s break those pieces down:

Model

Our state of the world, our Model, will need to capture the contents of the form, the list of matching places, and the chosen place. And for UX-sweetness we’ll add in a loading-flag, so we can disable the form while we’re fetching results.

In Elm-land we’ll model this as a record. First we’ll need to define the type of a search result, which I’ll call a Place:

elm

type alias Place =
  {address: String
  ,latitude: Float
  ,longitude: Float}

Now we can define the whole app state:

elm

  type alias Model =
    {searchTerm : String
    ,loading : Bool
    ,places : Maybe (Result Error (List Place))
    ,chosenPlace : Maybe Place}

chosenPlace is a Place, of course, but initially no-one’s chosen anything, so we’ll wrap it in a Maybe.

places is the trickiest type to read here. Maybe (Result Error (List Place)) is saying:

Maybe We may or may not have…
Result Error ... the result of a request, which may be failure, or may be…
List Place a list of places.

That’s it. That’s the state of the world, the whole state of the world, and will fully describe the state of the world for the lifetime of the app¹.

Action

By convention, Elm describes all events that could occur in the system with an Action type. What are the events that can occur in our app?

The user types something, changing our search term.
The user hits submit.
The internets give us a list of places.
The user chooses a place.

That translates into a union type of:

elm

type Action
  = ChangeTerm String
  | Submit
  | PlacesResponse (Result Error (List Place))
  | ChoosePlace Place

For those unfamiliar with the syntax, this is saying that an Action can be any one, and only one of the values that follow. Some, like ChangeTerm carry a payload value of String, some like Submit are just markers.

That’s it. Those are the events, all the events, and Action fully describes every event that can occur for the lifetime of the app².

update

The next piece we need is an event handler, which we call update. Let’s start with an easy, but incomplete version:

elm

update : Action -> Model -> (Model, Effects Action)
update action model =
  case action of
    ChangeTerm s -> ({model | term <- s}, Effects.none)

This is saying, “if you give me an Action and the current Model of the world, I can process that Action and give you the new Model, plus any side-effects that need to occur (like kicking off an AJAX request).”

In this case, we’re just handling the “user types into the input box” case, to get used to the syntax. We unpack the string that the ChangeTerm action holds, and update the model with it. This case causes no side-effects, so we say that explicity with Effects.none.

The other easy event is “user chooses a place”. Let’s add that into our update function’s case statement:

elm

  case action of
    ...
    ChoosePlace p -> ({model | chosenPlace <- Just p}, Effects.none)
    ...

The next up is submit:

elm

  case action of
    ...
    Submit -> ({model | loading <- True}
              ,Effects.map PlacesResponse (findPlaces term))
    ...

That’s more interesting. We’re kicking off a search. In the model, we set the loading flag. This is purely for UX. Alongside that, for the first time, we have an interesting side-effect. We’re calling findPlaces to create a request, and then using map to wrap the response in a PlacesResponse Action, so that we’ll know know what kind event it is when we come to process it later.

We’ll see how findPlaces is defined later. For now, let’s finish off update by defining how to handle the response:

elm

  case action of
    ...
    PlacesResponse xs -> ({model | places <- Just xs
                                 , loading <- False}
                         , Effects.none)
    ...

When we get a response, switch that loading flag off, and store the places.

That’s it. That’s the business logic, all the business-logic, and fully describes how every event changes the state of the world for the lifetime of the app³.

Pause

This seems like a lot, so let’s take a breather. We have events. We have a state of the world. We have a function that moves the state of the world forward.

It’s nice that the events follow pretty directly from asking, “What can the user do?”

It’s also nice that once we’ve describe how to handle each event, we’re done with business logic.

In the next release of Elm, the compiler will be able to warn you if you miss out any event handlers⁴, and that will be very nice indeed.

So now we just needed to describe that findPlaces function, and worry about the UI. To do that we’ll need to talk about Effects, JSON-parsing and HTML-rending.

Why don’t you go and make a cup of tea⁵ and join me for part two.

In the meantime, you can play with the finished hackday app here and read the source code to learn more.

Good luck trying to say something that definitive in an Angular app. ↩
Good luck trying to say something that definitive in an Angular app. ↩
Good luck trying to say…oh, you get the point. ↩
And quickly, at that. Elm’s compiler is impressively fast. ↩
Make it /properly/ mind. ↩

Geocoding With Elm and ArcGIS, Part 2

Fri, 25 Sep 2015 00:00:00 GMT

Following on from part 1, we had completed our description of the app, and the event handlers, and we just needed to worry about making an AJAX request, decoding the response, and rendering the HTML.

Just. :-)

AJAX

ArcGIS has a nice API for geocoding¹. We send a GET with the term as a query parameter, we get back some JSON for places in the world that match that search term. It’s easy to use.

For our wiring, we have a search term, and we want to generate the side-effect that eventually returns a list of places. Remembering that that side-effect may fail with an HTTP error, we get this type signature:

elm

findPlaces : String -> Effects (Result Error (List Place))

To read the actual implementation you’ll need to know that |> is an operator that basically reads as a unix pipe | ²- take the results of the last function and pass them in as the input to the next function:

elm

findPlaces term =
  "http://geocode.arcgis.com/arcgis/rest/services"
    ++ "/World/GeocodeServer"
    ++ "/findAddressCandidates?f=json&singleLine=" ++ Http.uriEncode term
  |> Http.get decodePlaces
  |> Task.toResult
  |> Effects.task

There’s a lot of type-wrangling in there, but the noteworthy bit is that Http.get takes a JSON-decoder as its first argument, which will fire if the get request was successful.

JSON

Let’s look at a sample response from the API:

json

{
  "spatialReference": {
    "wkid": 4326,
    "latestWkid": 4326
  },
  "candidates": [
    {
      "address": "Royal Festival Hall",
      "location": {
        "x": -0.11599726799954624,
        "y": 51.50532882800047
      },
      "score": 100,
      "attributes": {},
      "extent": {
        "xmin": -0.120998,
        "ymin": 51.500329,
        "xmax": -0.110998,
        "ymax": 51.510329
      }
    }
  ]
}

For JSON-decoding I always pull in [JSON extras](https://twitter.com/circuithub][CircuitHub]]’s [[http://package.elm-lang.org/packages/circuithub/elm-json-extra/latest/), so let’s import those:

elm

import Json.Decode as Json exposing (..)
import Json.Decode.Extra exposing (apply)

Now the parsing is reduced to two functions, which make use of := to access a property and at to walk a nested path down the tree.

Decode one `Place`

elm

decodePlace : Decoder Place
decodePlace = Place
  `map`   ("address" := string)
  `apply` (at ["location", "x"] float)
  `apply` (at ["location", "y"] float)

Decode the `List` of `Places`

elm

decodePlaces : Decoder (List Place)
decodePlaces =
  at ["candidates"] (list decodePlace)

At the actual hackday I wrote this code, I found this kind of thing very fast to write. You’d think having to write a parser for JSON before you can use it would slow you down, but certainly on that day I found it sped things up: The first pass was slower, sure, but as I built out the app and discovered that my initial assumptions about the schema were wrong, having the compiler tell me all the places my assumptions were wrong saved a lot of time.

Right, That’s It, We’re Done

Oh wait, no. We haven’t actually displayed anything to the user yet.

Rendering

So the last piece we need is a rendering function. HTML gets huge, so I’ll just focus on an illustrative piece and point you to the source.

Most HTML-rendering functions in Elm have essentially the same signature. You might expect it to be as simple as:

elm

searchForm : Model -> Html

…but it’s slightly more complicated than that, because DOM.

It’s not enough to just generate HTML. We also need a way for user clicks and keypresses to generate events (as Elm Actions). So the most common type signature is some variant of:

elm

searchForm : Address Action -> Model -> Html

That Address Action will give us a way to make onClick events generate a Action, without polluting our rendering code with business logic. It’s a channel, queue, or postbox where we can send event data³.

So, let’s look at our search box. (Here’s a primer for reading Elm’s HTML syntax.)

elm

searchForm address model =
  div [class "form-group"]
      [input [class "form-control"
             ,autofocus True
             ,on "keyup"  targetValue (message address << TermChange)
             ,type' "text"] []
      ,hr [] []
      ,button [class "btn btn-lg btn-block btn-success"
              ,disabled model.loading
              ,type' "submit"
              ,onClick address Submit]
              [text "Search"]]

Let’s look at a few of those pieces:

onClick

This makes a button click turn into a Submit action:

elm

  ...
  onClick address Submit
  ...

disabled

Here we disable the button when there’s a load in-flight. It’s an easy way to give the user some feedback that the click actually caused something:

elm

  ...
  disabled model.loading
  ...

…recall that model.loading is handled in the update function. The button is not made responsible for tracking which AJAX requests are in-flight, as that would be an insane abuse of the single responsibility principal. It’s just a button that displays how it’s told.

autofocus

For another UX nicety, it’s trivial to make this form field capture the cursor when the page renders:

elm

  ...
  autofocus True
  ...

on

The only really juicy part of this code is the input keyup event handler. It’s a little more complex because it needs to refer to the value of the event target. It’s also interesting because it shows how you can handle any arbitrary event type:

elm

  ...
  on "keyup"  targetValue (message address << TermChange)
  ...

Whenever there’s a keyup event, grab the element’s value, wrap it up as a TermChange Action, and send it on its merry way.

We’re done

That’s it. A complete overview of a Geocoding module for an app written in Elm. It feels like we covered a lot of pieces:

Describing data
Describing events
Handling events
AJAX-y side-effects
Decoding JSON
React-style rendering

And that is a lot of pieces, but any app would need all of them somewhere. The nice thing about this approach is that each piece is discrete - you can think about it and write it in isolation, and then glue them together later⁴. You can also easily split the writing of them, so that coders who are good with HTML can focus on the rendering. I am muchly fond of this approach in general, and Elm’s implementation of it specifically. Especially when it’s a hack day and I need to move fast without breaking things.

Fin

And they’re not paying me to say that. Well, actually, this app won a prize at the hackday, and ArcGIS were the sponsors, so maybe that does count as payment. :-D ↩
Thank you Richard Feldman for that analogy. ↩
Clojurians might find it interesting that this is very much like a core.async channel, except it’s one-way. You can only write to an Address, and it has a counterpart on the other side that can only be read from. ↩
It’s an approach that can work in many languages, but Elm makes the distinctions really clear. ↩

Things I Love About Languages I Hate

Fri, 25 Sep 2015 00:00:00 GMT

I’m worried I’ve seemed a bit grumpy about tech lately. So, in an attempt to find silver linings and stream some positivity from my corner of the technosphere¹, here are some things I love about the things I hate:

PHP

It has all the best content-management systems. God only knows why, but it does.
It gets fast results for beginners, and you can’t underestimate the value of that.

Bash

If you just typed it at the command line, you’re 80% of the way to automating it.

JavaScript

It’s the only programming language my mother has installed. And my grandmother. And, basically, everyone.

PL/SQL (and variants)

It has, hands down, the best SQL integration of any language. That’s the only thing it’s got, but it’s got that. If your job just involves manipulating database rows, it’ll be written before your package manager has finished downloading an internet’s worth of ORM dependencies. And if your program processes a million database rows, a PL/SQL can finish the job before your ORM has finished deserialising the first 10,000.

Scala

Along with a few other, better languages, it’s a bit of a magnet for smart people right now.

MEAN

Mongo, Express, Angular & Node. It’s a really good acronym. I mean, really good. It’s memorable and sounds cool & edgy.

How can you have a corner of a sphere? ↩

Timing Code in ClojureScript

Fri, 25 Sep 2015 00:00:00 GMT

Here’s a handy little ClojureScript macro that helps you time how long code blocks take. It’s exactly like (clojure.core/time), in that it takes a code block and tells you how long it took to execute, but specially for ClojureScript it piggy-backs on the JavaScript console API, for pretty output in the inspector pane. You’d use it like this:

clojure

(console-time "Rendering Big Thing"
              (render-to-dom lots-of-data))

And the browser’s inspector console will print out Rendering Big Thing: 10070.868ms when it’s done.

How does it work? Well, it’s really just wrapping a code block in two console API calls, .time(<label>) and .timeEnd(<label>). We could almost say:

clojure

(defmacro console-time
  [label & body]
  `(do
     (.time js/console ~label)
     ~@body
     (.timeEnd js/console ~label)))

…splicing the body between those calls. But that would have a flaw: instead of returning the body’s result value, we’re returning the result of console.timeEnd, which is always nil.

So we take a second pass:

clojure

(defmacro console-time
  [label & body]
  `(do
     (.time js/console ~label)
     ~@body
     (.timeEnd js/console ~label)
     ~@body))

It’s pretty obvious that’s wrong too - right return value, but it’s executing the body twice. That’s bad for performance, and outright broken if body contains side-effects¹.

There’s nothing for it - we’re going to have to run the block once, capture its return value, and return that captured result later.

To capture it, we’ll have to create a temporary local var, with a guaranteed-unique name. We can do that by appending a hash symbol, as result#.

With that, the final code should make sense:

clojure

(defmacro console-time
  [label & body]
  `(do
     (.time js/console ~label)
     (let [result# (do ~@body)]
       (.timeEnd js/console ~label)
       result#)))

Remember if you’re using this forreals, all ClojureScript macros must go in a Clojure file, and be imported with (refer-macros ...).

Don’t worry, I know you would never write code with side-effects. We’re worrying about them. ↩

Dude, Do You Even Fizzbuzz?

Sat, 12 Sep 2015 00:00:00 GMT

Ah, fizzbuzz, that venerable challenge of our industry. I’ve lost count of the number of times I’ve been asked to fizzbuzz in an interview. Seriously, I’d like to get a blindfold with “fizzbuzz” embroidered on it, so if I’m sent to the whiteboard I can cover my eyes and see if I’m yet able to Pin the Tail on the Fizzbuzz Donkey.

Some people dislike it. Some refuse to do it at interview. Not me. I’ve been on both sides of the interview table, and I’ve seen glistening CVs turn into candidates that can barely find the right side of the keyboard. So I’m fine with starting an interview with an idiot filter. We’ll soon be on to talking about something more interesting.

But why am I writing about fizzbuzz? More importantly, why am I on holidaycation, sitting at a free bar, and writing about fizzbuzz? Because Someone Is Wrong On The Internet.

I saw a blog post today on the “right way” to TDD fizzbuzz in an interview, and it’s got my goat.

I have mixed feelings about the value of TDD. I won’t get into the gory details, because that’s worth an essay in itself, but the particular aspect on my mind is that TDD has a tendency to lead you into spec blindness. That condition that every junior developer has to learn about, where you think that correctly fulfilling the spec leads to correct code (and a pat on the head). It does not. You graduate from junior developer when you start seeing the need behind the presenting problem¹. And with fizzbuzz, the spec is deeply misleading.

The spec goes something like this:

Given an integer n, print the integers from 0 to n.
Every time you reach a number wholly divisible by 3, print ‘fizz’ instead.
Every time you reach a number wholly divisible by 5, print ‘buzz’ instead.
Every time you reach a number wholly divisible by 3 and 5, print ‘fizzbuzz’ instead.

So, Mr Wrong On The Internet advocates starting with this test case:

java

assertEquals("fizz",fizzbuzz(20).split(",")[3]);

Yes, this hits the spec. But it’s testing too many things, and has tests of trivia mixed in with testing useful code. But the big problem is, if you start with “test the spec” as your guide then it’s encouraging you to write the wrong code. It’s the programming equivalent of treating the symptoms instead of going for the root cause.

Instead of string-munging through the symptoms, let’s step back and do some quick analysis² on this problem. Fizzbuzz has few parts, and almost all of them are crashingly dull:

Print stuff as a string

Yawn.

Generate a sequence from 0 to n.

Yawn. (range n), or your language’s equivalent. Next.

Apply a transformation to a sequence.

Dull programming, but it does open the opportunity to pontificate about functional programming. I could probably even get the word ‘Functor’ in there, and that always gets the day moving. Still, dull programming.

Conditionally transform an integer.

There’s the nugget. This is the /only/ interesting part of the challenge. Fizzbuzz can be written as:

clojure

(->> (range n)
     (map f)
     (str))

…where f is the /only/ thing doing anything novel. Any coding you do outside of defining f is merely gluing the algorithm into the core libraries. And so I put it to you, any testing you do outside of testing f is an integration test, and a crashingly dull one at that. You might as well include writing tests for .toString().

So if you ever ask me to write fizzbuzz in an interview, I’ll start by writing a function of type Int -> String. Actually I’ll probably do it in Clojure, because I think it comes out neatest:

clojure

(defn fizzbuzz
  [n]
  (match [(mod n 3) (mod n 5)]
    [0 0] :fizzbuzz
    [0 _] :fizz
    [_ 0] :buzz
    :else n))

Or maybe I’ll go typecrazy with Haskell:

haskell

data Fizzbuzz
  = Natural Int
  | Fizz
  | Buzz
  | FizzBuzz
  deriving (Show)

fizzbuzz :: Int -> Fizzbuzz
fizzbuzz n =
  case (mod n 3,mod n 5) of
    (0,0) -> FizzBuzz
    (0,_) -> Fizz
    (_,0) -> Buzz
    _ -> Natural n

That’s sweet. It changes the type and still keeps the untransformed integers available.

Someone will say I’m doing TDD wrong. So be it. I’m a heretic. I’m saying if TDD is, “translating the spec-as-given into a test and then writing passing code,” then TDD is wrong³. However I solve fizzbuzz, you won’t see me start with string-slicing tests⁴.

And you graduate to senior developer when you learn to stop always trying to see the bigger picture, and remember that you’ve also just gotta ship. Balancing the bigger picture with the smaller one never ends. ↩
Henderson’s Law states that all problems can be solved with one of two approaches: 1) Think, and 2) For chrissakes, THINK! ↩
Someone will say that that isn’t TDD either. Bravo! Please continue to say it. Please say it louder, so as to drown out the mainline rhetoric. ↩
Besides, for fizzbuzz you want property-based testing, not example-based. ↩

Responsive YouTube Videos With Elm

Sun, 06 Sep 2015 00:00:00 GMT

I recently wrote an navigation app for a hack day, and since it’s written in Elm, and there aren’t enough blog posts about Elm, I thought I’d break down some of the pieces. Let’s warm up with a simple, but exceptionally useful one: Embedding a responsive video.

This is so easy it’s almost cheating. Most of the hard work is offloaded to Bootstrap 3. But since it’s so useful it’s worth spelling out. All we need is a function that takes a YouTube URL, and returns some HTML. That translates to this type signature:

elm

youtube : String -> Html

Then we need to know how to generate HTML in Elm. It’s pretty easy. Almost every HTML function takes a list of attributes, followed by a list of children. So a Bootstrap alert messsage might be:

elm

alert =
    div [ class "alert alert-danger" ]
        [ text "Ohnoes!" ]

…or with a title:

elm

alert =
    div [ class "alert alert-danger" ]
        [ h1 []
            [ text "Error!" ]
        , text "Ohnoes!"
        ]

(You can see there in the h1 tag that you must specify the list of attributes, even if it’s empty.)

With that, the code for our youtube function is an easy translation of the Bootstrap docs:

elm

youtube url =
    div [ class "embed-responsive embed-responsive-16by9" ]
        [ iframe
            [ class "embed-responsive-item"
            , src url
            ]
            []
        ]

And that’s all there is to it.

(For the curious, I’ve published a slightly fancier version as a library.)

When Should You Bother With A Reader Monad?

Thu, 06 Aug 2015 00:00:00 GMT

I have been thinking about Reader monads, and the bothering-with thereof. Here’s this morning’s stream of consciousness…

Thought of the day - what’s the difference between using a function foo :: Config -> SomeResult and using the monad foo :: Reader Config SomeResult?

Answer: Technically none. data Reader r a = Reader { runReader :: r -> a } is really an equation saying they’re exactly equivalent: Reader r a equals r -> a.

So what’s the point of the Reader monad? Why (or rather, when) should you bother with a Reader monad instead of a straight argument?

Answer: composition. The Reader version composes (maps/sequences/etc.) trivially, to produce a single function that’s just waiting for its config. The vanilla version requires more plumbing (& thought) to compose.

Partly that’s offset in Clojure - we don’t just use functional composition, we can use syntactic composition too (because macros).

But that’s my conclusion of a night in the hammock. The Reader monad is a transformational trick to enable easier functional composition. And so, the “should I use it?” question follows as, “am I doing a lot of config plumbing to sequence these function calls?”

Extending Esqueleto for Type-Safe Postgresql Queries

Tue, 28 Jul 2015 00:00:00 GMT

It’s no great secret that I’m not a huge fan of SQL DSLs. They tend to be great for the simple stuff, but they break down very quickly as soon as you try to do anything non-trivial. So it’s a bit odd that I find myself picking up a Haskell one - Esqueleto.

Why am I risking heartbreak? Well, because as long as I can stay on the Persistent/Esqueleto path, I’ll get DB migrations for free, and compile-time integrity checks that can cover my database, webserver and frontend in one pass.

That kind of compile-time checking - for almost-free - is not to be sniffed at. If I change a datastructure, I know that the all the server code, all the UI code and the complete database schema are in sync. I don’t think I’ve ever worked on a project that’s achieved that dream, so it’s worth a little perseverance.

As predicted, Esqueleto broke down for me pretty quickly. But it got back up off the floor quickly too, and that’s the point of this post: Let’s see how we can get a typesafe database query that uses SQL features the library authors didn’t consider…

My use-case, I confess, is weird. But that’s sort of the point - this is stuff the authors wouldn’t have foreseen. I am importing some data from a shoddy source, and then cleaning it up later. So I already had this SQL statement:

sql

  UPDATE product
  SET description = trim(chr(160) from description)
  WHERE description != trim(chr(160) from description);

For some reason $DATA_SUPPLIER is sticking weird characters around the description, and we didn’t catch it on import. We want to update any products that have the magic character chr(160) stuck on them.

Let’s see that update statement naively ported to Esqueleto:

haskell

  tidyDescriptions :: SqlPersistM ()
  tidyDescriptions =
    update $
    \product -> do set product [ProductDescription =. trimDescription product]
                   where_ ((product ^. ProductDescription) !=. (trimDescription product))
    where trimDescription product =
            (trim (chr (val 160))
                  (product ^. ProductDescription))

The syntax is a little funky, but even if you don’t really read Haskell you can probably see how it’s trying to say the same thing as the SQL above.

Unfortunately, Esqueleto has already broken down on us: trim and chr are apparently too Postgres-specific, and aren’t supported.

So what shall we do? Break out a horrible, string-guzzling rawSql function, losing type safety and opening up SQL Injection vectors? No! We just teach Esqueleto about these functions. First chr:

haskell

chr :: (IsString s)
    => SqlExpr (Value Int) -> SqlExpr (Value s)
chr = unsafeSqlFunction "chr"

This code basically reads as, “There’s a function called =chr= that takes an integer and returns a string, both in a SQL-ish context. Trust me. There really is.”

And then:

haskell

trim :: (IsString s)
     => SqlExpr (Value s) -> SqlExpr (Value s) -> SqlExpr (Value s)
trim pattern target =
  unsafeSqlFunction "trim"
                    (unsafeSqlBinOp "FROM" pattern target)

Which reads as, “There’s a function called trim that takes two strings and has weird syntax, but if you call it this way you’ll get a string back. Honest.”

With that, our update statement works. We don’t get type-safety within these functions - Haskell is just blindly taking our promise that these types are correct - but I can live with that. This is code we’ll write once and never change. The important thing is that every use of these new functions get the guarantees we’re looking for, and they were pretty trivial to add.

So an SQL DSL that doesn’t abandon you to DSHell. Cautiously, I think I could make friends here…

Map, Reduce and the Fold-Twins

Tue, 07 Jul 2015 00:00:00 GMT

map & reduce ¹ are the bread & butter tools of functional programming, but if you’re new to FP it’s easy to get muddled about which to use when, so here’s my mental checklist:

If you have a sequence and you want a sequence, use map.
If you have a sequence and you want a single value, use reduce (or
fold).
If you have a single value and you want a sequence, use unfold.

That’s a simplification, but a good rule of thumb to get started with. (The twist comes when you realise that the “single value” that reduce produces can itself be a new sequence.) But hopefully that will serve as a simple signpost if you’re coming from a world of for loops and wondering how to break things down.

Or map and fold/foldl/foldr , depending on your language. ↩

This Week In The Browser

Wed, 01 Jul 2015 00:00:00 GMT

ClojureScript has opened a window to no-JVM…

Elm has released a big improvement in its compiler errors…

PureScript has released an order of magnitude performance boost in its compiler times…

And there is genuine hope of progress for practical GHCJS…

I’m calling this a good week for the browser.

JavaScript - The Collection of Parts

Wed, 17 Jun 2015 00:00:00 GMT

JavaScript is awful.

It’s got Good Parts!

Yes, but it’s also got Bad Parts. Lots of them.

We won’t use those parts.

How?

We’ll be disciplined.

Seriously, how?

We’ll be really disciplined.

Hmm. What about the pitfalls you don’t all know about?

Okay, we’ll use a linter.

That would shield you from the Worst Parts.

We’ll use ES2015.

That would shield you from the Next-to-Worst Parts.

What about the poor async support?

We’ll use Promises.

What about the broken data-modeling?

We’ll use ImmutableJS.

What about the broken data-processing?

We’ll use Underscore.

What about the broken data-flow?

We’ll use RxJS.

What about the broken data-consistency?

We’ll use TypeScript.

What about the broken rendering?

We’ll use React.

Why don’t you just start with a well-designed language?

Oh we couldn’t do that - there’d be too much to learn!

A Brief And Partial Review Of Haskell In The Browser

Tue, 03 Feb 2015 00:00:00 GMT

Ah, the browser. Like iOS, it’s a place of pleasure and pain. Love the audience it reaches; hate the default language. So I have a keen interest in the rise of “Compiles to JavaScript” languages. ClojureScript is my current weapon of choice, and I’m very happy with it. But I’ve also spent a chunk of time researching Haskell-like options for the browser, and I thought I’d write up my experiences.

Before we begin, for context, here is:

What I’m Looking For In A Browser Language

Sane language design

That rules out JavaScript, CoffeeScript, TypeScript, ScalaJS and several others.

Separation of logic & rendering

If your onClick handler references business-logic functions directly, something is wrong.

No manual DOM manipulation

Are you still doing that? Stop it. Unless you enjoy manual memory management too. (In which case good luck to you, my fine retronaught.)

No callbacks

Our brains are happiest with a single line of control flow. Yes you can get comfortable with callback-passing, but it never sits as easily as not having to. And like DOM manipulation, you don’t have to anymore. It’s a job for the language, not for you. Save your brain.

The Players

GHCJS

GHCJS reuses the GHC compiler we all know & love, but modifies it to emit JavaScript. The result is extremely impressive. You really do get a full Haskell experience. The language features are the same, the compiler messages the same, even most stuff on Hackage just works. It’s good.

I’ve heard people complain about the size of the JS files it produces. Apparently in the early days it was particularly bad, but to me, now, it seems perfectly acceptable. Once a production webapp will minify & gzip to less than the size of an animated cat gif, it’s time to stop yapping about file sizes.

There were three downsides with GHCJS for me. The first is, how should you structure a full client-side app? I did some work on this with Ben Ford, and then he took it much further, but it’s still very much an area of research. I couldn’t say to anyone, “This is the way to build it; you’ll need these pieces.”

The second problem is that GHCi isn’t supported yet. Without that, I found incremental build times to be just a bit too long, even for a small app.

And the third problem, and for me the nail in the coffin, is it’s a real pain to install. If you thought cabal hell was hell, I can tell you there’s another layer of problems lying in wait. Honestly, I couldn’t even say to anyone, “You should try GHCJS.” Getting started is asking too much.

All these problems will be addressed, and quickly I expect. Simply because of the quality of compiler, GHCJS is the one to watch. But I’ve put it aside for now.

Haste

If not GHCJS, how about another, “Let’s make regular GHC emit JavaScript!” approach? That’s what Haste does.

I liked Haste initially, until I got to that absolutely necessary evil, JavaScript interop, and everything fell apart. I recall I wanted to share a simple list of integers between Haste and JavaScript, and I couldn’t do it. The official position seems to be, “Yes, it ought to be easy. If you have a patch I’ll merge it.”

If Haste needs serious expertise to pass [1, 2, 3], then it’s not for me.

PureScript

PureScript moves us away from GHC and into languages that are styled-after Haskell. PureScript seems to be winning a popular vote (in this ultra-niche), and I can see why. The language has the headline features from Haskell, interop is good, it all works, and the documentation is excellent.

There are a few downsides for me that eventually stacked up too high. Tooling’s a bit of a pain. (You’ll need Node. And Grunt. And Bower. Wait, are we actually building a JavaScript project?)

The compiler errors are…disappointing. Haskell development leans heavily on fast, high-quality feedback from the type system. If you squint, the GHC compiler is almost a REPL. From what I gather, GHC has taken a vast amount of engineering effort to get to where it is, so you can’t blame PureScript for lagging behind. But lag it does, and when PureScript spits out a type error, I feel very much alone.

And finally, the “how do we structure/how do we render?” question felt unanswered. There is a virtual DOM library, but it was scrappy. I didn’t have an Om-quality experience using it. I’m sure it will get better, but like GHCJS it feels like an untrodden path.

Bonus Tip: If you’re playing with PureScript, Take a look at Bodil Stokke’s Pulp for your building needs. That worked well for me.

Elm

Elm is nice. Very nice. The language is good. Control flow is sane. The documentation is expansive (though more scattered about than PureScript). Incremental compilation times are pleasing. The compiler feedback is good. (No GHC, but better than PureScript.) Interop is so easy it makes you wonder why interop is hard.

The tooling’s excellent, even before you get into the fancy stuff like the time-travelling debugger. You can just write some code, compile it in seconds, get a single page you can upload straight to S3, and your website’s running. And the path from that to a proper minified, production app is smooth enough.

I’ve heard people say Elm is great for game-like things, but little attention has been given to regular HTML apps. That’s not true. The elm-html library is, give or take, a perfect substitute for Om/React.

And the FRP approach may prove a bit of a learning curve for some, but I found it very natural. There’s a certain way that I’ve been structuring my ClojureScript apps, and Elm feels like it’s taking the same route, but has baked it into the language. I heartily approve.

On the downside, I do really wish it had typeclasses. And not even for shiny things like putting my monoids in the category of endofunctors. Just for simple stuff. Having to qualify List.map, Signal.map and Set.map gets old very quickly. And Json decoding would be much simpler if it supported return-type polymorphism.

Conclusion

As you can probably guess, for now I’ve settled on Elm. I’m building an MVP for a startup of my own at the moment, and the front end is Elm. I’ll let you know how it goes.

Oh, and I’ll be keeping my eye on GHCJS…

Coda

Last week I went to an Elm night at Twitter HQ and Tom Ashworth was speaking. I asked him if he thought Elm was ready for production. He said no. I wish I’d also asked if he thought JavaScript was ready for production… :-)

An Elegant Little core.async Solution

Thu, 20 Feb 2014 00:00:00 GMT

Yesterday I had a problem that core.async provides an incredibly neat solution to. Tim Baldridge was kind enough to sanity check it for me, so I thought I’d write it up.

Imagine you want to process a message at a maximum rate of one per second. In another language you’d probably say something like:

python

while true:
    start_time = date()
    process_message(queue)
    end_time = date()

    elapsed_time = end_time.subtract(start_time)

    if elapsed_time < 1000
        sleep(1000 - elapsed_time)

Not hard to write, of course, but kinda clumsy. Here’s the core.async version:

clojure

(while true
  (let [t (timeout 1000)]
    (process-message queue)
    (<!! t)))

In English, pick up a ticket that lasts for 1000 milliseconds; process a message; wait until the ticket expires. If processing the message took more than 1000ms, the ticket returns immediately, otherwise it blocks for the remaining time.

Simple and easy. A really neat solution.

Backwards Compatibility Testing For Your Clojure Project

Thu, 20 Feb 2014 00:00:00 GMT

Leiningen makes it really easy to test your project against multiple versions of Clojure. Really staggeringly easy. Like, “I don’t know a single other language where this is so seamless,” easy. It’s a little trick I picked up from clojure.java.jdbc, and I make good use of it in Yesql. Here’s how it goes:

Since version 2, Leiningen lets you specify multiple profiles, with their own dependency lists. What’s interesting is that Clojure itself can be one of those dependencies, so we can meaningfully add this to project.clj:

clojure

:profiles {:1.4 {:dependencies [[org.clojure/clojure "1.4.0"]]}
           :1.5 {:dependencies [[org.clojure/clojure "1.5.1"]]}
           :1.6 {:dependencies [[org.clojure/clojure "1.6.0-beta1"]]}}

Now calling lein with-profile +1.4 test means “run lein test under default profile dev, plus merge in the 1.4 profile”. Upshot: the code runs as normal, but under Clojure 1.4.

Now just add this alias:

clojure

:aliases {"test-all" ["with-profile" "+1.4:+1.5:+1.6" "test"]}

…and lein test-all will run the test suite three times, under each of Clojure 1.4, 1.5 and 1.6, automatically.

You can use that same trick for any dependency variations you like - so you could check against different versions of your database driver, for instance - but it’s this 4-line use case that really made my jaw drop.

Using Joda Time as your Clojure inst Class

Sat, 08 Feb 2014 00:00:00 GMT

In Clojure, the reader-literal #inst reads & writes java.util.Dates. For instance:

clojure

(type #inst "2014-02-06T00:00:00Z") ;=> java.util.Date

(pr-str (java.util.Date.)) ;=> #inst "2014-02-08T15:54:49.949-00:00"

But it’s possible to replace that behaviour to read and write Joda time instances. Here’s how:

DISCLAIMER This may be a terrible idea. In fact it probably is. Go away, unless you enjoy standing in the middle of a thunderstorm cackling, “It’s Alive! It’s Alive!”

Still with me? That’s the spirit!

First you’ll need clj-time. Add this to your project.clj’s :dependencies:

clojure

[clj-time "0.6.0"]

Second To get Clojure to read #insts as org.joda.time.DateTime classes, add this to src/data_readers.clj:

clojure

{inst clj-time.format/parse}

That just switches out the function that turns strings into date objects. Restart your REPL session to pick that up.

Third You’ll need to implement a printing method, so that the classes are written as #inst-tagged strings:

clojure

  (defmethod print-dup org.joda.time.DateTime
    [o writer]
    (.write writer (format "#inst \"%s\"" o)))

Make sure that code’s been imported into any namespace where you want this to work. Now you can say:

clojure

  {:start #inst "2014-02-06T00:00:00.000-00:00"
   :end #inst "2014-02-06T02:00:00.000-00:00"}

And get a map whose values are Joda DateTime objects.

As I say, this is probably a bad idea. It’ll unexpectedly break things for people. But it’s a fun hack anyway, so I thought I’d record it.

Our Total Quality Approach to Software

Sat, 25 Jan 2014 00:00:00 GMT

Here at CueCorp Ltd. we have a total approach to software Quality, experimenting with every possible angle to maximize shareholder value.

Our Quality Insurance department collects a regular monthly premium from our developers. In the event that bad code gets released to production, the devs put in a claim and get sent a cheque to cover the cost of fixing it. Premiums are climbing, but we have high hopes for this innovative, market-driven approach to Quality.

Our Quality Assurance department wanders around the office daily, talking to devs, managers and all other staff, confidently assuring them that the software will work. We’re not 100% sure of the value of this, but we checked the dictionary and it confidently assured us that that’s what ‘assurance’ means.

Our Quality Ensurance department is our most successful arm. Though their remit is wide, they take every angle and every opportunity to ensure the quality of our software. By writing test suites, by training our devs, by explaining the value of refactoring to our management team, by working with our architects to choose simpler and more reliable tools - there is no approach they won’t try to ensure that they ensure what they’re supposed to ensure.

Obviously, Quality is our highest priority, and our structure is an ongoing experiment. If you care more about software Quality than labels, rest assured (ensured? insured?) there’s a place for you here at CueCorp!

Book Review - Clojure for Domain Specific Languages

Sat, 18 Jan 2014 00:00:00 GMT

What a great title, eh? Clojure’s a fantastic language for writing DSLs, and yet it’s a hard topic get into. A good book on the subject is just what the Clojure ecosystem needs.

Sadly, this isn’t that book.

The biggest problem is that there’s hardly any content about writing DSLs. This is a ~240 page book, and about 200 of that is just “Intro to Clojure” stuff. That would be fine, but there are much better introductory books out there.

Filler aside, that leaves maybe 40 pages that are actually about writing DSLs. Sadly it’s the weakest stuff in the book. It consists of two examples of using Clojure to wrap existing Java libraries. One’s a Twitter client, the other’s a wrapper around Java’s Date & Calendar classes. And there’s no way around saying this - both are badly written.

It would be a terrible shame if any beginner in Clojure read this stuff and thought it was a good way to proceed. What should be a simple function to read a config file into a map is presented in an overwrought & brittle fashion. Macros are used frequently but never appropriately. Clojure code drops down into Java interop for no good reason at all. And the author seems to have an obsession with dynamically-binding variables, regardless of whether it’s appropriate. I strongly suspect this betrays a lack of understanding about what they’re for. Either way, the Twitter DSL is going to break as soon as you do anything interesting with it, like delay the results. (Stuart Sierra has a great post about it.)

So is there anything here to like? Yes, the author is quite lucid. His technical writing reads easily. I hope he writes a book on another topic one day - but he’ll need to be far better versed in the material.

In the meantime, if you want an intro to Clojure book, check out the superb Joy of Clojure. And if you’re a seasoned Clojure programmer looking for a good book on DSL writing…well…so am I. In the meantime, here’s a good article on one application of the technique…

Tech I Want To Build/Learn In 2014

Tue, 31 Dec 2013 00:00:00 GMT

Last day of 2013, and it’s time to look ahead to the year to come. Here are my tech aims for 2014:

Learn Haskell

I decided last year I wasn’t going to learn any new languages¹. Instead I was going to double down & get really tasty at Clojure. There’s still plenty of deliciousness left in that journey, but I think I’ve done well enough to pick up a little something new.

But what new thing should I choose? Well, probably the best advice I heard all year was from Steve Purcell, which I’ll paraphrase as, “Figure out what the smartest, nicest people you can find are learning, and learn that.” I look around and see Clojure & Haskell fitting that description, so it feels like the time is coming.

(Actually, Haskell isn’t new to me. It’s just my Haskell skills are so crusty as to earn a seat in the House of Lords. A 15-year hiatus, by my calculations.)

Chances of it happening: Strong. I love a new language. Would be nice if I had a commercial reason to motivate it though.

Write A Good Database Reporting System

Reports servers, universally, suck. In roughly-chronological order, I’ve used Oracle Reports, BIRT, Oracle Apex, JasperReports & Crystal Reports, and not one of them has shed a glimmer of hope. They’re tolerable, but they’re not good.

Here’s a test you can apply to see if your report-server sucks: Which is easier? Adding a pink circle to the top-left corner of a report, or adding a new column to it and pushing it live?

The fact that it’s easier to find the ellipse button than the deploy button is proof that something is tragically wrong.

I know why they all suck. I know exactly why. It’s because they’re designed on a lie. The lie is, “Use our tool and you can just drag & drop reports together. You’ll never need to learn SQL and you’ll never need to ask IT again.”

BS. Never happens. Putting a drag & drop interface on it doesn’t save you from learning the database schema, so the job gets pushed to IT anyway. Except now you’ve spent a packet on forcing a dumbed-down interface on people who’d be happier with SQL and git push. If we assumed the report-reader was non-technical, but the report-writer was a geek, we could get the turnaround time on new reports down to less than half an hour. And everyone would be happier.

Chances of it happening: Unknown. I can see how to build it. I believe it’s commercially viable. But I can’t see how to market it. That’s something to mull.

Write an EDN Parser for Emacs

I’ve been playing around with a few CIDER tricks lately - such as this bind-a-Clojure-command tip - to improve my Clojure-on-Emacs workflow. I could do some far cooler stuff if Emacs could read Clojure data structures, and an EDN parser’s the route to that.

For instance, I’m thinking a much better org-babel/Clojure integration² for starters. After that, I’d love to have a buffer that pretty-prints the value of an atom, watches it for changes, and does some watch-style syntax-highlighting for any diffs. That would rock.

Chances of it happening: Good. Geeky confession: I enjoy writing parsers. And I’m always buggering around with Emacs these days, so it’s probably just a matter of time.

Write An Editor

If you’ve been here before you’ve probably gathered that I’m an editor junkie, and there’s a constant pull to scratch my own itch. My ideal editor would be a mashup of these principles:

Treat The User Like A Programmer, The Way Emacs Does.

Why is Emacs the only programming tool that assumes you’ll want to program it? Sure, almost every editor has some means of extension, but Emacs really is in a different league. It opens on a “Program Me!” buffer for chrissakes.

(If you don’t believe Emacs is uniquely extensible, check out my video, Writing A Spotify Client in 16 Minutes.)

Treat The User Like An Expert, The Way Vim Does.

For any tool you’ll use for 40 hours a week, for the next decade, the initial learning curve is irrelevant. It’s how productive it makes you after the first month that counts.

Treat The World Like It’s The 21st Century, The Way LightTable Does.

In short, I would trade terminal support for an <svg> tag.

Chances of it happening: Slim, sadly. Writing a good editor - and a good one is the only kind I have any interest in writing - is a long, long road. Unless it can pay its way, I’m going to have to resist going down that rabbit hole.

Find A Day Job

2013 has been something of a sabbatical year for me. It’s been fantastic for feeding my brain, but too isolated, and if I’m brutally honest, a little navel-gazing. I’ve decided to brush up my CV and find some smart people to build interesting things with. I did a few contracts last year, and there is something delightful about building to spec. This year I’m going to dive in with both feet.

Chances of it happening: Strong. Tell me if you know of any interesting opportunities!

Okay, I tripped up & fell into Scala, but I got back up quickly. ↩
Better than this quick fix at least! ↩

Clojure On Emacs - A CIDER Workflow Hack

Sat, 28 Dec 2013 00:00:00 GMT

TLDR; In a few lines of Emacs Lisp, you can speed up your Clojure workflow, and it’s easy.

Recently I’ve been experimenting with clojure.tools.namespace and Stuart Sierra’s Reloaded Workflow. If you haven’t read that post, you should add it to your reading list. It’s a smart way of adding a reset button to your REPL session, to ensure a clean environment without having to restart the JVM.

This post isn’t really about that workflow per se, it’s just a good use-case to demonstrate a little Emacs/Clojure/CIDER-fu.

If you adopt that workflow, you’ll find yourself repeatedly typing this into the REPL:

clojure

  (require 'clojure.tools.namespace.repl)
  (clojure.tools.namespace.repl/refresh)

There are a few ways to save that typing, but a little Emacs Lisp is the most satisfying.

It’s Simple

The easiest way in is to use the function cider-interactive-eval.

emacs-lisp

  (defun cider-namespace-refresh ()
    (interactive)
    (cider-interactive-eval
     "(require 'clojure.tools.namespace.repl)
      (clojure.tools.namespace.repl/refresh)"))

  (define-key clojure-mode-map (kbd "M-r") 'cider-namespace-refresh)

Now M-x cider-namespace-refresh, or simply M-r, will run our Clojure string and hit the reset button. Job done.

It’s Reusable

It doesn’t take much imagination to see how we can re-use this code.

Say this afternoon’s development task means you’re going to be looking at the value of an atom a lot. These few lines put viewing that atom a single keystroke away:

emacs-lisp

   (define-key clojure-mode-map (kbd "M-r")
     (lambda ()
       (interactive)
       (cider-interactive-eval
         "(require '[clojure.pprint :refer [pprint]])
          (pprint @interesting-atom)")))

That’s a recipe for a function you can set up in a heartbeat, and throw away after a much zippier afternoon.

Popup Help In Emacs Lisp

Sat, 28 Dec 2013 00:00:00 GMT

If you use Clojure & Emacs, you probably use ac-nrepl-popup-doc to get the docs for the thing under the point, in a temporary popup window. And if you write any Elisp, you might be hankering for the same thing for Emacs’ docs.

Here’s some code that displays the same text as describe-function, but in a popup overlay:

emacs-lisp

(require 'popup)

(defun describe-function-in-popup ()
  (interactive)
  (let* ((thing (symbol-at-point))
         (description (save-window-excursion
                        (describe-function thing)
                        (switch-to-buffer "*Help*")
                        (buffer-string))))
    (popup-tip description
               :point (point)
               :around t
               :height 30
               :scroll-bar t
               :margin t)))

This code is actually fairly simple, so let’s break it down:

symbol-at-point picks up the thing under the cursor, which should be the function you’re hovering over.

The shenannigens inside save-window-excursion is just opening the regular describe-function window, snarfing its contents, and then closing it immediately. That puts the help text in description.

Then we just call popup-tip to display it. Here’s a screenshot:

You’ll probably want to set up your own favourite keybinding, but here’s one example, binding it to M-5:

emacs-lisp

(global-set-key (kbd "M-5") 'describe-function-in-popup)

Update

Thanks to a comment from Steve Purcell, we can refine this to work for variables & faces, not just functions. The big change here is that we replace describe-function with help-xref-interned.

emacs-lisp

(defun describe-thing-in-popup ()
  (interactive)
  (let* ((thing (symbol-at-point))
         (help-xref-following t)
         (description (with-temp-buffer
                        (help-mode)
                        (help-xref-interned thing)
                        (buffer-string))))
    (popup-tip description
               :point (point)
               :around t
               :height 30
               :scroll-bar t
               :margin t)))

One of the nice things about this change is that help-xref lets us supply our own temporary buffer, instead of re-using *Help*. That means we don’t clobber anything that’s currently in your help buffer.

To take advantage of that, we just need to set help-xref-following to t, and put the temp buffer in help-mode.

Emacs - Reindent The Whole Buffer

Thu, 26 Dec 2013 00:00:00 GMT

Quite often in Emacs, especially when writing Clojure code, I want to reindent the entire file. I think that arises from the fact that I’m writing a lot of Lisp these days, and Lisp is the only language I’ve ever encountered where automatic indenting actually works well.

Anyway, here’s the code I use to map meta-shift-q to “reindent the whole buffer”:

emacs-lisp

(defun reindent-whole-buffer ()
  "Reindent the whole buffer."
  (interactive)
  (indent-region (point-min)
                 (point-max)))

(global-set-key (kbd "M-Q") 'reindent-whole-buffer)

Pretty simple, huh?

A Tip About Clojure's Less Than & Greater Than Operators

Wed, 27 Nov 2013 00:00:00 GMT

Although it initially takes some getting used to, I think Clojure/Lisp’s prefix operators are a great idea. They make multi-argument functions really easy, so you can say things like:

clojure

(= a b c)

…instead of the much-clunkier:

javascript

a == b && b == c;

After a while you start to wonder why languages bother with operators that are limited to two arguments.

The < and > variants are equally cool, as you can do things like replace the classic range-check:

javascript

if ((0 <= x) && (x <= 10)) ...

…with:

clojure

(if (<= 0 x 10) ...)

But in the early days, I found it hard to remember whether I wanted (< x 2) or (> x 2). For the simple pair, greater-than or less-than just seemed to naturally belong in the middle, for the visual reminder.

So here’s my mind-hack tip: Think of them as ‘getting-larger’ and ‘getting-smaller’ operators, and look at the shapes of the characters.

The glyph < is a triangle that starts small on the left and gets larger as it goes right. (< 2 x) is checking if the arguments start small on the left, and get larger as you go right. Two is small, x is larger.

Similarly for >. It starts large on the left of the shape & gets smaller as it goes right.

I found that helped the penny drop for me. Hopefully it’ll help a reader or two…

Dear Starbucks, I Like Clojure

Wed, 23 Oct 2013 00:00:00 GMT

Dear Starbucks,

I like Clojure. It’s a relatively new JVM-based language in the family of Lisp languages, which puts it in a heritage that’s practically as old as programming itself.

It favours modelling problems using simple, immutable data structures - and let me tell you, once you’ve got used to immutable data structures you’ll never want to go back to a language without them.

It uses a syntax that’s akin to writing your code directly as an AST, which enables some very handy language features and editing tricks and again, will be hard to give up once you’ve gotten used to it.

And it has a REPL-driven development model that give you such a tight feedback loop on your code it’s more like you’re having a conversation with your computer - one that results in working code. It’s so much fun.

And that reaction you have right now, the one that says, “I’m glad you’re happy, but I don’t know what you’re talking about and I don’t really care so please just quietly get on with whatever you do best without telling me the details ever again.”

That’s how I feel when you ask me which kind of coffee bean I want.

Getting the Bleeding Edge Build of an Emacs MELPA Package

Tue, 15 Oct 2013 00:00:00 GMT

Today I learnt a great tip [Steve Purcell](https://github.com/clojure-emacs/nrepl.el/issues/382][here]] via [[http://www.sanityinc.com/). If you want to get the absolute latest build of an Emacs package, without waiting for the MELPA rebuild cycle to take place, just:

Clone the Melpa repo.
Open the recipes/ file in emacs.
Accept the dir-locals warning.
M-x package-build-current-recipe to evaluate that recipe locally and install the latest package.

I must remember that one…

Using OAuth2 with Emacs

Sun, 13 Oct 2013 00:00:00 GMT

I’ve been playing with the GitHub API, through Emacs. For all the interesting stuff, it needs OAuth2 authorization, and though there’s good support in Emacs for OAuth2, I couldn’t find a getting started guide. So, here goes:

First you’ll need an access token. The standard way of getting an access token - redirect to the service (Github), authorize, redirect back to your website - isn’t going to help us much. We don’t have a website to come back to. It’s just an Emacs client. But there’s an alternative, and all the providers I’ve played around with so far support it. They’ll issue you an access token manually for personal use.

So, we head off to Github > Account Settings > Applications and there’s a box marked “Personal Access Tokens”. Creating one of those gives us a token like 8914803c3f5038531dec13166ce098710c649281.

With that magic string we’re away. Require the package (you may need to M-x package-install it first), then create an oauth2-token struct:

emacs-lisp

(require 'oauth2)
(defvar my-token
  (make-oauth2-token :access-token "8914803c3f5038531dec13166ce098710c649281"))

With that we turn to oauth2-url-retrieve-synchronously. It will take a token and a url and return a buffer containing the HTTP response. We can dip into that buffer and read the JSON like this:

emacs-lisp

(require 'json)
(defvar user-data
  (with-current-buffer
      (oauth2-url-retrieve-synchronously my-token "https://api.github.com/user")
    (goto-char url-http-end-of-headers)
    (json-read)))

Job done. Let’s just pull out an item:

emacs-lisp

(cdr (assoc 'name user-data))

plaintext

Kris Jenkins

And that’s all there is to it. (Or at least, enough to get you started…)

Finding Files In Emacs Helm

Tue, 08 Oct 2013 00:00:00 GMT

Emacs’ Helm is awesome. (I’m on a bit of a helm kick this week). You can [use it as a code-browser](http://blog.jenkster.com/2013/10/writing-a-spotify-client-in-16-minutes.html][use it to write a Spotify client]], you can [[http://blog.jenkster.com/2013/10/a-tip-for-navigating-clojure-files-in-emacs.html), but you can also use it for simple things like finding files in a directory, tree or project.

Setup

The setup’s pretty easy. You’ll want to install it of course:

M-x package-install
helm

And then edit your ~/.emacs and kick it off:

emacs-lisp

(helm-mode t)

How It Works

Most helm stuff involves calling (helm ...) function with some named arguments. Once you’ve found the ones you want, you’ll want to bind it to a shortcut key, but for now let’s try a basic invocation from the scratch buffer.

The two named arguments I use most are :sources and :buffer. :sources gives helm a list of places it can find search candidates. :buffer just names the buffer.

Some Searchy Examples

So, a simple way to search for files under the current directory is:

emacs-lisp

  (helm :sources '(helm-source-findutils)
        :buffer "*helm-findutils*")

It’ll wait for you to enter at least 4 characters to narrow things down, then present a list.

We can add to that a few extra file-finding sources. If you package-install helm-ls-git, you can search the git project for the current file:

emacs-lisp

  (helm :sources '(helm-source-ls-git)
        :buffer "*helm-find-git*")

If you’re using Linux or OSX, you can search machine-wide through locate:

emacs-lisp

  (helm :sources '(helm-source-locate)
        :buffer "*helm-locate*")

Mix & Match

And rather cool, you can mix all three of these sources, in whichever order you prefer:

emacs-lisp

  (helm :sources '(helm-source-findutils
                   helm-source-ls-git
                   helm-source-locate)
        :buffer "*helm-find-files*")

Each section will be filtered separately, and you can jump between them with C-o.

More Than You Asked For

And finally, here’s a kitchen sink example. Have a look at the names of these sources and play around with any that sound useful:

emacs-lisp

  (helm :sources '(helm-source-findutils
                   helm-source-recentf
                   helm-source-bookmarks
                   helm-source-buffers-list
                   helm-source-google-suggest
                   helm-source-locate
                   helm-source-ls-git)
        :buffer "*helm all the things*")

And if that’s not enough, with helm enabled, C-h v helm-source will give you even more to digest…

Writing a Spotify Client in 16 Minutes

Tue, 08 Oct 2013 00:00:00 GMT

And the code is on Github.

Set An OSX Window To A Specific Size

Mon, 07 Oct 2013 00:00:00 GMT

This is a handy tip: I wanted to set a terminal window to a specific size, to check it was going to work on a projector later. This command line does the trick:

$ osascript -e 'tell application "terminal" to set the bounds of the first window to {0, 0, 1280, 720}'

That’s {left, top, right, bottom}.

A Tip For Navigating Clojure Files in Emacs

Wed, 02 Oct 2013 00:00:00 GMT

Here’s a little Emacs trick I’ve starting using recently for navigating Clojure. It’s an easy way to get a map of the file you’re working on, and jump around it. Once it’s working, it looks like this:

To set it up, first you’ll need helm, so if you don’t have it installed call:

  M-x package-install
  helm

Then, stick this in your .emacs:

emacs-lisp

(defun helm-clojure-headlines ()
  "Display headlines for the current Clojure file."
  (interactive)
  (helm-mode t)
  (helm :sources '(((name . "Clojure Headlines")
                    (volatile)
                    (headline "^[;(]")))))

Eval it, and you’re away. M-x helm-clojure-headlines will bring up the map. You can type a string to narrow down the list. Hitting return will leap to that point in the file. C-n and C-p will move through the list of matches.

How does it work? Mostly through Helm magic. All you really need to think about is the:

emacs-lisp

(headline "^[;(]")

…part. That’s just a regular expression that tells helm which lines in the file are headlines. For Clojure, I like “starts with a semicolon or opening parenthesis”.

For a more involved example, I use this version for Objective-C:

emacs-lisp

(defun helm-objc-headlines ()
  (interactive)
  (helm :sources '(((name . "Objective-C Headlines")
                    (volatile)
                    (headline  "^[-+@]\\|^#pragma mark")))))

A trickier regex, but still a very easy thing to setup for all it supplies. And this trick works for any filetype you can think of a sensible pattern to match against.

Give it a whirl. If you like it, I leave it to you to bind your own shortcut key. Oh, and you might like another, more complex thing I built with helm: The Clojure Cheatsheet for Emacs. :-)

Chicken, Chicken, Chicken

Tue, 27 Aug 2013 00:00:00 GMT

Chicken:

clojure

  (require '[clojure.walk :as walk]
           '[clojure.pprint :refer [pprint]]
           '[clojure.string :refer [replace]])

  (defn chicken
    [form]
    (walk/postwalk (fn [x]
                     (cond
                      (= x '&) x
                      (string? x) (replace x
                                           #"\w+"
                                           "chicken")
                      (symbol? x) 'chicken
                      :else x))
                   form))

Chicken chicken:

clojure

(pprint
 (chicken
  '(defn alter-map-values
     "Given a map, derive a new one where all the keys are passed through f.
      If f returns nil, the key is unchanged."
     [f m]
     (reduce (fn [altered [k v]]
	       (assoc altered
		 k (or (f v) v)))
	     {}
	     m))))

Chicken, Chicken, Chicken:

clojure

(chicken
 chicken
 "chicken chicken chicken, chicken chicken chicken chicken chicken chicken chicken chicken chicken chicken chicken chicken.
  chicken chicken chicken chicken, chicken chicken chicken chicken."
 [chicken chicken]
 (chicken
  (chicken
   [chicken [chicken chicken]]
   (chicken chicken chicken (chicken (chicken chicken) chicken)))
  {}
  chicken))

Chickens:

Chicken

Chicken Chicken

West London Hack Night - August 2013

Fri, 16 Aug 2013 00:00:00 GMT

Anonther West London Hack Night, and another success I’d say. We picked a tough one this time: Write an interpreter and a virtual machine in 90 minutes. That challenge was made both easier and harder by the choice of language: Brainfuck. It’s got a very simple specification, but the language is so hard to read that debugging the interpreter becomes a real brain…ache.

By my count we mustered 20 people spread across 7 teams. Not everyone managed to finish, but here are the projects that got published:

Not a bad result at all - almost everyone went home able to say they’d implemented a Turing Complete programming language from scratch…

Designing An Accounting System

Tue, 06 Aug 2013 00:00:00 GMT

A fast thought on writing accounting software: The most common mistake programmers make when writing an accounting system is they think it’s all about balances. They think the point of an accounting system is to tell you your account balance, so they design the data model around that.

They’re wrong. The point of an accounting system is to track the complete history of the movements of money and other assets. Auditing isn’t just a feature of the system, it is the system. Balances are just a summary report on the real data.

The Turing Rule

Tue, 06 Aug 2013 00:00:00 GMT

If someone tries to end a debate about programming languages by invoking Turing Equivalence, you are allowed to give them a long reel of tape and steal their laptop.

If mathematical equivalence were the only thing that mattered in language design, we’d all be using Turing tape. Turing Equivalence is what qualifies a language to enter the debate, not what finishes it.

Quote of the Day

Thu, 18 Jul 2013 00:00:00 GMT

Programmers talk about software development on weekends, vacations, and over meals not because they lack imagination, but because their imagination reveals worlds that others cannot see. - Thinking in C#

West London Hack Night - July 2013

Sat, 13 Jul 2013 00:00:00 GMT

[West London Hack Night](http://www.meetup.com/West-London-Hack-Night/events/120495682/][Last Thursday’s]] [[http://www.meetup.com/West-London-Hack-Night/) was great fun. The chosen topic was Genetic Algorithms. More specifically, given a pool of random strings, can you use GA to get them to evolve towards a target?

It’s a simple application of a sophisticated idea, but perhaps a perfect way to get started as you actually can code it up in about an hour and a half.

There were four teams hacking in four different languages, and here’s their code:

Great fun, as usual. I’m already looking forward to the next.

Levenshtein Distance in Elisp

Sun, 30 Jun 2013 00:00:00 GMT

Here’s a quick implementation of the Levenshtein distance - a measure of the similiarity of two strings - in Emacs Lisp.

elisp

(defun head (string)
  (substring string 0 1))

(defun tail (string)
  (substring string 1))

(defun levenshtein
  (string1 string2)
  (cond ((= 0 (length string1)) (length string2))
        ((= 0 (length string2)) (length string1))
        (t (min (+ (levenshtein (tail string1) string2) 1)
                (+ (levenshtein string1 (tail string2)) 1)
                (+ (levenshtein (tail string1) (tail string2))
                   (if (string-equal (head string1)
                                     (head string2))
                       0
                       1))))))

(list (= 6 (levenshtein "" "Sunday"))
      (= 8 (levenshtein "Saturday" ""))
      (= 3 (levenshtein "Saturday" "Sunday"))
      (= 3 (levenshtein "sitting" "kitten"))
      (= 2 (levenshtein "sitting" "kittin")))

Note: I wrote this for interest’s sake, because I couldn’t sleep. If you need a real implementation, you’d probably be better off with the one from EmacsWiki.

Solving Logic Puzzles With Clojure's Core Logic, Part Two

Wed, 27 Feb 2013 00:00:00 GMT

In part one we laid the foundations for solving logic puzzles with Clojure and core.logic. We built a core.logic query that encodes

rules of this particular puzzle, and come up with a solution.

The Rules

Let’s recap the rules:

Of Landon and Jason, one has the 7:30pm reservation and the other loves mozzarella.

The blue cheese enthusiast subscribed to Fortune.

The muenster enthusiast didn’t subscribe to Vogue.

The 5 people were the Fortune subscriber, Landon, the person with a reservation at 5:00pm, the mascarpone enthusiast, and the Vogue subscriber.

The person with a reservation at 5:00pm didn’t subscribe to Time.

The Cosmopolitan subscriber has an earlier reservation than the mascarpone enthusiast.

Bailey has a later reservation than the blue cheese enthusiast.

Either the person with a reservation at 7:00pm or the person with a reservation at 7:30pm subscribed to Fortune.

Landon has a later reservation than the Time subscriber.

The Fortune subscriber is not Jamari.

The person with a reservation at 5:00pm loves mozzarella.

When you look through them there are only really 5 kinds of rule in this puzzle, so we’ll group them and tackle them in order of complexitye

Simple Facts

Rules (2) and (11) are the simplest:

plaintext

* The blue cheese enthusiast subscribed to Fortune.
* The person with a reservation at 5:00pm loves mozzarella.

The first we handle by saying, “one of the answers - and we don’t care which one - has the shape [* :fortune :blue-cheese *]. We handle * with an (lvar), and one-but-any-one with membero, to give us this:

clojure

  (defn rule-2
	"The blue cheese enthusiast subscribed to Fortune."
    [answers]
    (membero [(lvar) :fortune :blue-cheese (lvar)] answers))

In English, [* :fortune :blue-cheese *] is a member of the answer list.

That’s fine, but I’ve a feeling we’re going to be writing (lvar) a lot, so for rule 11 we’ll start using clojure.tools.macro/symbol-macrolet for easier syntax:

clojure

  (defn rule-11
	"The person with a reservation at 5:00pm loves mozzarella."
    [answers]
    (symbol-macrolet [_ (lvar)]
                     (membero [_ _ :mozzarella 5] answers)))

Negated Facts

The second class of rules are negations:

The muenster enthusiast didn’t subscribe to Vogue.
The person with a reservation at 5:00pm didn’t subscribe to Time.
The Fortune subscriber is not Jamari.

Before we solve those, let’s take a look at a naive first-pass that tripped me up:

clojure

  (defn BAD-rule-3
    "The muenster enthusiast didn't subscribe to Vogue."
    [answers]
    (symbol-macrolet
     [_ (lvar)]
     (not-membero (== [_ :vogue :muenster _] answers))))

not-membero isn’t a built-in function of core.logic, but it’s easy enough to find.

Sadly this doesn’t work. It looks like it’s saying, “Match a person who likes Vogue and muenster cheese and make sure it’s not in the list of answers.” But really it’s saying, “Is it possible to construct a tuple that has Vogue and muenster, but isn’t in the answer list?” That answer to that is, yes. [nil :vogue :muenster nil] would be such a tuple. So the rule is trivially and always true. In effect, it does nothing.

No, to encode these rules correctly we need to say something about what is in the answer list, and then make extra assertions about those things. We need to say, “There’s a tuple with Vogue, a tuple with muenster, they’re both in the answer list but they aren’t the same tuple.” Like this:

clojure

(defn rule-3
  "The muenster enthusiast didn't subscribe to Vogue."
  [answers]
  (symbol-macrolet
   [_ (lvar)]
   (fresh [s1 s2]
          (== [_ :vogue _ _] s1)
          (== [_ _ :muenster _] s2)
          (membero s1 answers)
          (membero s2 answers)
          (!= s1 s2))))

That works. And note the use of fresh to introduce the named, local logic variables (magic boxes) we need.

Let’s encode the other two rules of this type:

clojure

(defn rule-5
  "The person with a reservation at 5:00pm didn't subscribe to Time."
  [answers]
  (symbol-macrolet
   [_ (lvar)]
   (fresh [s1 s2]
          (== [_ _ _ 5] s1)
          (== [_ :time _ _] s2)
          (membero s1 answers)
          (membero s2 answers)
          (!= s1 s2))))

clojure

(defn rule-10
  "The Fortune subscriber is not Jamari."
  [answers]
  (symbol-macrolet
   [_ (lvar)]
   (fresh [s1 s2]
          (== [_ :fortune _ _] s1)
          (== [:jamari _ _ _] s2)
          (membero s1 answers)
          (membero s2 answers)
          (!= s1 s2))))

The third class of rules relate pairs of facts:

Either the person with a reservation at 7:00pm or the person with a reservation at 7:30pm subscribed to Fortune.”
Of Landon and Jason, one has the 7:30pm reservation and the other loves mozzarella.

For these we can use conde to make some OR statements. Here’s the code for the first rule:

clojure

(defn rule-8
  "Either the person with a reservation at 7:00pm or the person with a
   reservation at 7:30pm subscribed to Fortune."
  [answers]
  (symbol-macrolet
   [_ (lvar)]
   (fresh [r]
          (membero [_ :fortune _ r] answers)
          (conde [(== r 7)]
                 [(== r 7.5)]))))

This is saying, “There’s an answer with Fortune in it, and I want to make a further assertion about the reservation, so name it r. My further assertion is, r is either 7pm or 7:30pm.”

Fairly simple. And the next is only a little more complicated:

clojure

(defn rule-1
  "Of Landon and Jason, one has the 7:30pm reservation and the other loves mozzarella."
  [answers]
  (symbol-macrolet
   [_ (lvar)]
   (fresh [c1 r1 c2 r2]
          (membero [:landon _ c1 r1] answers)
          (membero [:jason _ c2 r2] answers)
          (conde
           [(== r1 7.5) (== c2 :mozzarella)]
           [(== r2 7.5) (== c1 :mozzarella)]))))

This is saying, “There’s an answer with Landon in, and I want to make some assertions about his choice of cheese & reservation. Ditto for Jason¹.” Then the assertions, “Either the first reservation is 7:30pm and the second cheese is mozarella, or the other way around.”

Mutual Exclusion

Now we’re on to the home straight. The next class of rules - actually there’s only one of them - is a description of five separate tuples:

The 5 people were the Fortune subscriber, Landon, the person with a reservation at 5:00pm, the mascarpone enthusiast, and the Vogue subscriber.

The thing to notice here is that the rule’s really saying, whoever has one trait in that list can’t have any of the others. So we can use permuteo again to describe all 5 tuples, allowing for the facts to appear in any order:

clojure

(defn rule-4
  "The 5 people were the Fortune subscriber, Landon, the person with a
   reservation at 5:00pm, the mascarpone enthusiast, and the Vogue
   subscriber."
  [answers]
  (symbol-macrolet
   [_ (lvar)]
   (permuteo [[_ :fortune _ _]
              [:landon _ _ _]
              [_ _ _ 5]
              [_ _ :mascarpone _]
              [_ :vogue _ _]]
             answers)))

Relative Facts

Okay, last class of rule:

The Cosmopolitan subscriber has an earlier reservation than the mascarpone enthusiast.
Bailey has a later reservation than the blue cheese enthusiast.
Landon has a later reservation than the Time subscriber.

These are all very similar to the related pairs above, but with a greater-than or less-than statement that makes things a little tricker. Actually, it makes things much trickier, as we can’t just use < and > in core.logic.

I’ve taken an easy way out here. If I say that the reservation times are in strict order [5 6 7 7.5 8.5], then I can make greater-than and less-than statements by saying one is to the left of another.

I’ll define to-the-left-of as this:

clojure

(defne lefto
  "x appears to the left of y in collection l."
  [x y l]
  ([_ _ [x . tail]] (membero y tail))
  ([_ _ [_ . tail]] (lefto x y tail)))

In English, “either x is the head of the list and y is a member of its tail OR drop the head and recur.”

With that helper in place, the next three rules should look pretty clear:

clojure

(defn rule-6
  "The Cosmopolitan subscriber has an earlier reservation than the
   mascarpone enthusiast."
  [answers]
  (symbol-macrolet
   [_ (lvar)]
   (fresh [r1 r2]
          (membero [_ :cosmopolitan _ r1] answers)
          (membero [_ _ :mascarpone r2] answers)
          (lefto r1 r2 [5 6 7 7.5 8.5]))))

clojure

(defn rule-7
  "Bailey has a later reservation than the blue cheese enthusiast."
  [answers]
  (symbol-macrolet
   [_ (lvar)]
   (fresh [r1 r2]
          (membero [_ _ :blue-cheese r1] answers)
          (membero [:bailey _ _ r2] answers)
          (lefto r1 r2 [5 6 7 7.5 8.5]))))

clojure

(defn rule-9
  "Landon has a later reservation than the Time subscriber."
  [answers]
  (symbol-macrolet
   [_ (lvar)]
   (fresh [r1 r2]
          (membero [_ :time _ r1] answers)
          (membero [:landon _ _ r2] answers)
          (lefto r1 r2 [5 6 7 7.5 8.5]))))

Take It Home

That’s it. Our driver function and our rules are all encoded. Running it produces a list containing a single answer:

clojure

(((:amaya  :fortune      :blue-cheese 7)
  (:bailey :vogue        :asiago      8.5)
  (:jamari :time         :mascarpone  6)
  (:jason  :cosmopolitan :mozzarella  5)
  (:landon :us-weekly    :muenster    7.5)))

On my machine it takes about 1s to find the first answer, and about 16s to search exhaustively (and thus demonstrate that one answer is all there is).

You can find the code on GitHub if you want to give it a whirl.

Thoughts? Questions? Suggestions? Leave a comment below…

One of the hard parts of writing this code was typing Jason instead of JSON. ↩

Solving Logic Puzzles With Clojure's Core Logic, Part One

Wed, 27 Feb 2013 00:00:00 GMT

I used to love solving logic puzzles as a kid. Bought books of ’em. Great fun. They’re a puzzle with this sort of shape:

There are four sets of five things. For example, 5 people, 5 magazines, 5 reservation times and 5 cheeses.
There are rules that connect them together, like, “Jason likes mozzarella.”
The rules are exclusive. If we’re told Jason likes mozzarella, then Jason likes no other cheese, and nobody else likes mozzarella.
You are given enough clues about the relationships between these things that only one possible configuration can exist. Find it.

It cries out, “Logic Programming”, doesn’t it? Well, recently the London Cloure group took a crack at it. We got pretty far, but didn’t complete it before the night was over. I thought I’d finish it off and write it up, for your pleasure. Here goes…

I’ve taken a puzzle from Logic Puzzles.org, who seem to have a army of them. Here are the sets of things we’re dealing with:

People: Amaya, Bailey, Jamari, Jason & Landon.
Cheeses: Asiago, Blue Cheese, Mascarpone, Mozzarella & Muenster
Magazines: Fortune, Time, Cosmopolitan, US Weekly & Vogue
Reservation times: 5pm, 6pm, 7pm, 7:30pm, 8:30pm.

And here are the rules. You can skim over them. We’ll go through them in detail in part two.

Of Landon and Jason, one has the 7:30pm reservation and the other loves mozzarella.

The blue cheese enthusiast subscribed to Fortune.

The muenster enthusiast didn’t subscribe to Vogue.

The 5 people were the Fortune subscriber, Landon, the person with a reservation at 5:00pm, the mascarpone enthusiast, and the Vogue subscriber.

The person with a reservation at 5:00pm didn’t subscribe to Time.

The Cosmopolitan subscriber has an earlier reservation than the mascarpone enthusiast.

Bailey has a later reservation than the blue cheese enthusiast.

Either the person with a reservation at 7:00pm or the person with a reservation at 7:30pm subscribed to Fortune.

Landon has a later reservation than the Time subscriber.

The Fortune subscriber is not Jamari.

The person with a reservation at 5:00pm loves mozzarella.

There’s only one possible solution, and core.logic is going to find it for us.

Before We Start, What’s An `lvar`?

Imagine a closed box. I’ll tell you that inside the box is something that is grey, weighs one tonne, and has a horn on its nose. When we open the box, you won’t be surprised to see it contains a rhino. That’s how I think of a logic variable, or lvar. It’s a box containing something that’s unknown right now. We can state some rules about what must be inside, and when we come to open the box, the right kind of thing will pop out.

So, with that in mind, let’s set up our solution.

The Setup

We’ll start with this code:

clojure

(let [people       (repeatedly 5 lvar)
      magazines    (repeatedly 5 lvar)
      cheeses      (repeatedly 5 lvar)
      reservations (repeatedly 5 lvar)
      answers (map list people magazines cheeses reservations)]
  (run* [q]
       (== q answers)))

Here, we’ve done four things:

Called the lvar function five times, to create boxes for the five people.
Repeated that three more times for magazines, cheeses and reservations.
Spliced those four lists together, using (map list ...), to create a structure that’s a list of five [person, magazine, cheese, reservation-time] tuples. When we open all those boxes, we’ll have our answer.
run a simple query: “Find all the qs, such that q is a possible answer.”

So far, so good. Now all we need to do is add the right constraints, and the boxes will magically open to reveal the single answer to the puzzle.

First Constraints

So the first thing we need to constrain is the list of people, magazines, cheeses and reservations. This would be a start:

clojure

(let [people       (repeatedly 5 lvar)
      magazines    (repeatedly 5 lvar)
      cheeses      (repeatedly 5 lvar)
      reservations (repeatedly 5 lvar)
      answers (map list people magazines cheeses reservations)]
  (run 1 [q]
       (== q answers)
       (permuteo people [:amaya :bailey :jamari :jason :landon])
       (permuteo magazines [:fortune :time :cosmopolitan :us-weekly :vogue])
       (permuteo cheeses [:asiago :blue-cheese :mascarpone :mozzarella :muenster])
       (permuteo reservations [5 6 7 7.5 8.5])))

That says that people matches the list of five names, but not necessarily in the same order. It’s a permutation of the given list. We repeat that pattern for all four kinds of thing.

That’s almost right, but there’s a problem with ordering. Even once we’ve figured out the correct tuples, we’re going to get them in all possible orders, which is 5! or 120 answers, all of which are equivalent.

There are a few ways to handle this. We could to use a Set for the answers, instead of a list. That would be good in theory, but core.logic doesn’t support sets¹. We could ignore the problem and just take whichever answer pops out first. Or we could define a fixed order for any one of the collections, and let the others fall in line with that.

Fixing the order is the approach I like, because then there’s only one possible solution, which makes it easy to check we’ve got the rest of the code right later. So let’s fix the order of the people:

clojure

(let [people       (repeatedly 5 lvar)
      magazines    (repeatedly 5 lvar)
      cheeses      (repeatedly 5 lvar)
      reservations (repeatedly 5 lvar)
      answers (map list people magazines cheeses reservations)]
  (run 1 [q]
       (== q answers)
       (== people [:amaya :bailey :jamari :jason :landon])
       (permuteo magazines [:fortune :time :cosmopolitan :us-weekly :vogue])
       (permuteo cheeses [:asiago :blue-cheese :mascarpone :mozzarella :muenster])
       (permuteo reservations [5 6 7 7.5 8.5])))

And with that, we’re ready to start encoding puzzle rules. We’ll pull them out as separate functions, so our final main function will look like this:

clojure

(let [people       (repeatedly 5 lvar)
      magazines    (repeatedly 5 lvar)
      cheeses      (repeatedly 5 lvar)
      reservations (repeatedly 5 lvar)
      answers (map list people magazines cheeses reservations)]
  (run 1 [q]
       (== q answers)
       (== people [:amaya :bailey :jamari :jason :landon])
       (rule-1 answers)
       (rule-2 answers)
       (rule-3 answers)
       (rule-4 answers)
       (rule-5 answers)
       (rule-6 answers)
       (rule-7 answers)
       (rule-8 answers)
       (rule-9 answers)
       (rule-10 answers)
       (rule-11 answers)
       (permuteo magazines [:fortune :time :cosmopolitan :us-weekly :vogue])
       (permuteo cheeses [:asiago :blue-cheese :mascarpone :mozzarella :muenster])
       (permuteo reservations [5 6 7 7.5 8.5])))

We’ll look at the individual rules in part two…

Yet. ↩

Writing Great Literature With Clojure

Mon, 25 Feb 2013 00:00:00 GMT

Witness the might of lazy evaluation:

clojure

(require '[library.reference-section :as reference])

(def complete-works-of-shakespeare
  (filter #(= reference/complete-works-of-shakespeare %)
          (map typing (repeat monkey))))

Thanks to lazy evaluation, an infinite number of monkeys can be easily evaluated, even in memory-constrained environments¹.

This technique can be used for any body of text for which you already have a reference copy. ↩

Prolog Quote of the Day

Fri, 22 Feb 2013 00:00:00 GMT

Prolog is a great language for writing append; after that it’s all downhill.

Paul Graham

I’m not saying I agree or disagree, but it’s a good line…

Org Babel, Clojure and nREPL

Thu, 21 Feb 2013 00:00:00 GMT

At the moment, org-babel’s Clojure evaluator assumes you’re running swank. But if you’ve made the switch from swank to nREPL, that’s not going to work. I bumped up against this problem this afternoon while writing this little Clojure-related blog post.

Getting org-babel to use nREPL is easy enough though. I’ve just added this to ~/.emacs:

emacs-lisp

(eval-after-load "ob-clojure"
  '(defun org-babel-execute:clojure (body params)
     "Execute a block of Clojure code with Babel and nREPL."
     (require 'nrepl)
     (if (nrepl-current-connection-buffer)
         (let ((result (nrepl-eval (org-babel-expand-body:clojure body params))))
           (car (read-from-string (plist-get result :value))))
       (error "nREPL not connected!"))))

It’s just a first pass. It works, but I’m not sure I’m handling all the options correctly, so I’ll let it sit here for a while until I’ve given it enough exercise to be worth submitting back to the main project. Still, you might find it useful. Any feedback would be appreciated. :-)

Deepest Node In A Binary Tree

Wed, 20 Feb 2013 00:00:00 GMT

Since a friend has just raised the question, “Find the deepest node in a binary tree,” here’s my Clojure-flavoured answer:

clojure

(defn leaf?
  [node]
  (not (map? node)))

clojure

(defn deepest
  [node]
  (if node
    (if (leaf? node)
      [0 node]
      (let [[depth leaf] (last (sort [(deepest (:left node))
                                      (deepest (:right node))]))]
        [(inc depth) leaf]))))

And here it is in action:

clojure

(deepest {:left "A"
          :right {:left {:left "B"
                         :right "C"}
                  :right "D"}})

plaintext

| 3 | C |

His response? “It’s a bit longer in C++.”

A Vim Front-End for gDocs

Wed, 20 Feb 2013 00:00:00 GMT

tl;dr Google Docs + Vim = Yay.

Ah, the grail. Vim everywhere. Once Vim’s got its claws in you, you can’t settle for anything else. Every other editor is frustrating and second-class.

But for for Google Docs I’m forced to make an exception. It’s my ideal way to work with normal people¹. Especially when it’s changing a lot. Google Docs is user-friendly, it’s got all the features we need, and no-one ever stops the conference call to say, “Wait - did I email you the latest version?”

But I still don’t want to give up Vim. How could we connect the two? The way I see it, there are three possible solutions.

Vim talks to Google.

We could build a plugin for Vim that connects to the Google Docs server. Ever since Google Wave came & went, there has been plenty of information available on how to build a collaborative document-editing system. We can assume that Google Docs approaches the problem along these lines. The problem I think, wouldn’t be with the protocol, but with Vim itself.

Vim is single-threaded, so it’s not going to lend itself well to listening asynchronously for changes from the central server. We could probably work around that by reaching out from Vim into Python or Ruby, but that instantly puts us into the realms of, “If this plugin doesn’t work, simply re-compile your copy of Vim.” Yuck.

Also, Vim has a strictly text-only display. Even if we could happily listen for changes, displaying them is going to be tricky. We could use some syntax-highlighting tricks to simulate multiple cursors, but I think it’s always going to look like a hack.

I don’t think building Google Docs collaborative editing into Vim is impossible², but I’m certain it would be swimming against the tide. Getting it to work well would be a mammoth project.

Maybe Emacs?

Emacs has a very good Vim-emulation mode, and copes well will multiple processes and displaying text-with-extras³. It would be possible. In fact, I’ve built a prototype myself, and whilst it’s not releasable, it certainly proves the concept. But it puts us in an even worse position: “To install this plugin, just install Emacs.” I can’t see any Vim user accepting that as a solution⁴.

Some other collaboration service.

Another way forward would be to bypass both Vim & Google Docs for something Vim-like and Google Docs-like, that’s better suited to the task. There are a few online editors that offer collaboration, and several of them are based on CodeMirror⁵, which has a passable Vi-emulation mode.

This is probably the shortest path to a solution, but it still suffers from our opening problem: You have to turn to your non-technical colleagues and say, “Can we switch from Google Docs just so I can use Vim?”, and they look at you like you’re an alien. No, the right solution won’t involve having to change an existing team’s behaviour just to accommodate our preferences.

Google Docs becomes Vim-like.

So the last solution, and I think the only viable one, is to graft a Vim-like input system on top of Google Docs. If we do that, we don’t have to disturb existing users, we ought to get live collaboration & display for free, and we just have to worry about one key problem: How much Vim do we need? How much would we have to implement before we felt at home in Google Docs?

Naive Vim extensions rarely seem to get much beyond ESC, h, j, k, l before they think they’ve got the essence of Vim. We know there’s much more to it. With that in mind, here’s my list of what we’d need, and a few things I think we can leave out:

Requirements

Vim-like editing for Google Documents, without disturbing ordinary users.
Support for:
- Modal editing. (Normal, insert and visual modes).
- First-class motions & text objects. d5j should delete 5 lines. ci( should change all the text in the current parentheses.
- Ex commands. Not a direct translation of the Vim ones, but something like :wordcount should trigger the word-count box. :help should trigger the Google Docs help.
- Custom keyboard mappings. Some equivalent to nnoremap.
- Windowing commands remapped to switch between main, comment and sharing panes.

Non-requirements

Plugins. Not for v1 at least.

Conclusion

I think it would be the right approach. I think it would be a reasonably-sized project. I think it would make the world a better place. So I’m putting the idea out into the world. What do you think?

Normal people. Non-geeks. People whom, I accept, shouldn’t be expected to learn Vim. ↩
I should note that it’s possible to edit Google Documents with Vim using the Google Command Line tool. But that approach struggles with translating between gDocs’ format and plain text, and it doesn’t work collaboratively at all. ↩
[multiple cursors](https://github.com/magnars][Magnar Sveen]] of [[http://emacsrocks.com/][Emacs Rocks]] has a nice [[https://github.com/magnars/multiple-cursors.el) project that would give some clues. ↩
As I’ve written, these days Emacs is one of my favourite versions of Vim. But I don’t expect to convert anyone. (Hell, I’m not even certain I’d want to.) But if you like Vim & Lisp, it’s worth taking a look at Evil-mode. ↩
CodeMirror’s Vim-mode is far from perfect, but it wouldn’t be hard to make it much, much better. And the same thing goes for all the browser-based Vim emulations: Not good enough, but a good start. ↩

A Simple Data Equation

Sat, 16 Feb 2013 00:00:00 GMT

Take the number of copies you have of your data, and subtract one. That’s how much your data matters to you.

.emacs, To Start With

Sat, 16 Feb 2013 00:00:00 GMT

This isn’t an original recipe, but this is how I kick off my .emacs settings to ensure that my-packages are auto-installed.

emacs-lisp

;;; Package magic.
(require 'package)
(package-initialize)

(add-to-list 'package-archives '("melpa" . "http://melpa.milkbox.net/packages/"))

(defvar my-packages '(evil key-chord helm elscreen ace-jump-mode
                           surround recentf magit)
  "A list of packages to check for and install at launch.")

(defun my-missing-packages ()
  (let (missing-packages)
    (dolist (package my-packages missing-packages)
      (or (package-installed-p package)
          (push package missing-packages)))))

(let ((missing (my-missing-packages)))
  (when missing
    ;; Check for new packages (package versions)
    (package-refresh-contents)
    ;; Install the missing packages
    (mapc (lambda (package)
            (when (not (package-installed-p package))
              (package-install package)))
          missing)))

Mapping "jk" to Escape in Emacs' evil-mode

Sat, 16 Feb 2013 00:00:00 GMT

I’m a Vim user. But more & more these days I find that one of my favourite versions of Vim is Emacs. Specifically, its evil-mode package, which is a very good Vim emulator¹.

If you get started with evil-mode, one of the first questions you might ask is, “How do I remap ESC to something more finger-friendly?” Some people like to map jj. Personally I like jk, because it rolls off the fingers a little more easily.

Anyway, there are a couple of ways to do it in Emacs. The simplest & cleanest I’ve found is to install the key-chord package:

emacs-lisp

M-x package-install
key-chord

…then add this to your .emacs file:

emacs-lisp

(require 'key-chord)
(key-chord-mode 1)
(key-chord-define evil-insert-state-map  "jk" 'evil-normal-state)

Easy.

Actually, it’s the only good Vim emulator/plugin I’ve ever encountered. ↩

Today I Learnt...Something About Stack Overflow

Tue, 12 Feb 2013 00:00:00 GMT

…that the way Stack Overflow detects the syntax for a code-block is by looking at the tags on the question. If your syntax-highlighting isn’t working, add the tag for the language.

Clojure Syntax Highlighting in Emacs

Sat, 26 Jan 2013 00:00:00 GMT

So you’ve just written a new macro for your Clojure DSL, and now you have something like:

clojure

(defwidget my-widget
  ...)

And you’d love to add syntax highlighting to Emacs so that it gets highlighted the way defn and defmacro do. Here are the magic words:

emacs-lisp

;;; Adds new defn-esque keyword highlighting.
(add-hook 'clojure-mode-hook
          '(lambda ()
             (font-lock-add-keywords
                nil
                '(("(\\(defwidget\\)\\s-+\\(\\w+\\)"
                   (1 font-lock-keyword-face)
                   (2 font-lock-function-name-face))))))

What’s going on here? Well, let’s break it down:

emacs-lisp

;;; Add a function that runs for any Clojure buffer.
(add-hook 'clojure-mode-hook
  ;; An anonymous function that takes no arguments.
  '(lambda ()

    ;; Set some new syntax-highlighting rules.
    (font-lock-add-keywords nil

      ;; So many escape codes! But we're really just saying:
      ;; Match the '(' character.
      ;; Match and group the string 'defwidget'.
      ;; Match some whitespace. \\s-+
      ;; Match and group some word characters. \\w+
      '(("(\\(defwidget\\)\\s-+\\(\\w+\\)"

            ;; The first regexp group is a keyword.
            (1 font-lock-keyword-face)

            ;; The second regexp group is a name.
            (2 font-lock-function-name-face)

      ))))) ; Close up the patient.

Lisp Quote of the Day

Tue, 04 Dec 2012 00:00:00 GMT

“Why are macros so important to Lisp programmers? Not merely for the syntactic convenience they provide, but because they are programs that manipulate programs, which has always been a central theme in the Lisp community. If FORTRAN is the language that pushes numbers around, and C is the language that pushes characters and pointers around, then Lisp is the language that pushes programs around.”

Gabriel & Steele, The Evolution of Lisp.

An old quote, but still thoroughly relevant. (Though today you’d probably pick something in place of Fortran, and add a note about Java being the language that pushes objects around.)

Google Bootcamp Sarajevo

Sun, 04 Nov 2012 00:00:00 GMT

Two weeks ago, I was invited to fly to Sarajevo and run a Google Bootcamp. We debated about the content for a bit, and settled on a training & hacking day around AngularJS. A couple of hour-long talks from me, and as much hacking as possible. Cue frantic writing of slides.

On Friday we flew there via Vienna - not quite believing it was real - and yesterday they came, they saw & they hacked:

It was a fantastic day. Everyone had great fun & got their brains filled with new ideas. Me too.

I’m still churning over the event in my mind, it’s still buzzing my brain, so I can’t really write an elegant post about it yet. But if you’ll indulge, here are a few random memories while they’re fresh:

The attendees were great. Apparently there’s not really a tech scene in Sarajevo like there is in London. Talks like this are a rare thing. But everyone was straight in and hacking. There were some really great ideas being hacked on & everyone was keen to learn. Geeks of Sarajevo, I salute you.

Doug Crockford - I stole some of your JavaScript jokes. I’m sorry. JavaScript jokes are hard to write and you cornered the market early. To make up for it I told people to use JSLint & buy your book.

I met the President! Well, I met a President. The event was held at a university, and the current Arch-Chancellor is the ex-President of Bosnia. He gives good handshake.

At the start of the day, they clipped a microphone on me and said, “Say something while we check the levels.” Normal people say, “One two. One two.” I recited The Owl & The Pussy Cat. Parenthood has changed me.

One of the local TV stations showed up and interviewed me. That was weird. On a day like this, hackers come up to you and ask things like, “How do you conditionally hide DOM elements generated by an ng-repeat tag?¹” But TV people ask you questions like, “What do you think this means for Bosnia?” Under the circumstances, I think I switched gear fairly smoothly. :-}

As if we didn’t have enough to worry about, I decided that we’d organize a couple of extra lightning talks. I figured on a day like this it’s my job to throw as much information into people’s heads as I possibly can. Amer - one of the guys helping me out - and I threw some ideas onto a whiteboard, and people voted. Less and Hadoop won the popular vote, so we each improvised 10 minutes of information. The bonus talks seemed to go down really well. I’m glad we did it. (For the record, I did the Less talk. That’s easy. You pretty much just have to say, “Less exists!” and people get it. Less rocks. Less is easy and it rocks.) (For the other record, I’d offered to do a Vim Masterclass. It lost to Less by one vote. Sadface.)

I love doing this kind of thing. It doesn’t matter if it’s a guitar or a slideshow clicker in my hand, I like it. There’s a feeling that comes upon you in the hours before the deadline. If you run away from that feeling it’s called stagefright. If you run towards it it’s a complete adrenaline rush. And speaking as a sedentary programmer who hates sports, it’s rare that my body has reason to produce adrenaline.

I wrote my own slideshow software. I’d normally scoff at such wheel re-invention, but I really wanted to be able to show Angular working live, within the slides, so I wrote an Angular slideshower that ran Angular code inside my Angular slides². But on the morning of the event I hit a minor crisis. The room was unexpectedly huge, and they handed me a wifi clicker. I’d expected to be standing near my laptop and hadn’t written clicker support into the code. To my relief & amazement, I managed to hack it in in about 5 minutes. Say what you like about JavaScript, but I couldn’t have done that in Java or Objective-C half as fast.

Answer: Don’t. Filter the ng-repeat values up-front instead. ↩
It’s rough - I might find some time and polish it up into a real app - but you can see it here if you like. ↩

LISP vs Java

Thu, 01 Nov 2012 00:00:00 GMT

This is quote of the day:

DOLIST is similar to Perl’s foreach or Python’s for. Java added a similar kind of loop construct with the “enhanced” for loop in Java 1.5, as part of JSR-201. Notice what a difference macros make. A Lisp programmer who notices a common pattern in their code can write a macro to give themselves a source-level abstraction of that pattern. A Java programmer who notices the same pattern has to convince Sun that this particular abstraction is worth adding to the language. Then Sun has to publish a JSR and convene an industry-wide “expert group” to hash everything out. That process–according to Sun–takes an average of 18 months. After that, the compiler writers all have to go upgrade their compilers to support the new feature. And even once the Java programmer’s favorite compiler supports the new version of Java, they probably still can’t use the new feature until they’re allowed to break source compatibility with older versions of Java. So an annoyance that Common Lisp programmers can resolve for themselves within five minutes plagues Java programmers for years.

Peter Seibel, Practical Common Lisp.

So true. And that’s just for a language feature that everyone will agree on. God help you if you want language feature in Java that’s only useful to web developers.

How To Make A Relational Database Schemaless

Sat, 14 Jul 2012 00:00:00 GMT

Just in case this isn’t completely obvious:

sql

  CREATE TABLE some_collection (
    id   VARCHAR NOT NULL,
    data BLOB    NOT NULL
  );

There you go - you can serialize any key-value pair you like, without a schema. Heck, the values don’t even have to be of the same type.

You may or may not want to do this, but you can. My point is, while there are many good things to say about NoSQL databases, schemalessness isn’t the interesting one. Enforcing a schema is a feature. Not enforcing one is not.

Programmers, Your Text Editor Wasn't Designed For You

Wed, 06 Jun 2012 00:00:00 GMT

As a programmer, the text editor is the core tool in your world. Regardless of platform, language or environment, almost everything you write will go through your editor of choice. And I’m here to tell you folks: chances are, your text editor is the wrong tool for the job. Here’s why….

Most text editors are built to support a writer’s workflow. And a writer’s workflow looks like this: Start a new document, append lots of text, then later on, jump around a little & tweak:

The amount of jumping will vary depending on the writer, but the workflow is essentially insert-centric. A tiny bit of time is spent moving the cursor, the vast majority spent entering new text.

And that’s the model that pretty much every text editor out there is built on. Notepad & Word are built for writers, but so is the editor in Eclipse, Xcode, IDEA & every other IDE I’ve ever seen. The workflow is insert-centric, and any other features are just tacked onto a menu at the top:

But we’re not writers! We’re programmers. Our workflow is completely different. We spend very little time appending large chunks of new text. We spend most of it jumping between functions, adding a bit here, jumping there, deleting a few lines and moving them over to a different file. We leap around, chopping, changing, expanding the code all the day:

No editor designed for writers will ever support a workflow like that well. And (almost) every editor is either designed for writers, or an extension of one that was. Adding more menu items to a fundamentally wrong workflow doesn’t solve the problem. We need an editor that isn’t insert-centric. We need one that’s manipulation-centric. An editor that assumes very little of our time will be spent inserting en-masse, and that most of it will be spent jumping around, tweaking, expanding and refining. We need an editor that assumes multiple files at once is the norm. We need an editor that makes changing five scattered sentences at least as easy as entering one new paragraph. The writer’s editor will not achieve this, no matter how many menu items it has.

If your editor sends you to a mouse to select a paragraph, it’s not good enough. If it expects you to edit one file at a time, it’s not good enough. If jumping between two related functions in two separate files requires a mouse, it’s not good enough. In short, if your editor doesn’t make movement a first-class citizen, it’s not good enough.

There is at least one solution to this problem. There is at least one editor that’s designed, from the ground up, for a programmer’s workflow. But I’m trying to avoid mentioning it, because this post isn’t about converting people to my favourite editor. It’s about the fact that most editors aren’t designed for us, and that we need a solution.

There are two and a half types of editor. Editors designed for writers. Editors designed for writers, with dozens of functions tacked onto the menu system. And editors actually designed for programmers. Whichever one you pick, please make sure you’ve picked it deliberately.

PS: Okay, my chosen editor is Vim. It addresses a programmer’s workflow beautifully. But this isn’t a pro-Vim rant. It’s a workflow plea. You don’t have to pick my solution, but I hope you see the problem & go looking for a solution.

PPS: Some of my colleagues think I’m anti-IDE. I’m not. I just keep coming back to the editor. Having an Integrated Development Environment is nice. Integration is a lovely thing. But having an editor designed for programming is essential. I’ll give up integration for that, every time.

A Handy Clojure Macro For Development vs. Production

Sat, 31 Mar 2012 00:00:00 GMT

Here’s a quick little Clojure¹ macro I’ve started using.

The Book Feed is a new website I’ve put together where you can write short book reviews, and optionally post them to Facebook. When testing, I don’t want the post-to-Facebook code to run while I’m developing, or my Facebook account would fill up with test-data. It should really only run in production.

That’s straightforward to code in any language. I write some kind of isLive() function, then wrap the relevant code in if ( isLive() ) {.... }. Easy, right?

Well, in Clojure we can go one better. Take a look at this code:

clojure

  (defn is-live? [] ...)
  (defmacro if-live [function & args]
    `(if (is-live?)
       (~function ~@args)
       (println "DEVELOPMENT Skipping:" '~function ~@args)))

…then later in my code I replace:

clojure

(post-to-facebook data1 data2 ...)

…with:

clojure

  (if-live post-to-facebook data1 data2 ...)

At first glance this looks like syntactic sugar. It’s super-short, but is it really any better than what we’d say in something like Java?

java

  if (isLive()) {
      postToFacebook(data1, data2, ...);
  } else {
      log.debug("DEVT Skipping: " + data1 + " " + data2 ...);
  }

Well, yes. This macro is significantly better, and not just because of all the typing it saves. The if-live macro has a magic property we’d struggle to get in other languages: Instead of just skipping the code, or printing a generic debugging message, it prints the exact code and arguments it would execute were we in production. In other languages we’re stuck with the tedious problem of keeping the log message & the call in sync with each other².

But because Clojure code can be treated as regular data, it’s trivial to print the call instead of executing it. With no effort, I can see exactly what the call would send to Facebook, without worrying about keeping the debug message in sync with the function call.

Yay macros.

Everything in this post is applicable to LISPs in general. ↩
I can think of a few workarounds for a couple of languages, but nothing that comes close to the elegance of this macro. Nothing that says, “Add these 8 characters to any function and it just works.” ↩

It's Complicated

Tue, 20 Mar 2012 00:00:00 GMT

This evening I saw a man who was incredibly buff. Take the fittest I’ll ever be in my life, and he was far, far fitter at 60. He was so toned he could easily have been a world-class yoga teacher. He wasn’t a world-class yoga teacher. He was a world-class musician. He was Sting.

Sometimes I feel inadequate.

I pray to God.

I say, “God, sometimes I feel inadequate.”

“That’s okay,” says God. “That’s why I created Tom Jones. He’s being awesome enough for everyone.”

He’s right.

“Have a Gin & Tonic,” says God.

He’s right again.

Letters to Juliet, A Film Review

Mon, 05 Mar 2012 00:00:00 GMT

Just in case you’re tempted to watch this film, let me save you the bother and tell you what happens:

Gael Garcia Bernal plays, ‘The only person in the film with an ounce of charisma.’ He is engaged to a blonde personality vacuum with bad dress sense. They fly to Verona. He runs off to do interesting things, we are forced to stay in the hotel and watch her pout.

After an hour of tedium, Ms Airhead finds the man she’s been looking for. She is overjoyed. His soulless husk is as badly dressed as hers. He is overjoyed too. He hasn’t had any real work since he left Home & Away.

Vanessa Regrave picks up a cheque, a free holiday in Italy, and a kiss from Italy’s answer to Tom Jones. Given how little screen time is given to septuagenarian nookie these days, she chalks the film up as a personal victory.

The End.

How Do I Pick The Wrong Programming Language?

Sun, 12 Feb 2012 00:00:00 GMT

A little thought experiment has been running around in my head for the past couple of days. I thought I’d share it with you.

It’s very hard to rank programming languages. We get a sense that some are good & some are bad, but the more experience you have the more caveats you have to allow: X is good (but it’s hard to find good X programmers); Y is fairly bad (but it’s good for data-wrangling), etc. Choosing the right language for the job is an art. But, as a novel way of looking at the problem, here’s my thought experiment:

There’s a 5-year project coming up. You don’t know what it is. You don’t know how many programmers will be working on it, or how good they’ll be. You know nothing, except that it must be fairly large because it’s a 5-year project.

You have one choice, and one objective. You get to choose the programming language for the project. And your objective is to guarantee that at the end of five years, the code-base is a sprawling, hideous, unmaintainable nightmare.

Which language do you pick?

You’re not allowed to cheat. You have to pick a language you’d use for a real project. You can’t choose assembly language, and you certainly can’t choose Brainfuck.

I think this is an interesting way of thinking about the question for two reasons. First, whilst we spend so much of our time as programmers maintaining code, we spend surprisingly little time talking about maintenance. Or thinking about it. And second, the question isn’t, “Which is the least maintainable language?”, but rather, “Which languages tends to encourage the least maintainable behaviour?”

I think I’ve made my choice. I think I’m picking Perl.

Now don’t get me wrong, I like Perl a lot. I seem to like Perl more than most people I meet. It’s particularly strong at text-handling, and makes a good choice when you’re forced to resort to screen-scraping semi-structured data.

But Perl does not lend itself well to large, clear code bases. It has a good object model, but it doesn’t enforce it. It doesn’t even really recommend it. It’s just there in case anyone’s interested.

It has something like a recommended project structure, but unless you’re producing modules for CPAN or you’ve stumbled upon the object model, you’ll probably miss it.

And worst of all there’s very little unity of approach. Ask five Java programmers to write a simple program, and by-and-large all five of them will come up with a similar answer. Ask five Perl programmers, and they’ll all have a different preferred approach. And they’ll probably all take pride in how novel their solution is. That’s all good fun, but having five ways to do the same simple thing doesn’t lend itself to large scale coding.

I could pick other languages. The worst coding crimes I’ve ever seen have all been in PHP. But for an absolute lack of unity, I’m picking Perl.

What would you pick?

Naming Things

Sun, 06 Nov 2011 00:00:00 GMT

Yesterday we went to a toy store called Fun Junction. I think this is a terrible name. To me, Fun Junction is the point where Fun & No Fun meet…

Because the Kids are Lighting Fireworks Already

Thu, 03 Nov 2011 00:00:00 GMT

Remember, remember the 3rd of November, Blowing things up is a lark, I don’t care a jot about gunpowder plot, I’m off to light fires in the park…

Internet Explorer to Enter the Deadpool

Sun, 04 Sep 2011 00:00:00 GMT

In a surprise move, Microsoft have today announced that they’re getting out of the browser game. “It’s time to call it a day,” said a Microsoft spokesperson. “We’ve been working on Internet Explorer for over fifteen years now, and we still can’t render basic layouts reliably.”

“‘Does it work in IE yet?’ has become a stock question for web developers around the globe, and it’s getting embarrassing.”

“It’s time we left the browser game to the professionals, and focussed on what we’re good at: rearranging the menu bar in Word every 6 months.”

How To Get X11 Working On An EC2 AMI

Wed, 17 Aug 2011 00:00:00 GMT

It’s actually pretty easy, but worth noting:

ssh ec2-user@ec2machine
sudo yum install xauth xeyes
exit

ssh -X ec2-user@ec2machine
xeyes

In short, the magic’s in installing Xauth and then re-logging in with the ssh -X flag.

And I admit, xeyes isn’t strictly necessary, but I find it’s a nice, lightweight confirmation that both display and mouse are working as expected.

Diagnosing Problems On The Web - A Rule Of Thumb

Wed, 03 Aug 2011 00:00:00 GMT

My mother is asking for technical support ‘for a friend’. She’s reminded me of this quick diagnostic tip that’s worth noting:

plaintext

Geek: What browser are you using?
Patient: What's a browser?

They’re using Internet Explorer. Someone might have installed a better browser for them, but if they had someone like that in their lives, they wouldn’t turn to you for tech support.

Vim vs. Emacs - The War Is Over

Mon, 01 Aug 2011 00:00:00 GMT

Vim is the greatest text editor in the world. I have strong opinions about why this is so, and why Eclipse & co.’s editors are inadequate by design, but that’s for another day. No, this is just a quick post to say the Vim vs. Emacs wars are over. I’m a Vim user, but if you like Emacs that’s just fine with me. We mustn’t fight. We must save our energy. There are people out there writing code in Notepad FFS…

Star Pelican

Sat, 30 Jul 2011 00:00:00 GMT

I think I’m blogging again. Certainly the current frequency of posts would imply I am. And if I’m blogging again, I’m doing it because there are thoughts that are too public for Facebook & too long for Twitter. Thoughts like, “Google’s webapps are so good that they show webapps are doomed.” And non-technical thoughts too, like, “In the future, Pelicans will be armed.”

Native Apps vs Web Apps

Fri, 29 Jul 2011 00:00:00 GMT

An interesting question at the moment is, should I build native iOS & Android apps, or build one webapp to rule them all? There are pros and cons to each. Most of the arguments are obvious and well-trodden, but here’s one point that’s important and not being said: Google.

Google have put their stake in the ground and are investing heavily in the webapp route. Their Gmail & Google+ clients are are easily the most advanced mobile webapps in the game. And do you know what? They’re still a bit rubbish. They’re a bit slow. They’re a bit flakey. They all have the, “This looked really good in the lab”, feel to them.

Google’s webapps are “impressive, for a webapp”, which is like saying, “he’s cool, for a geek”. A qualified compliment isn’t much of a compliment at all. I really want them to succeed, and they’re doing an amazing job, but for all their effort I still go to the app store to find the real app.

If Google, with all their talent and resources, can push the possibilities of webapps and we still need to offer excuses, maybe that says it’s not the way to go…

London 2012, One Year To Go

Wed, 27 Jul 2011 00:00:00 GMT

Wow! I just found out I’ve been elected as Chairman Of The Olympic Committee For Getting People Who Don’t Give A Stuff About The Olympics To Give A Stuff About The Olympics.

My first campaign will be, “Come To London in 2012: The Aquarium Will Still Be Ace.”

I’ll follow up with my second campaign, “Enjoyed The Aquarium? You Could Watch Some Humans Swim Next, I Suppose.”

OSX Apps I Can't Live Without

Wed, 27 Jul 2011 00:00:00 GMT

Frabjous day - I have a new laptop. I’d been waiting for ages for the next model of MacBook Air to be released, so I pounced at the weekend. And I think that makes it time for the geekblogger’s favourite: the “OSX Apps I Install First” list. I’ll avoid the obvious ones like Chrome, Spotify, Dropbox & XCode, and skip straight to the meat. Hopefully there’s something here you haven’t heard of before that you can try…

Alfred

Do you like Terminal, Spotlight or Chrome’s universal search feature? Then Alfred is for you. It’s a fast app-launcher, file-finder & web-searcher that’s absolutely the first thing I put on every Mac. More flexible than Spotlight, faster than QSB and currently thriving (unlike the almost-zombie Quicksilver), I can’t recommend it enough.
Backblaze

We were burgled last year. My laptop - my working life - was taken. I had Time Machine backups, and I got lucky (they didn’t steal my external hard drive) so I got it all back. But never again will I risk my music & my photos¹ on the laziness of thieves.

In the weeks following the burglary I tried several off-site backup solutions². BackBlaze is the bee’s knees. It’s written by ex-Apple employees, and it shows. Actually, it hides. It just sits quietly in the background ensuring that my data’s always backed up to a second country. It also does second duty as a poor man’s version control system, and third duty as a find-if-stolen service. But it’s absolutely, 100% worth the price of a latte a month. Get it.
Pixelmator

I’m not a graphic designer. I’ll never be a graphic designer. So I can’t tell you exactly how Pixelmator compares to Photoshop. I can tell you that it does more than everything I need, it’s about 95% cheaper than Photoshop, and it’s one of the most polished pieces of software I’ve ever used.

My ‘hobby’ iPad app, Time for Tea, was designed in Pixelmator and I think the result is pretty good (for a non-graphic designer):

(That’s the current peak of my design skills. Any shortcomings are mine, not Pixelmator’s.)

Of course all my work is backed up again to remote git repositories, but you guessed that, right? ↩
Mozy sucks, by the way. ↩

Kris Jenkins' Blog

Why fsyncing a File Isn't Safe Enough

Footnotes

Vibe Management

The AI Bubble Will And Won't Burst

The Shelf Life of an AI Study Is About Six Months

Footnotes

Making Your Agent See What It's Missing

Configuring Neovim & Nix to use Java's LSP Server

Neovim’s LSP Config

Nix’s flake.nix

Gradle’s build.gradle.kts

Delegated Coding Has A Bright Future

Tustain's Law

Why We Podcast

Let’s Start With The Business Case

A Tech Company Must Talk To Developers (Not At Them)

What’s In It For The Listener?

What’s In It For The Guest?

What’s It Like Behind The Mic?

Footnotes

Setting Up A kcat Config

Example kcat Config Files

Supporting Schema Registry

MicroServices - Check Size Before Ordering

Emacs Feature Branch Tip

Footnotes

There's Immutable And Then There's Immutable

Footnotes

Type Bombs in Elm

The Hole Truth

One More Example

Half A Hole Is As Good As A Whole One

Footnotes

Functional Mumbo Jumbo - ADTs

Footnotes

How Elm Slays a UI Antipattern

The Problem

To Elm!

The Solution

The Leveling-Up

The Download

Clojure-Fu with Juxt

The Recap

The Trick

Footnotes

How I Structure Elm Apps

Startup

src/App.elm

The Featureful Four

.../feature/Types.elm

.../feature/State.elm

.../feature/View.elm

.../feature/Rest.elm

Wrap Up

Caveats

Git for Static Sites

Publish on Github

Orphan Branches

Worktrees

To Recap

Why?

How

What Is Functional Programming?

Side-Effects are the Complexity Iceberg

Isn’t This What Encapsulation’s About?

Are Side-Effects Bad?

We Can Always Surface Side-Effects

Cons

Pros

What is a ‘Pure Function’?

What is ‘Functional Programming’?

What is a ‘Functional Programming Language’?

Is That It?

Acknowledgments

Footnotes

Which Programming Languages Are Functional?

Pot Shots Across the Programming Landscape

Functional Programming is not…

It’s not map and reduce

Nix’s `flake.nix`

Gradle’s `build.gradle.kts`

`src/App.elm`

`.../feature/Types.elm`

`.../feature/State.elm`

`.../feature/View.elm`

`.../feature/Rest.elm`

It’s not `map` and `reduce`

Decode one `Place`

Decode the `List` of `Places`