Recently I learnt something interesting about writing files to disk. Something that connects my first ever low-level database PR with the reason you can’t pull a USB disk out of your laptop without ejecting it first.

As you may know, when you write a file on a computer it doesn’t actually get written to the disk straight away. In reality it gets written into some cache in memory, with a promise that the information will be permanently stored on disk “soon”. It’s the operating system’s job to fulfil that promise. To prioritise getting your data safely stored on disk along with the dozens of other tasks it’s trying to do at the same time. Usually that’s quite fast, but it depends on a lot of factors. In practice, writing to disk “soon” may mean milliseconds or minutes.

There is a risk to that though. What if you save your file, then the machine immediately crashes? Was the file saved to disk in time, or was it still only in memory? Does a crashed operating system mean lost data?

The answer depends on fsync. Now this part I’ve known for years - if you want to write a file to disk, and actually be certain it’s on the disk, you have to fsync it. You have to tell the operating system that you want your data written, and you want to wait until it’s actually guaranteed on-disk. To do that, you write the data then call fsync. When fsync returns, the OS has completed its promise and your data is durable.

Databases use fsync all the time to make sure data is permanent. It’s also why you can’t pull out a USB disk without pressing eject first. The eject button is the user-facing version of fsync - it makes sure that everything you think you wrote to the disk is actually physically stored before you disconnect it.

But here’s the thing I didn’t know about fsync1. If you call fsync on a file, it only makes the contents of the file permanent - not the things that file depends on. So imagine this scenario:

  1. Create a new file.
  2. Write some data to it.
  3. fsync the file.
  4. The computer crashes.

What happens? In the worst-case scenario you’ve lost your data, even though you fsync’d it! The file itself was written permanently, but the new directory entry pointing to the file was not. So the data’s on disk as promised, but there’s no way to access it. You can’t see the new file entry in the directory. A forensic specialist could track the data down, but you and I have lost the route in. And that’s the important thing I learnt: fsync doesn’t save all your work to disk. If you modified the parent directory by creating a new file, you have to fsync the parent too.

Doubtless the writers of the USB disk’s eject button knew this. I learnt it while I was looking through the code of SpacetimeDB. There’s a case where it’s building up the database’s write log, and it has to create a new segment file. The code correctly fsync’d the file, but not the parent directory. An unlucky crash could have led to data loss. Of course, it’s an easy fix - you just make the extra fsync call. I patched it and it got released this week in v2.9.0.

And that small-but-important improvement is my first truly low-level database contribution. Never have I patched so close to the hardware before. 😊

Footnotes

  1. Perhaps I should have known. Perhaps when I first learnt about fsync it worked differently. Certainly there are some filesystem configurations where a single fsync is enough. But no one wants to leave these things to chance. ↩