Over the past few weeks, some of you noticed that certain older articles were incomplete or wouldn’t download. We tracked down the cause, stopped the problem, and are now starting the restoration process. Here’s what happened.

The short version

One of our storage servers ran into a software bug when its disks were near full. Most of what it stored was fine. For some articles, though, the data was written to disk, but the location reported back to our system didn’t match where it actually landed. When someone requested those articles, our system looked in the wrong place, making them appear incomplete or unavailable. We’re now moving the intact data to healthy storage and beginning work to restore access to the affected articles.

What went wrong

Our servers use widely used disk-management software called btrfs, which compresses data to save space. The version on the affected server had a known bug that appeared when its disks were near full.

For some writes, the location the software reported didn’t match the location where the data was stored. Think of putting a box on one shelf but recording it as being on another: the box is there, but looking where the record says it should be won’t find it. That’s what happened when our system tried to retrieve the affected articles.

Newer versions of the software fix this issue, but the fix never reached the version we were running.

Who was affected

Only one storage server was involved, and only articles stored on that server were affected. We checked every storage server in our network, and no others were affected.

What we’re doing

• Stopped it. That server no longer accepts new data, so the problem can’t recur there.
• Moving the intact data to safety. We’re checking the data on the affected disks and moving all the intact data to healthy storage. That covers the large majority of what those disks hold.
• Starting restoration. We’re beginning the process of restoring access to the affected articles.
• Checked everything else. Every other storage server is healthy.

Making sure it doesn’t happen again

• Updating software. We’re moving our storage servers to a version where this bug is fixed.
• Randomly checking saved articles. Our storage software now checks randomly selected saved articles to confirm they can be retrieved correctly. If a check fails, it alerts our team right away.

Thank you

Reliable storage is the core of what we do, and we’re sorry for the trouble. We’ll keep you posted as restoration progresses.