We took Pangea down for 25 minutes on 24 October

On Saturday 24 October, from about 09:20 to 09:46 GMT, our database stopped accepting connections. For those 25 minutes the Pangea app could not sign in or connect, and the account pages on the website would not load. We caused it. No data was lost.

What you would have seen

The app failed to sign in or start a new connection.
On the website, your dashboard, billing page and checkout returned errors. The public pages stayed up.
If you were already connected, your tunnel lives on the VPN server, which does not need our database to pass traffic, so it should have kept running.

What happened

We were shipping a privacy change, which is at the end of this post. Before we change data we take a backup and check that it restores. The check is what broke things.

It restored the backup into a scratch area of the live database, all in one go. Our backups used to be about 5 MB. This one was 71 MB, most of it website analytics. The likeliest explanation is that the database server, which is small, ran out of memory trying to take in 71 MB at once. Our provider's status page showed nothing wrong on their side.

We restarted the database at 09:46 GMT. The restore ran as one transaction, so the restart undid all of it. It left an empty scratch area behind, which we deleted.

What we changed

The backup check no longer touches the live database. It restores into a separate, throwaway database, and it refuses to run if pointed at the real one.

We also found that some error messages from our website's API said too much while the database was down. They included the query that failed, with the values from the request that caused it. That text only went back to whoever sent the request, but it shouldn't have gone anywhere. Now any error we didn't raise on purpose just says "Internal server error", and the details stay in our server logs.

The change we were shipping

When you rename a device, the name is now stored encrypted. Our default names, like "Swift Fox", were never personal, but one you type can be. From today, a copy of our database on its own doesn't show it. Our servers still decrypt the name to show it back to you, so this guards against a leaked copy of the data. It doesn't hide the name from us. Everything else we store is on our no-logs page.

Sorry. This one was entirely ours. If anything still looks wrong on your account, email support@pangeavpn.org.