Migration seems to have been successful. We are checking for any signs of remaining issues, but it looks like that's a wrap for tonight.

Show thread

Maintenance update: it looks like we haven't had the downtime we're concerned about, but we're still getting the "pending database migrations" error message. Reviewing log files.

Show thread

Starting the server maintenance for Weirder Earth - updates as needed.

Sidekiq has now cleared the initial backlog and is handling additional batches well as they come in. There may be continued oddities for a while, but everything seems to be working and you should be able to use the instance as normal. Let us know about any issues you see.

Show thread

From what we can see, it looks like we've dealt with the issues causing Sidekiq jobs to fail. We will have to monitor the situation to confirm that the backlog is clearing properly, and it will be some time before the instance is fully caught up, but you should be seeing your home timelines filling again.

Show thread

The instance is up, but the Sidekiq queues don't look healthy yet - expect delays on timelines, notifications, and so on. You may have to log back in to get access to your account. Further updates as we know more.

Show thread

We have confirmed that the database is present, suggesting no data was lost, but we are having to manually modify configuration files to point to it. Working on that now.

Show thread

Hitting some disturbing snags - during the update process, we attempted a database migration that doesn't seem to have happened properly. We're digging into the system to see if we can recover the state pre-update; in the worst case, downtime will be extended and/or we will have to revert to an old snapshot.

(deleted and reposted for threading purposes)

Show thread

Weirder Earth instance update is in progress! Expect lag spikes and downtimes for the next bit - updates as things progress.

Rage.love

Generalist Hometown instance with a strong focus on community standards. No TERF, no SWERF, no Nazi, no Centrist.