HOWTO upgrade Mastodon to v4.2.7 running on Ruby 3.2

sean

We recently updated our Mastodon installer to use Mastodon v4.2.5 running on Ruby v3.2 with Node.js v20. If you are running an earlier version of Mastodon and would like to update it to the latest version with the newer Ruby and Node.js then you can do so with the following steps.

Note that the procedure below uses placeholders which you'll need to substitute as follows:

app_user - the name of your Mastodon application's shell user. Your shell user names are available at: https://my.opalstack.com/apps/
app_name - the name of your Mastodon application. Your app names are available at: https://my.opalstack.com/apps/
db_user - the name of your database user, available in: ~/apps/app_name/mastodon/.env.production
db_name - the name of your database, available in: ~/apps/app_name/mastodon/.env.production

To proceed:

Run the following commands to make a backup (if you're on a VPS make sure you have sufficient space first by running "df -h /" on the server):

pg_dump -U db_user -f ~/apps/app_name/db.sql db_name
cp -Rp ~/apps/app_name ~/app_name_backup

Run the following command to disable your cron jobs:

crontab -l | sed -e 's/^/# /g' | crontab

Note: this will disable all of your cron jobs for your shell user. If you have cron jobs unrelated to your Mastodon app and you don't want them disabled then run crontab -e to edit your cron jobs manually.

Run the following commands to perform the upgrade, replacing app_user and app_name with your own.

Note that the app will be offline during the upgrade due to the memory constraints of your hosting plan. If you're certain you have a lot of spare RAM (for example if you are on a 4GB VPS or higher with at least 2GB of free RAM) and don't want the site to be offline then you can omit the ../stop command at your own risk.

cd ~/apps/app_name/
./stop
mkdir old
mv {node,restart,setenv,start,stop,supervisord.conf} ./old/
mkdir -p node/bin
scl enable nodejs20 -- corepack enable --install-directory=$PWD/node/bin
scl enable nodejs20 -- $PWD/node/bin/yarn set version classic
wget https://github.com/opalstack/installers/raw/master/core/mastodon/mastodon_scripts.tar.gz
tar zxf mastodon_scripts.tar.gz --strip 1
sed -i -e 's/SHELL_USER/app_user/g' {restart,setenv,start,stop,supervisord.conf}
sed -i -e 's/NAME_OF_APP/app_name/g' {restart,setenv,start,stop,supervisord.conf}
chmod u+x {restart,start,stop}
source setenv
cd mastodon
git checkout -- .
git fetch && git checkout v4.2.7
bundle config set jobs 4
bundle install
yarn install --frozen-lockfile
bundle exec rails assets:precompile

# run the next command ONLY if upgrading from <v4.2.0, otherwise proceed to the kill command
SKIP_POST_DEPLOYMENT_MIGRATIONS=true bundle exec rails db:migrate

kill $( cat tmp/pids/supervisord.pid )
# wait a few seconds
../restart

# run the next command ONLY if upgrading from <v4.2.0
bundle exec rails db:migrate

Run the following command to re-enable your cron jobs (or run crontab -e to edit your cron jobs manually):

crontab -l |  sed -e 's/^# //g' | crontab

Your backup will very likely use a lot of disk space. To remove it following the upgrade, run the following command:

rm -rf ~/app_name_backup

You're done!

db122

Thanks, I appreciate you keeping this up-to-date!

A few problems I had/things I noticed:

After upgrading, there were some strange dotfiles in the app dir that appear to be cruft from a macOS install. Particularly, ._restart, ._start etc. I'm guessing they are in the mastodon_scripts.tar.gz.

I wasn't upgrading from < 4.2.0 so this didn't matter for me, but I think there are two different steps where the db:migrate command could potentially be run -- it seems a bit confusing. Is that correct?

I saw the following errors when I ran the script:

tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemWhereFroms'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.provenance'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemWhereFroms'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemWhereFroms'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemWhereFroms'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.provenance'
tar: Ignoring unknown extended header keyword `LIBARCHIVE.xattr.com.apple.metadata:kMDItemWhereFroms'

and

Fetching https://github.com/jhawthorn/nsa.git
Git error: command `git reset --hard e020fcc3a54d993ab45b7194d89ab720296c111b` in directory
/home/[username]/apps/[appname]/mastodon/vendor/bundle/ruby/bundler/gems/nsa-e020fcc3a54d has failed.
fatal: Could not parse object 'e020fcc3a54d993ab45b7194d89ab720296c111b'.

and towards the end:

https://github.com/mastodon/rails-settings-cached.git (at v0.6.6-aliases-true@86328ef) is not yet checked out. Run `bundle install` first.

In the end, my instance did not come back up so I reverted to the backup. Some errors in the supervisord.log:

2024-02-05 15:41:00,896 INFO spawned: 'puma' with pid 5619
2024-02-05 15:41:00,897 INFO spawned: 'sidekiq' with pid 5620
2024-02-05 15:41:01,033 WARN exited: puma (exit status 127; not expected)
2024-02-05 15:41:01,107 INFO gave up: puma entered FATAL state, too many start retries too quickly
2024-02-05 15:41:01,112 WARN exited: sidekiq (exit status 127; not expected)
2024-02-05 15:41:02,113 INFO gave up: sidekiq entered FATAL state, too many start retries too quickly

and sidekiq.log:

bundler: command not found: sidekiq
Install missing gem executables with `bundle install`

Rerunning the script led to slightly different errors, but similar issues with the nsa package.

sean

db122 I think there are two different steps where the db:migrate command could potentially be run -- it seems a bit confusing. Is that correct?

That is per the upgrade instructions in the Mastodon 4.2.0 release notes. It's two steps, 1 for pre-deploy migrations and 1 for post-deploy migrations.

DB migrations are usually only needed on minor version releases, but you should always check the release notes to be sure.

db122 I saw the following errors when I ran the script:

I've updated the script tarball to remove the macOS junk.

Regarding the other errors, I tested the upgrade procedure by upgrading from a fresh Mastodon v4.1.0 install and did not run into any of those issues. What version are you upgrading from, and what version was it originally installed as?

db122

sean Got it, thanks. The upgrade was originally from whatever version the OpalStack install steps first supported. I think it was 4.0.2? But I'd since upgraded to 4.1.something and then up to 4.2.5 without updating node/ruby. So these updated steps I ran on a functioning 4.2.5 install. I haven't had much time to look at it since yesterday. I'll post back with more info when I can or maybe I'll just submit a support ticket and you could poke around when you have a chance.

db122

db122 Closing the loop here, I finally had time to iron out my upgrade woes. They were related to puma and sidekiq not starting due to some Ruby issues. There was some extra cruft hanging around in the local mastodon git repository clone directory. I cleaned up that git repo (git status, etc) and now the instance is running clean w/node 20 and ruby 3.2.2. Cool! Happy Friday.

mergy

Worked to get me going on the opalstack "standard" kind of setup there for mastodon in addition to the 4.2.6 update.