Media upload errors with Mastodon instance

brianrlawson

Hi Opalstack Support - I'll preface this with the fact that I'm a noob and not nearly qualified to be messing with stuff, but I wanted to give it my best shot before turning to Support for help. I apologize for the long post but I'm including a lot of details to help walk through my process, findings, and to explain why I finally gave up and bothered y'all.

I'm having a problem uploading animated gifs and videos to my Mastodon instance at https://hoosier.social. Jpegs, PNGs, and other still images work just fine regardless of file size. You can see some of my testing at: https://hoosier.social/@hoosiersocial. Two different errors/scenarios below.

Error #1: When I try uploading a very small animated gif (I've tried as small as 51KB) I get an error. The Mastodon UI error says "500 Error processing thumbnail for uploaded media." Example log file entries that might be related to this:

nginx_access.log

<IP Address Removed> - - [26/Dec/2022:20:45:05 +0000] "POST /api/v2/media HTTP/1.1" 500 68 "https://hoosier.social/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36 Edg/108.0.1462.54"

nginx_error.log

2022/12/26 20:45:04 [error] 234113#234113: *74577 connect() to unix://home/brianrlawson/apps/hoosiersocial/mastodon/tmp/sockets/streaming.sock failed (111: Connection refused) while connecting to upstream, client: 127.0.0.1, server: hoosier.social, request: "GET /api/v1/streaming/? HTTP/1.1", upstream: "http://unix://home/brianrlawson/apps/hoosiersocial/mastodon/tmp/sockets/streaming.sock:/api/v1/streaming/?", host: "hoosier.social"

I think that the problem here is with a service not starting up. When I do a restart, I'll get a message saying "streaming: ERROR (spawn error)" in the SSH console. The streaming.log says "ERR! Could not start server, the port or socket is in use."

Error #2: Maybe related, or maybe something different, is a 413 error I get when trying to upload a larger gif/video file (e.g., 2MB).

The Mastodon UI error message simply says "413"...that's all. Example log file entries that might be related to this:

nginx_access.log

<IP Address Removed> - - [26/Dec/2022:20:59:20 +0000] "POST /api/v2/media HTTP/1.1" 413 585 "https://hoosier.social/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36 Edg/108.0.1462.54"

nginx_error.log

2022/12/26 20:59:20 [error] 234113#234113: *75297 client intended to send too large body: 2035024 bytes, client: 127.0.0.1, server: hoosier.social, request: "POST /api/v2/media HTTP/1.1", host: "hoosier.social", referrer: "https://hoosier.social/"

This entry is interesting because it says "client intended to send too large body" I did some Googling for how to fix this and can confirm that my mastodon/app/models/media_attachment.rb has IMAGE_LIMIT=10.megabytes. Also, I followed advice from https://www.tecmint.com/limit-file-upload-size-in-nginx/ and edited the nginx.conf file. I set client_max_body_size 40M; in the "http {" section and restarted. It didn't work, so I went back in to that file and commented that back out and restarted again, which is how it sits now. Editing these files started to push beyond my comfort limits, so I decided to reach out and ask for help from the experts. Again, sorry for the long.

Thanks,
Brian

sean

brianrlawson there are 2 things going on here.

The 413 error is happening because our installer wasn't setting some necessary parameters in the backend nginx config. The installer has been updated, but to correct this on an existing installation you'll need to add the following lines to the app's nginx/nginx.conf, just below the listen directive in the server block, then restart the app:

keepalive_timeout 70;
sendfile on;
client_max_body_size 80m;

"500 Error processing thumbnail for uploaded media" is happening because ffmpeg isn't installed on your server. We hope to have ffmpeg rolled out on all servers soon but for an immediate workaround you can install a local copy into your app directory with the following commands:

cd ~/apps/name_of_app
wget https://johnvansickle.com/ffmpeg/releases/ffmpeg-release-amd64-static.tar.xz
tar -xf ffmpeg-release-amd64-static.tar.xz
mv ffmpeg-*-amd64-static/{ffmpeg,ffprobe,qt-faststart} ./mastodon/bin/
rm -rf ffmpeg-*

Since you've opened a ticket I've already done this for your app.

nick

Hi Brian,

When you say you get the 500 error, is that when you try uploading the gif as an attachment after saving it to your local device? It also sounds like your restart script is "stuck" and can't restart the app properly since it thinks it is still running. So I restarted your Mastodon instance manually for you.

As far as I understand it, the correct method is to right-click save any GIF to your local device that you want to share, then upload it as an attachment. Mastodon will then "optimize" it by converting the GIF to a MP4 file that should "autoplay" if you have the "Auto-play animated GIFs" setting set in the appearance preferences section of your profile.

NOTE: I will double check the veracity of the above to confirm if there are other methods.

UPDATE: It looks like the upload process on mobile clients is a bit easier, as you can simply copy/paste animated GIFs directly into posts.

UPDATE2: We're reviewing the installer config to see if changes can be made to optimize GIF integration.

If the errors still occur on uploads, please submit a ticket so we can take a closer look at your setup.

brianrlawson

nick Hi Nick, thanks for the quick reply. Yes, both errors (500 and 413) happen when I try to upload the file from my local device when creating a Mastodon post. In the compose box, there is a little paperclip to add media. I click that and browse to the file. After the file explorer closes (i.e. upon upload attempt, even before clicking "Publish" to submit the post) is when the Mastodon UI error message pops up. Happens from any device, but right now I'm trying from Windows on both Chrome and Edge...same result. Happens from my iPhone too (metatext app, Chrome, Safari).

Yup, I agree with your understanding re: optimizing to MP4. I've read the same thing when I've tried to track this down. To me, it seems like it isn't even getting that far in the process.

Another thing that is probably pretty important to add is...I can't play embedded stuff reliably either. As an example, if I view this post on its original site it plays fine: https://mstdn.social/@pollak/109571783072803153.

However, when I view it within my timeline on hoosier.social, I can see a black rectangle where the video would go and I can see the video controls, but it won't play. Not on any device...this Windows computer (Chrome or Edge) or my iPhone (metatext app, Chrome, or Safari browser). Reliably plans when I view it hosted from mstdn.social, but won't play on hoosier.social on any device or under any account. This happens with any video/animated content from anyone that I follow.

I would be happy to create an account on hoosier.social so that someone can log in and see things 1st hand if that would help.

Again, I'm kind of a noob about all of this stuff, but I'm guessing that when my instance pulls posts that I'm following into my Home or Federated timeline, it is also pulling media? And, it is not getting encoded or streaming properly, too.

You suggested that I should submit a ticket if the error still happens on uploads, so I did that through the my.opalstack.com site. I included a link to this forum post as reference.

brianrlawson

nick Just acknowledging that I saw your Update and Update2. They showed up after I posted my reply. Yup, I've tried on mobile (client and browser) as well as multiple desktop browsers. My reply added some additional info that my be relevant re: failure to play back other user's posts, too...not just issues with my own.

nick

brianrlawson This is just to let your know that we're looking into a solution for this. We thank you for your patience!

johns

Howdy,

This line in ~/logs/apps/APPNAME/puma.log is the error we need to focus on,

[c29b9213-1226-4dd0-a2f0-17ecab24ab61] method=POST path=/api/v2/media format=html controller=Api::V2::MediaController action=create status=500 duration=30.31 view=0.84 db=0.70

However this is not enough verbosity really, please increase verbosity to debug,

https://docs.joinmastodon.org/admin/config/#rails_log_level

Then tail the log at the specified location and upload a gif. Or reply here letting us know you made this change, or if it is OK if we do. Once we know the full error with debug verbosity we should know why.

brianrlawson

johns Yes, I’m fine with you guys making that logging level change on my behalf. Thanks for your help with this.

brianrlawson

sean Woot! That did it. I was sniffing around the fix to the 413 error earlier in the thread and had tried setting max body size to 40 MB, but it obviously wouldn't have worked if ffmpeg wasn't installed in the first place. I wouldn't have been able to suss that one out. Thanks again for your persistence and expertise.

On last, low priority question. Since ffmpeg is installed manually, will I need to do anything special/different when other things are upgraded on a managed server? I suspect that you guys use some sort of scripting or Salt automation or something to patch. Will this one-off still get touched during those since it may not fit the ongoing, future pattern?

sean

brianrlawson the ffmpeg that I set up for you is a static build that shouldn't be impacted by routine system updates that we perform. If you want to upgrade it yourself you can do so by repeating the commands in my previous reply.

Once we have a system-wide ffmpeg installation we'll maintain that like we do everything else.

horvath

sean

My instance is doing it too. (Upload failed 413) from iOS.

Do I understand that a system wide fix is planned? (And if so, any known timeframe)

sean

horvath we've already implemented the fix for 413 errors as of 4 days ago, but the fix is in the installer itself.

If you have an existing instance installed earlier than that and are seeing the 413 errors then you'll need to adjust your Nginx config as described above and then restart your app.

If your instance was installed using the new installer and you're still seeing the 413 errors then try raising client_max_body_size in the Nginx config to a higher value. The default is 80m which is 80 megabytes.

Note that the Mastodon software limits image uploads to 10MB and video uploads to 40MB.

Either way be sure to restart your instance after changing your config.

horvath

sean

Perfect... that got rid of the 413!

sean

ffmpeg is now installed systemwide on all Opalstack servers.