A couple of days ago I removed Whitenoise. Since my site uses Cloudflare, Whitenoise wasn't really adding much. I can't say I notice a difference in RAM or CPU usage, but haven't run any thorough benchmarks or anything. The website runs the same, fast as ever. One big plus is that the deployment process is much faster, as Whitenoise was post-processing all static files, which took a few minutes. Often the site would go down at this time. Django's normal "collectstatic" command only takes a few seconds to run.
As far as minification goes, I was already using django-compressor for that. It's not Whitenoise that does this (my bad), but the two apps seem to be working tightly together as they both process static files and complement each other. However, django-compressor works fine without Whitenoise (and vice-versa, I'm sure).