pecus

  • Joined Oct 24, 2020
  • Thanks Peter, I had the feeling this was the case.
    I know Cloudflare can rewrite Link headers (which can be generated at the application level) as 103 Early Hint, but moving an entire domain to Cloudflare might not be for everyone.

  • I'd like to use early hints on some pages served on an apache+php configuration. Early hints work similarly to HTTP2 push, and allow a browser to initiate download of resources early, while the actual HTML is still being generated or downloaded.

    Is this possible with the current configuration? My understanding is that with the nginx proxy acts as SSL terminator and HTTP2 protocol wrapper, the communication with the apache server is on HTTP which would not allow configuring and sending the headers. Are there workarounds short of having a cloud worker handle traffic and rewrite responses?

    • sean this should be clarified in the README that is created by the installer. It's easy to assume the project runs optimized, and while the assumption has some ground, it's always best to clarify the mode the server runs (dev or production).

      I'm closing the PR but I'd really like this behavior to be documented.

    • We identified an issue with the installer. It currently starts Nextjs in development mode, which is prone to memory leaks and very slow (dev mode compiles pages at runtime).

      I opened a PR to ensure NextJs is started in production mode (see https://github.com/opalstack/installers/pull/20). Please note production mode requires a build (npm|yarn|pnpm|bun build).

      • sean replied to this.
      • You can save time by using a precompiled build, generously made available unofficially (and guarantee to work — recent NodeJs versions do not build on Centos 7 due to missing libraries and/or need to customize includes) by a group of volunteers at https://unofficial-builds.nodejs.org/

        For instance, the latest (as of today) Node 20 LTS that you can install on Opalstack (shared hosting) is node-v20.13.1-linux-x64-glibc-217.tar.gz. If you try to build this version from Node's official source, it will fail due to dependencies not up to date on glibc.

        cd ~/apps/name_of_app
        wget https://unofficial-builds.nodejs.org/download/release/v20.13.1/node-v20.13.1-linux-x64-glibc-217.tar.gz
        tar zxf node-v20.13.1-linux-x64-glibc-217.tar.gz
        mv ~/opt/node ~/opt/node.old
        cp -r node-v20.13.1-linux-x64-glibc-217 ~/opt/node
        node -v

        it should not be necessary to restart the shell, and by using the default path, node should be available and first in the user PATH.

        Hope I'll save you people (and your CPU!) some time ;-)

      • The last tidbit of this article is gold (upcoming AlmaLinux v9 servers with managed MySQL).
        Any ETA on this?

        Incidentally, CraftCMS v5 has changed its preferred database from MariaDB to MySQL and might eventually drop it due to its divergent implementation of InnoDB (see https://craftcms.com/docs/5.x/requirements.html), making it a high priority feature for us. Looking forward to it!

        • This single step solved a massive performance issue on one of our sites, so I encourage anyone struggling with db performance inconsistencies to setup a private instance.

          One piece of advice: this instance will not be backed up by Opalstack daily process, so you need to take care of it yourself (we have a daily cron job that dumps the db, gzips it and rotates it every week)

          • sean replied to this.
          • I think I was approaching this the wrong way. I can configure GuzzleHttp with a proxy, and I can setup a nginx proxy on opalstack and configure it to reverse proxy my app remapping traffic from the CDN domain to the local opalstacked.com domain. I'll try this way (I think most of the work is already documented here https://community.opalstack.com/d/464-howto-run-a-private-php-fpm-stack-on-nginx-or-apache-on-opalstack )

            EDIT: it's a bit more complicated than it seemed, due to this being a forward proxy and needing a MitM approach (thus a set of certificates to handle SSL traffic)

          • sean to specify, I need a script running on opalstack to resolve the domain locally, avoding the CDN and short-circuiting to the server. I cannot replace the external domain with the opalstacked.com alias.
            The DNS is managed externally and uses a CNAME record for the domain (so that it can point to the CDN).

            I would bet that the best solution would be to go with the hosts file, but this needs to be done on the Opalstack shared host, and I believe I am not allowed to.

            • sean replied to this.
            • I have a weird case where I am making http calls to my own site to save a static version of the dynamically generated page. The site is behind a WAF (web application firewall) and I'm finding it hard to identify why traffic is blocked, despite having whitelisted the server IP and the header that the Guzzle HTTP client appends when making requests.

              An alternative would be to resolve the domain locally, so that instead of pointing to the CDN/WAF, it is resolved to the local Opalstack server. Due to the app configuration it is difficult to pass an alias to the HTTP client (e.g. the opalstacked.com domain for this site).

              Is it possible to add a custom route or dns resolution on a shared hosting so that domain.tld points to the IP address of opalXX.opalstack.com?

              TIA

              • sean replied to this.
              • makes sense, but you should probably customize the user agent to mimic the RFC for the ACME protocol. anything with acme inside would probably do (and, as a side effect probably be picked up as an allowed client by bot-screening waf rules)

                • I'm reopening (so to speak) this issue because recently we had a failure in renewing the certificate. It seems that one of the challenges is performed by a script that does not advertise itself as an acme client in the user agent string, instead using a generic user agent that gets blocked by our WAF. It is reported as python-requests/2.28.1.
                  We whitelisted any traffic to /.well-known/acme-challenge/* but are wondering if there was a change.

                  • sean replied to this.
                  • Today I noticed that using php74 in the CLI returns several messages that the default version of php (7.3.33) doesn't. Everything runs but CLI commands display a lot of unwanted messages.
                    It seems a systemwide configuration (we did not change our php.ini recently and it worked nicely up until a few weeks ago).

                    php74 -v
                    Mon Jan  9 09:54:31 2023 (254218): Debug Loading blacklist file:  '/etc/opt/remi/php74/php.d/opcache-default.blacklist'
                    PHP 7.4.30 (cli) (built: Jun  7 2022 08:38:19) ( NTS )
                    Copyright (c) The PHP Group
                    Zend Engine v3.4.0, Copyright (c) Zend Technologies
                        with the ionCube PHP Loader + ionCube24 v12.0.2, Copyright (c) 2002-2022, by ionCube Ltd.
                        with Zend OPcache v7.4.30, Copyright (c), by Zend Technologies

                    the unwanted message is

                    Mon Jan  9 09:54:31 2023 (254218): Debug Loading blacklist file:  '/etc/opt/remi/php74/php.d/opcache-default.blacklist'

                    When running scripts, these lines can add up in tens.
                    anyone with the same issue? can this be fixed?

                    • sean replied to this.
                    • sean SOLVED

                      StackPath WAF has a Known Bot rule that by default filters Let's Encrypt ACME challenge client. Enabling it allows the challenge to go through to origin (and additional edge rule that bypasses the cache for /.well-known/acme-challenge/* helps forwarding the traffic as a passive proxy)

                      • sean if manual certbot is available (with cron support to allow certificate renewal), I might be able to solve this by generating a certificate only for the site.user.opalstacked.com domain, skipping the user facing domain

                      • sean Certificate generation failed. I am not familiar with the challenge/response protocol of LE, but it might be blocked by the firewall ahead of the CDN.

                        Lets Encrypt failed to generate a certificate for your site: lightbox_prod. The site is served by a CDN which failed to serve the HTTP challenge.

                        I am working with StackPath support to verify if custom edge rules are needed in order to let the challenge through.

                        • sean replied to this.
                        • sean StackPath has provided the 2 anycast IP addresses they use for public facing sites:

                          151.139.128.11 and 151.139.128.10

                          These IPs don't call OpalStack in a CDN configuration but resolve the domains mapped to a site configuration.
                          Hope it helps (I opened a ticket to OS support referencing this issue)

                          • sean replied to this.
                          • sean unfortunately I still get

                            Unable to generate LetsEncrypt certificate. All site domains must either point to the site IP address (46.165.217.151) or use Opalstack DNS servers.

                            I verified that the IP address that my domain resolves to is not in the list of IPs that StackPath provides as Edge locations. This might explain why LE on OS does not work (but it does not explain why the IP is not in the block list)

                            • sean replied to this.
                            • sean Thanks Sean, I'll summarize but we can go private with a support ticket.

                              The CDN works (we use StackPath), but the setup on OpalStack is tricky.
                              The edge SSL certificate is for a domain that is not managed via OpalStack DNS. But we do need to add the domain to our website configuration so that host headers are correctly forwarded. The CMS uses the host header to serve content for the specific site.

                              When we have a opalstack.com domain (website.user.opalstacked.com) and an external domain (website.com), we cannot use the automatic generation of SSL certs via Let'sEncrypt.

                              • sean replied to this.
                              Mastodon