anjanesh At the moment, Opalstack doesn't provide any platform-level protection specifically aimed at AI crawlers and other scraping bots.
A robots.txt file can help discourage well-behaved crawlers, but it is entirely voluntary and won't stop bots that ignore it.
If your site is experiencing abusive crawling or excessive bot traffic, our general recommendation is to place the site behind a reverse proxy/WAF service such as Cloudflare. Cloudflare offers several features that can help mitigate unwanted AI crawling and other automated traffic, including AI bot blocking, Bot Fight Mode, Browser Integrity Check, and AI Labyrinth. These features continue to evolve and can be enabled independently depending on how aggressive you want the protection to be.
Beyond that, application-level measures such as rate limiting, caching, and blocking abusive IPs or user agents can also help, although determined scrapers can often work around those techniques.
There are no immediate plans to add AI crawler protection directly to the Opalstack platform, but we're keeping an eye on developments in this area as the landscape is changing rapidly.
