

So many of the AI crawlers don't use JS-enabled browsers so you need to access raw HTTP access logs to see the actual requests.
The problem is that most of this data is terminated at your CDN level and you might never know the true amount of referrals and crawlers.
For example, attached you can see both OpenAI crawler and referrals.