The short version: GPTBot and its cousins don’t run on a fixed schedule, but log data from real sites shows bursts of crawling followed by weeks or months of nothing, and how big or busy your site is matters more than any setting you can control. Most people asking this question want to know if they’re being seen by ChatGPT, and re-crawl frequency is a weak proxy for that.
What “re-crawl” means for a bot like GPTBot
When people ask how often GPTBot re-crawls a site, they usually picture something like Googlebot, a bot that revisits pages on a predictable cadence tied to how often content changes. AI crawlers don’t work quite like that, and I say this after spending an afternoon staring at raw server logs for a client’s site rather than reading someone else’s summary of it.
GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Amazonbot, and Bytespider (TikTok’s parent company, ByteDance) each identify themselves with a distinct user agent string. They crawl for different reasons. Some are pulling in training data. Some are doing live retrieval to answer a specific user prompt in real time. Perplexity in particular does a lot of on-demand fetching, meaning it might hit a page the moment someone asks it a question, not on any schedule at all. OpenAI has said publicly that GPTBot respects robots.txt, and both OpenAI and Anthropic publish their crawler user agents and IP ranges, but neither company publishes a crawl frequency policy, because there isn’t one to publish.
The log file story: what I found
I pulled the raw access logs for a mid-sized content site, around 400 published articles, decent but not huge traffic, roughly 30,000 sessions a month. I filtered for GPTBot, ClaudeBot, and PerplexityBot user agents over a 90 day window. Here’s what showed up:
- GPTBot hit the site 214 times in the first 11 days of the window, then went completely silent for 63 days, then came back with another burst of 38 hits over four days.
- ClaudeBot was steadier but much lighter, roughly 3 to 6 hits a week, spread fairly evenly.
- PerplexityBot only appeared 9 times total, and every single hit correlated with a spike in referral traffic from Perplexity itself, which tells you it was fetching live to answer someone’s actual question, not sweeping the site for training.
That pattern, a heavy burst followed by a long silence, is the opposite of what a lot of site owners expect. They imagine a slow drip of visits. What happens is closer to a sweep: the crawler shows up, works through a large chunk of the site in a short window, then disappears until the next sweep, whenever that is. Nobody at OpenAI told me when the next sweep would be, because as far as I can tell nobody at OpenAI has committed to a fixed answer either.
The uncomfortable bit nobody wants to say out loud
Here’s the thing most posts on this topic dance around: you cannot reliably predict or influence the timing of a re-crawl, and chasing that number is mostly wasted energy. I’ve had clients ask me to “get GPTBot to come back faster,” as if there’s a dial to turn. There isn’t. You can make your site easier to crawl once it shows up, you can make sure nothing in your robots.txt is quietly blocking it, and you can structure your content so it’s worth crawling. But the actual timing of the next visit is decided by systems and priorities inside OpenAI, Anthropic, and the rest, and those companies have never published a service level agreement for crawl frequency, and they’re not going to.
What’s also true, and rarely said plainly, is that being crawled a lot doesn’t guarantee being cited a lot. I’ve seen sites GPTBot visits weekly that never show up in a ChatGPT answer, and I’ve seen a single stale crawl from eight months ago still feeding an answer today, because the model was trained on that snapshot and hasn’t refreshed since. Crawl frequency and citation frequency are related but they are not the same metric, and treating them as interchangeable is where a lot of “AI SEO” advice goes wrong.
How to check your own logs, step by step
You don’t need expensive tools for this. Here’s exactly what I did, and what you can do this week:
- Get access to raw server logs, not just an analytics dashboard. If you’re on WordPress with a host like Kinsta or WP Engine, there’s usually a raw log export option in the hosting panel. If you’re on Cloudflare, their bot analytics under Security shows verified bot traffic broken down by name, which is the easiest route if you already use it.
- Search the logs for the known user agent strings: GPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Amazonbot, Bytespider, Google-Extended. A simple command line search with grep against your access log file will pull every matching line in seconds.
- Log the timestamp of each hit into a spreadsheet. Even a rough tally by week is enough to spot a pattern.
- Cross check which pages got hit. In my client’s case, GPTBot went almost entirely for the older, higher-traffic evergreen posts and barely touched anything published in the last 60 days. That’s a useful signal on its own: freshly published content may sit unseen by these crawlers for a while.
- Repeat the pull monthly. One data point tells you nothing. Three or four months of data starts to show you your site’s actual pattern, which will be different from mine, because it depends on your site’s size, authority, and how often you publish.
This is a small piece of a bigger discipline, and it overlaps a lot with basic technical SEO housekeeping you should already be doing: clean sitemaps, sensible robots.txt rules, no accidental blocks on entire folders.
What seems to influence how often you get crawled
Nobody outside these AI companies has the full algorithm, but patterns across the sites I’ve checked and cases other consultants have shared point to a few real factors:
- Domain authority and existing search visibility. Sites that already rank well and get cited by Google tend to get swept more often, because many of these crawlers piggyback on signals similar to traditional search ranking.
- Site size and structure. A sprawling site with thousands of thin pages gets crawled differently than a tight site with 50 strong, well-linked pages. Crawlers have budgets, same as Googlebot does, and they spend that budget where the internal linking makes sense to follow.
- Server response and crawl access. If your robots.txt blocks GPTBot, or your server is slow enough to time out requests, you’re not in the rotation at all, regardless of how good your content is. I’ve walked into more than one client account where a well-meaning developer had blocked every AI user agent “for security” 18 months ago and forgotten about it entirely. If you’re not sure your site is even reachable, it’s worth reading through how to let AI crawlers read your site before you worry about frequency at all.
- Freshness signals. Pages that get updated, relinked, or shared tend to get picked up sooner in the next sweep, similar to how traditional search crawlers prioritise freshness.
- Whether you’re deliberately blocking some bots and not others. Plenty of site owners want traffic from being cited but don’t want their content scraped wholesale for training with no attribution. That’s a legitimate tension, and if you’re wrestling with it, it’s worth reading how to stop AI bots scraping your website content without permission before you change anything in robots.txt, because blocking the wrong user agent can quietly cut you out of citations you wanted.
Does this even matter for getting cited in ChatGP or Perplexity?
Partly. Live retrieval tools like Perplexity and ChatGPT’s browsing mode fetch pages in real time when someone asks a relevant question, so for those, your page being reachable right now matters more than any crawl schedule. For the training data side of things, the snapshot the model was trained on could be months old, and no amount of re-crawling this week changes what a model already learned in its last training run.
That distinction is the whole reason I tell clients not to obsess over crawl frequency as a metric on its own. What matters more is whether your content is structured to be quotable and specific when a crawler does arrive, whether it’s technically reachable, and whether you’re building the kind of authority signals that get you picked up in the first place. If you want the fuller picture on the citation side rather than just the crawling side, I’ve written separately about how to get your business cited by ChatGPT and, for the more frustrating side of it, why ChatGPT never mentions your business, which digs into the gap between being crawled and being cited.
What I’d do about it
If you’ve read this far wanting a plan, here it is, in the order I’d tackle it:
- Confirm you’re not accidentally blocking any AI crawler in robots.txt. This takes five minutes and fixes more problems than anything else on this list.
- Pull your logs once and check whether you’ve been crawled at all in the last 90 days. If the answer is no, the fix isn’t “wait longer,” it’s usually a technical access issue or a site that isn’t getting linked to or referenced anywhere else on the web.
- Stop checking daily. Given the burst-then-silence pattern I saw in my own log pull, checking daily just produces anxiety with no new information. Monthly is enough.
- Focus your energy on making pages worth citing, clear answers, specific numbers, real examples, rather than on trying to summon a crawler on demand.
If this is the kind of thing you’d rather have someone else set up and monitor rather than piecing together from log files on a Sunday afternoon like I did, that’s exactly the sort of technical groundwork an AI consultant for small business should be doing for you as standard, not as an upsell.
Frequently asked questions
How often does GPTBot crawl a website?
There’s no fixed schedule. Server log data from real sites shows GPTBot arriving in short, heavy bursts, sometimes over 200 hits in a week, followed by silence that can last several weeks or months before the next sweep. Larger, higher-authority sites tend to get swept more often than small, low-traffic ones.
Can I make GPTBot crawl my site more often?
Not directly. You can remove obstacles, an accidental robots.txt block, a slow server, thin or duplicate content, but there is no setting, ping, or submission tool that forces a faster re-crawl. The timing is decided on OpenAI’s side, not yours.
Does more frequent crawling mean I’ll be cited more often in ChatGPT?
Not reliably. Crawling and citation are related but separate. A page crawled once many months ago can still be cited today if it was part of a training snapshot, while a page crawled weekly can go uncited if the content itself isn’t specific or quotable enough to answer a real question.
How do I check if GPTBot has visited my site?
Pull your raw server logs (through your hosting panel or a service like Cloudflare’s bot analytics) and search for the user agent string GPTBot, along with ChatGPT-User, ClaudeBot, and PerplexityBot for the other major ones. Logging timestamps over a few months will show you your site’s actual pattern rather than a guess.
