Google: Crawl Spikes Aren't Global, Normally Noticed On A "Per Site Vibe"

There is an interesting conversation on Twitter about Google crawling spikes and (1) if these spikes are global or on a per-site basis and (2) if the URL parameter tool going offline caused a global spike in crawling.

If you don’t want to read more, the short answer crawl fluctuations and spikes are not normally a global thing but more on a site-by-site basis. Plus, John Mueller of Google said that the URL parameter tool going offline likely would not result in any spikes to crawling.

Ben Cook and Rebecca Lehmann asked on Twitter:

I’m seeing that, *and* URLs for which we’ve requested temporary removal that are appearing in the index again after only a month. I thought it was supposed to be six months?

— Rebecca Lehmann (@BeckyLehmann) September 9, 2022

John responded that this would likely be unrelated to that tool going offline:

Got it. Thanks for chiming in.

Seems like the dial currently is set to “if there’s a link pointing to it, we’re indexing it whether we can crawl it or not.”

— Ben Cook (@BenjaminCook) September 10, 2022

Adding that these things are more on a site by site “vibe” and not a global thing:

Oh that’s very interesting!

Am I correct that if you aren’t allowed to crawl it, you can’t determine the usefulness of the content?

— Ben Cook (@BenjaminCook) September 10, 2022