Google indexing https://public-inbox.org/git (was: BUG in git diff-index)
- From
- Marc Herbert <marc.herbert@intel.com>
- Date
- Sep 26, 2017, 23:27 UTC
- Message-ID
- <cb495eeb-73d0-ef88-57b6-a601e1d8cfc0@intel.com>
- In-Reply-To
- <20170926201112.GA26968@whir>
[Reduced Cc: and change Subject:]
On 26/09/17 13:11, Eric Wong wrote:
Show 7 quoted lines
> There's no blocks on public-inbox.org and I'm completely against > any sort of blocking/throttling. Maybe there's too many pages > to index? Or the Message-IDs in URLs are too ugly/scary? Not > sure what to do about that... > > Anyways, I just put up a robots.txt with Crawl-Delay: 1, since I > seem to recall crawlers use a more conservative delay by default:
Not sure what made the difference: Google can now easily find this thread on both https://public-inbox.org/git and on spinics too.
Same change observed for another, unrelated thread on this list.
Nevermind.
"Unsearchable are his judgments, and his ways past finding out"