From: Marc Herbert Date: Tue, 26 Sep 2017 23:27:15 GMT Subject: Google indexing https://public-inbox.org/git (was: BUG in git diff-index) Message-ID: In-Reply-To: <20170926201112.GA26968@whir> [Reduced Cc: and change Subject:] On 26/09/17 13:11, Eric Wong wrote: > There's no blocks on public-inbox.org and I'm completely against > any sort of blocking/throttling. Maybe there's too many pages > to index? Or the Message-IDs in URLs are too ugly/scary? Not > sure what to do about that... > > Anyways, I just put up a robots.txt with Crawl-Delay: 1, since I > seem to recall crawlers use a more conservative delay by default: Not sure what made the difference: Google can now easily find this thread on both https://public-inbox.org/git and on spinics too. Same change observed for another, unrelated thread on this list. Nevermind. "Unsearchable are his judgments, and his ways past finding out"