Re: Cloning from sites with 404 overridden

From: Shawn Pearce <spearce@spearce.org>
Date: Mar 22, 2006, 03:12 UTC
Message-ID: <20060322031200.GB17954@spearce.org>
In-Reply-To: <20060322025921.1722.qmail@science.horizon.com>

'0' x 40. :-) There's some places already in the GIT source which would have ``issues'' if they got an object with this hash. Not sure if it is actually an entirely impossible hash or just one that is highly improbable.

My own website has this problem and its because I'm using WordPress to handle all URLs on the site; I haven't yet found a way to configure WordPress to return a proper 404 when the URL can't be mapped to something on the server. Note that 404 status codes can in fact return pretty HTML content for the user, and many websites do this and many browsers display that pretty HTML. But a bot can then also recognize the status code and DTRT.

The webservers are just plain broken, mine included. I think the best option is to delay corrupt object reporting to the end of the download process if you get only one corrupt object and that corrupt object was actually attainable from a pack. And in this case its just a minor warning:

	Warning: The server appears to not return proper HTTP status
	codes on missing files.  The files were found in one or
	more packs so the download is OK, but the server administrator
	should really fix their server.  If you know the server
	administrator you might want to prod them to do so.

But that's already been suggested and I thought someone worked up a patch based on that idea? If not I could try to do so since my own damn server has the problem. :-)

linux@horizon.com wrote:

Show 16 quoted lines

> If someone feels ambitious, you can detect this condition automatically
> by searching for a file that you know won't be there and seeing if you
> get a 404 response to that.
> 
> To avoid punishing good servers, it would be nice to defer the test
> until reciving the first corrupted object.
> 
> I'm not sure what the best "object that's not supposed to be there" is.
> It could just be a random hash, or would a malformed object file name
> be better?  Any fixed name has a finite chance of being created by
> someone somewhere, but generating 160-bit random numbers is a PITA on
> non-freenix platforms.
> 
> 
> (As an aside, I suspect this is all caused by Microsoft's "friendly HTML
> error messages" invention.)

-- 
Shawn.