Re: Cloning from sites with 404 overridden

From: Junio C Hamano <junkio@cox.net>
Date: Mar 19, 2006, 23:21 UTC
Message-ID: <7vmzfmm35t.fsf@assigned-by-dhcp.cox.net>
In-Reply-To: <e5bfff550603191340u466d3551t8a95c3808eb977c1@mail.gmail.com>

"Marco Costalba" <mcostalba@gmail.com> writes:

Show 10 quoted lines

> Finally accessing the missing object with a browser
>
> http://digilander.libero.it/mcostalba/
> scm/qgit.git/objects/8d/ea03519e75f47da91108330dde3043defddd60
>
> gives a pre-canned (in italian) 'Sorry page not found' stuff.
>
> So I really think the site "HTTP/1.0 200 OK" response it's a fake.
> Perhaps security related to avoid sniffing (just a guess because I have
> absolutely zero competence in security related things).

I think you are just rephrasing what I said. From the HTTP protocol perspective, you _do_ have that 8d/3a0351 thing on that server, because you do not correctly say "No we donot have it" using 404 response.

Your inability to produce 404 is a different matter -- often the hosting server is not under your control. But that does not change the fact that the repository observed by your clients is "broken". That is why a workaround flag like I suggested may be needed for such a setup.

This is totally untested, but maybe something like this?

---

diff --git a/http-fetch.c b/http-fetch.c
index 7de818b..d523798 100644
--- a/http-fetch.c
+++ b/http-fetch.c
@@ -8,6 +8,7 @@
 #define RANGE_HEADER_SIZE 30
 
 static int got_alternates = -1;
+static int unreliable_404 = 0;
 
 static struct curl_slist *no_pragma_header;
 
@@ -822,12 +823,18 @@ static int fetch_object(struct alt_base 
 		close(obj_req->local); obj_req->local = -1;
 	}
 
+	
+
 	if (obj_req->state == ABORTED) {
 		ret = error("Request for %s aborted", hex);
-	} else if (obj_req->curl_result != CURLE_OK &&
-		   obj_req->http_code != 416) {
+	} else if ((obj_req->curl_result != CURLE_OK &&
+		    obj_req->http_code != 416)  ||
+		   (unreliable_404 &&
+		    obj_req->curl_result == CURLE_OK &&
+		    obj_req->zret != Z_STREAM_END)) {
 		if (obj_req->http_code == 404 ||
-		    obj_req->curl_result == CURLE_FILE_COULDNT_READ_FILE)
+		    obj_req->curl_result == CURLE_FILE_COULDNT_READ_FILE ||
+		    unreliable_404)
 			ret = -1; /* Be silent, it is probably in a pack. */
 		else
 			ret = error("%s (curl_result = %d, http_code = %ld, sha1 = %s)",
@@ -966,6 +973,8 @@ int main(int argc, char **argv)
 			arg++;
 		} else if (!strcmp(argv[arg], "--recover")) {
 			get_recover = 1;
+		} else if (!strcmp(argv[arg], "--unreliable-404")) {
+			unreliable_404 = 1;
 		}
 		arg++;
 	}