{"thread":{"id":"23193","subject":"Tree with leading '0' modes in 1.7.0.3","startedAt":"2010-03-26T21:56:00Z","lastAt":"2010-03-28T23:28:28Z","messageCount":33,"participants":["Shawn O. Pearce","Jonathan Nieder","Mike.lifeguard","Junio C Hamano","Avery Pennarun","Nicolas Pitre","Scott Chacon","A Large Angry SCM","Sitaram Chamarty"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"137904","messageId":"20100326215600.GA10910@spearce.org","threadId":"23193","inReplyTo":null,"subject":"Tree with leading '0' modes in 1.7.0.3","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-03-26T21:56:00Z","receivedAt":"2010-03-26T21:56:00Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Mike (CC'd) found a bad Git tree today, where the modes for subtrees\nwhere formatted using a leading '0':\n\n $ od -c tree\n 0000000 1 0 0 6 4 4 R E A D M E \\0 244 \\r\n 0000020 233 214 350 375 0 263 374 227 264 343 $ 031 027 ` 373 301\n 0000040 ! h 0 4 0 0 0 0 m o d u l e s\n 0000060 \\0 262 z K 240 4 377 \\ 245 C c \" 231 377 \\n t\n 0000100 , \\n O R E 0 4 0 0 0 0 s t e w\n 0000120 a r d b o t \\0 037 \\b 5 262 345 234 034 303 C\n 0000140 373 335 207 300 u 341 277 \\f ] 320 207\n 0000153\n\nThe '0' on the 3rd line after '! h' is wrong. It shouldn't be here.\nLikewise the '0' on the 5th line after \"O R E\" is also wrong.\nAt least its consistently broken. But its still broken by fsck\nstandards:\n\n $ git fsck --full a39aa6d\n warning in tree a39aa6d4a6dcfd6c14d8f818bbdf1dfcb3e11771: contains zero-padded file modes\n\nMike claims this tree was created with git-core 1.7.0.3. This thread\nactually started over on Gerrit Code Review's mailing list [1],\nbecause JGit refuses to allow this malformed tree mode to pass its\nfsck implementation.\n\nAny ideas? Why is Git 1.7.0.3 jamming a leading '0' on a file mode?\n\n\n[1] https://groups.google.com/group/repo-discuss/browse_thread/thread/6ff8d7ffba5a9775\n\n-- \nShawn.\n"},{"id":"137906","messageId":"20100326222659.GA18369@progeny.tock","threadId":"23193","inReplyTo":"20100326215600.GA10910@spearce.org","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-03-26T22:26:59Z","receivedAt":"2010-03-26T22:26:59Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Shawn O. Pearce wrote:\n\n> Any ideas? Why is Git 1.7.0.3 jamming a leading '0' on a file mode?\n\nSee http://thread.gmane.org/gmane.comp.version-control.git/141028\nand commit c88f0cc (notes: fix malformed tree entry, 2010-02-24).\n\nThe regression that that fixes appeared in 61a7cca0 (Notes API:\nwrite_notes_tree(): Store the notes tree in the database, 2010-02-13),\nwhich is not part of 1.7.0.3.\n\nStill, HTH,\nJonathan\n"},{"id":"137907","messageId":"20100326222950.GB10910@spearce.org","threadId":"23193","inReplyTo":"20100326222659.GA18369@progeny.tock","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-03-26T22:29:50Z","receivedAt":"2010-03-26T22:29:50Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jonathan Nieder wrote:\n> Shawn O. Pearce wrote:\n> \n> > Any ideas? Why is Git 1.7.0.3 jamming a leading '0' on a file mode?\n> \n> See http://thread.gmane.org/gmane.comp.version-control.git/141028\n> and commit c88f0cc (notes: fix malformed tree entry, 2010-02-24).\n> \n> The regression that that fixes appeared in 61a7cca0 (Notes API:\n> write_notes_tree(): Store the notes tree in the database, 2010-02-13),\n> which is not part of 1.7.0.3.\n\nThat may be true... but I doubt the tree in question was a notes\ntree. The path entries were names like 'README', 'modules' and\n'stewardbot'. Something I would assume was the project's source\ntree, not its notes tree. Unless someone abused the note tree\neditor to edit the README or something...\n\n-- \nShawn.\n"},{"id":"137908","messageId":"20100326224038.GA18454@progeny.tock","threadId":"23193","inReplyTo":"20100326222950.GB10910@spearce.org","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-03-26T22:40:39Z","receivedAt":"2010-03-26T22:40:39Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Shawn O. Pearce wrote:\n> Jonathan Nieder wrote:\n>> Shawn O. Pearce wrote:\n\n>>> Any ideas? Why is Git 1.7.0.3 jamming a leading '0' on a file mode?\n>> \n>> See http://thread.gmane.org/gmane.comp.version-control.git/141028\n>> and commit c88f0cc (notes: fix malformed tree entry, 2010-02-24).\n>> \n>> The regression that that fixes appeared in 61a7cca0 (Notes API:\n>> write_notes_tree(): Store the notes tree in the database, 2010-02-13),\n>> which is not part of 1.7.0.3.\n>\n> That may be true... but I doubt the tree in question was a notes\n> tree. The path entries were names like 'README', 'modules' and\n> 'stewardbot'. Something I would assume was the project's source\n> tree, not its notes tree.\n\nYes, true. The problem is probably elsewhere, especially because\n1.7.0.3 doesn’t even have that commit. Still, I find this a bit\nstrange because such breakage should have been noticeable if it\nhappens often.\n\nWhat has changed recently that involves writing trees?\n\nJonathan\n"},{"id":"137911","messageId":"4BAD3C6E.4090604@gmail.com","threadId":"23193","inReplyTo":"20100326222950.GB10910@spearce.org","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"Mike.lifeguard","fromEmail":"mike.lifeguard@gmail.com","sentAt":"2010-03-26T22:59:58Z","receivedAt":"2010-03-26T22:59:58Z","isPatch":false,"sender":{"key":"mike.lifeguard@gmail.com","avatar":"https://gravatar.com/avatar/a12ecdf9f8b0f34d981d1ae8d7e74205358548f48936c86f240308e942544928?d=mp&s=160"},"body":"-----BEGIN PGP SIGNED MESSAGE-----\nHash: SHA1\n\nOn 10-03-26 07:29 PM, Shawn O. Pearce wrote:\n> Something I would assume was the project's source\n> tree, not its notes tree.\n\nYes, it is the source tree. We don't even know what a notes tree is.\n\nApparently Scott Chacon has some clue about this error:\nhttp://support.github.com/discussions/repos/2566-strange-warning-from-fsck-and-github-repo-using-too-much-diskspace\nso I've added him to CC. (Note that changing all SHA1s is not really a\nproblem for us, there are only 3 copies of the repo, and the project has\nonly been using version control for 2 days)\n\nThanks, all\n- -Mike\n-----BEGIN PGP SIGNATURE-----\nVersion: GnuPG v1.4.9 (GNU/Linux)\n\niEYEARECAAYFAkutPG0ACgkQst0AR/DaKHuRtQCdEyy/KIWwpNYUA4EnkHGy2Y3D\nchwAoLDzdhD9dmmn5mdkJxGrL5Kjlf4/\n=k2Eg\n-----END PGP SIGNATURE-----\n"},{"id":"137912","messageId":"20100326230537.GC10910@spearce.org","threadId":"23193","inReplyTo":"4BAD3C6E.4090604@gmail.com","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-03-26T23:05:37Z","receivedAt":"2010-03-26T23:05:37Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"\"Mike.lifeguard\" wrote:\n> Apparently Scott Chacon has some clue about this error:\n> http://support.github.com/discussions/repos/2566-strange-warning-from-fsck-and-github-repo-using-too-much-diskspace\n> so I've added him to CC. (Note that changing all SHA1s is not really a\n> problem for us, there are only 3 copies of the repo, and the project has\n> only been using version control for 2 days)\n\nScott, please fix that library on GitHub. JGit's fsck has a hard\nfailure on these malformed trees, because the leading '0' mode\ncauses the tree to come up with the wrong SHA-1 hash given its\nlogical content. They shouldn't be created like this.\n\n\nMike, it sounds like you might be able to fix your project by just\nrunning something like:\n\n $ git filter-branch --index-filter '' --all\n\nIt rewrites the trees, which will change their SHA-1s (and the commit\nSHA-1s downstream from there) with correctly formatted tree objects.\n\n-- \nShawn.\n"},{"id":"137913","messageId":"7vbpeaadf5.fsf@alter.siamese.dyndns.org","threadId":"23193","inReplyTo":"20100326224038.GA18454@progeny.tock","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-03-26T23:09:02Z","receivedAt":"2010-03-26T23:09:02Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder writes:\n\n> Shawn O. Pearce wrote:\n>> Jonathan Nieder wrote:\n>>> Shawn O. Pearce wrote:\n>\n>>>> Any ideas? Why is Git 1.7.0.3 jamming a leading '0' on a file mode?\n>>> \n>>> See http://thread.gmane.org/gmane.comp.version-control.git/141028\n>>> and commit c88f0cc (notes: fix malformed tree entry, 2010-02-24).\n>>> \n>>> The regression that that fixes appeared in 61a7cca0 (Notes API:\n>>> write_notes_tree(): Store the notes tree in the database, 2010-02-13),\n>>> which is not part of 1.7.0.3.\n>>\n>> That may be true... but I doubt the tree in question was a notes\n>> tree. The path entries were names like 'README', 'modules' and\n>> 'stewardbot'. Something I would assume was the project's source\n>> tree, not its notes tree.\n>\n> Yes, true. The problem is probably elsewhere, especially because\n> 1.7.0.3 doesn’t even have that commit. Still, I find this a bit\n> strange because such breakage should have been noticeable if it\n> happens often.\n>\n> What has changed recently that involves writing trees?\n\nAsking grep for \"%06o\" reveals nothing. Perhaps somebody else's imitation\nimplementation?\n"},{"id":"137915","messageId":"4BAD41C4.7050508@gmail.com","threadId":"23193","inReplyTo":"20100326230537.GC10910@spearce.org","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"Mike.lifeguard","fromEmail":"mike.lifeguard@gmail.com","sentAt":"2010-03-26T23:22:44Z","receivedAt":"2010-03-26T23:22:44Z","isPatch":false,"sender":{"key":"mike.lifeguard@gmail.com","avatar":"https://gravatar.com/avatar/a12ecdf9f8b0f34d981d1ae8d7e74205358548f48936c86f240308e942544928?d=mp&s=160"},"body":"-----BEGIN PGP SIGNED MESSAGE-----\nHash: SHA1\n\nOn 10-03-26 08:05 PM, Shawn O. Pearce wrote:\n> $ git filter-branch --index-filter '' --all\n\nThis and a few other variations I tried does rewrite things, but the\nproblem persists:\n\nmikelifeguard@arbour:~/Code/git/stewbot (master)$ git filter-branch\n- --subdirectory-filter '' -- --all\nRewrite 5b5d93ca1ebcbc90c3ad688b9b9751d014b452a8 (8/8)\nWARNING: Ref 'refs/heads/master' is unchanged\nWARNING: Ref 'refs/remotes/origin/master' is unchanged\nWARNING: Ref 'refs/remotes/origin/gh-pages' is unchanged\nWARNING: Ref 'refs/remotes/origin/master' is unchanged\nmikelifeguard@arbour:~/Code/git/stewbot (master)$ git fsck\nwarning in tree a39aa6d4a6dcfd6c14d8f818bbdf1dfcb3e11771: contains\nzero-padded file modes\n\n- -Mike\n-----BEGIN PGP SIGNATURE-----\nVersion: GnuPG v1.4.9 (GNU/Linux)\n\niEYEARECAAYFAkutQcQACgkQst0AR/DaKHudmACgtZ67Tv1pO769BL5OvduZacix\nBvMAoLr1UTf5UTd6zowRDHLovBVSY+0R\n=n/G1\n-----END PGP SIGNATURE-----\n"},{"id":"137917","messageId":"20100326234923.GA18759@progeny.tock","threadId":"23193","inReplyTo":"4BAD41C4.7050508@gmail.com","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-03-26T23:49:24Z","receivedAt":"2010-03-26T23:49:24Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Mike.lifeguard wrote:\n> On 10-03-26 08:05 PM, Shawn O. Pearce wrote:\n\n>> $ git filter-branch --index-filter '' --all\n>\n> This and a few other variations I tried does rewrite things, but the\n> problem persists:\n\nYes, I think git write-tree does not rewrite subtrees. How about\n\ngit fast-export --all |\n(\n\tcd /empty/repository &&\n\tgit init &&\n\tgit fast-import\n)\n\n?\n"},{"id":"137918","messageId":"7v7hoyabiv.fsf@alter.siamese.dyndns.org","threadId":"23193","inReplyTo":"20100326230537.GC10910@spearce.org","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-03-26T23:50:00Z","receivedAt":"2010-03-26T23:50:00Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> Scott, please fix that library on GitHub. JGit's fsck has a hard\n> failure on these malformed trees, because the leading '0' mode\n> causes the tree to come up with the wrong SHA-1 hash given its\n> logical content. They shouldn't be created like this.\n\nWhat is curious is that even though 6407180 (git-fsck-cache: be stricter\nabout \"tree\" objects, 2005-07-27) does talk about zero-padding, it appears\nthat we never had a version of git that padded mode in '0' in the entire\nhistory of write-tree (except that \"notes tree\" one, but even that didn't\nescape the laboratory).\n\nBut now we know there is a tool in the wild creating broken objects left\nand right, jgit's fsck routine might need to be more lenient (while\nwarning loudly) in what it accepts.\n\nScott, does your tool have outside users (i.e. being freely distributed\nand you have no control over the continued use of existing copies that\ncreate broken objects)? If not, then there won't be further damage once\nyou fix it at Github, and we may not have to worry about changing jgit\nafter all.\n"},{"id":"137919","messageId":"32541b131003261656h430d77a8q753c6141297e8f86@mail.gmail.com","threadId":"23193","inReplyTo":"7v7hoyabiv.fsf@alter.siamese.dyndns.org","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-03-26T23:56:01Z","receivedAt":"2010-03-26T23:56:01Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Fri, Mar 26, 2010 at 7:50 PM, Junio C Hamano wrote:\n> \"Shawn O. Pearce\" writes:\n>\n>> Scott, please fix that library on GitHub.  JGit's fsck has a hard\n>> failure on these malformed trees, because the leading '0' mode\n>> causes the tree to come up with the wrong SHA-1 hash given its\n>> logical content.  They shouldn't be created like this.\n>\n> What is curious is that even though 6407180 (git-fsck-cache: be stricter\n> about \"tree\" objects, 2005-07-27) does talk about zero-padding, it appears\n> that we never had a version of git that padded mode in '0' in the entire\n> history of write-tree (except that \"notes tree\" one, but even that didn't\n> escape the laboratory).\n\nIt's apparently an easy mistake to make. bup did this for a while\nuntil I added a 'git fsck' to its automated tests :)\n\nThe problem is that everything in git works perfectly with these\ninvalid file modes *except* fsck, and there's rarely a need to run\nfsck, so this problem can hide for a long time.\n\nHave fun,\n\nAvery\n"},{"id":"137920","messageId":"4BAD4A82.5070703@gmail.com","threadId":"23193","inReplyTo":"32541b131003261656h430d77a8q753c6141297e8f86@mail.gmail.com","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"Mike.lifeguard","fromEmail":"mike.lifeguard@gmail.com","sentAt":"2010-03-27T00:00:02Z","receivedAt":"2010-03-27T00:00:02Z","isPatch":false,"sender":{"key":"mike.lifeguard@gmail.com","avatar":"https://gravatar.com/avatar/a12ecdf9f8b0f34d981d1ae8d7e74205358548f48936c86f240308e942544928?d=mp&s=160"},"body":"-----BEGIN PGP SIGNED MESSAGE-----\nHash: SHA1\n\nOn 10-03-26 08:49 PM, Jonathan Nieder wrote:\n> git fast-export --all |\n> (\n> \tcd /empty/repository &&\n> \tgit init &&\n> \tgit fast-import\n> )\n\nThat one did something:\n*When I cd-ed into the repo, there were staged changes waiting for me\n(O.o) -- the changes would have simply deleted every file in the source\ntree.\n*git fsck had no warnings\n*As predicted, SHA1s changed\n\nOn 10-03-26 08:56 PM, Avery Pennarun wrote:\n> The problem is that everything in git works perfectly with these\n> invalid file modes *except* fsck, and there's rarely a need to run\n> fsck, so this problem can hide for a long time.\n\nSo, does the error matter or not? If it doesn't matter, then shouldn't\nJgit stop whining? If it does, then whatever-it-is needs to be fixed.\n\nWe're still not sure what was done with github to cause this. I've done\nnothing with github's web interface, and the project lead can't recall\ndoing anything prior to this error (only stuff today, but this error\ncropped up yesterday). I suspect witchcraft :P\n\n- -Mike\n-----BEGIN PGP SIGNATURE-----\nVersion: GnuPG v1.4.9 (GNU/Linux)\n\niEYEARECAAYFAkutSoIACgkQst0AR/DaKHsblwCcC5j2jDuy95EOjhkK8adfWXl7\nZFEAnRn2bi9glDh6RR3xTwYkjxnMQYqx\n=YSTH\n-----END PGP SIGNATURE-----\n"},{"id":"137925","messageId":"20100327012211.GD10910@spearce.org","threadId":"23193","inReplyTo":"4BAD4A82.5070703@gmail.com","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-03-27T01:22:11Z","receivedAt":"2010-03-27T01:22:11Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"\"Mike.lifeguard\" wrote:\n> On 10-03-26 08:56 PM, Avery Pennarun wrote:\n> > The problem is that everything in git works perfectly with these\n> > invalid file modes *except* fsck, and there's rarely a need to run\n> > fsck, so this problem can hide for a long time.\n> \n> So, does the error matter or not? If it doesn't matter, then shouldn't\n> Jgit stop whining? If it does, then whatever-it-is needs to be fixed.\n\nIts less harmful than other types of corruption. But its quite\nwrong from a format perspective. The hash of the tree differs even\nthough there is no semantic difference in the tree content.\n\nGiven that GitHub has blessed the world with this corruption,\nwe may need to modify JGit to accept it.\n\n-- \nShawn.\n"},{"id":"137926","messageId":"alpine.LFD.2.00.1003262125120.694@xanadu.home","threadId":"23193","inReplyTo":"20100327012211.GD10910@spearce.org","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-03-27T01:30:13Z","receivedAt":"2010-03-27T01:30:13Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 26 Mar 2010, Shawn O. Pearce wrote:\n\n> \"Mike.lifeguard\" wrote:\n> > On 10-03-26 08:56 PM, Avery Pennarun wrote:\n> > > The problem is that everything in git works perfectly with these\n> > > invalid file modes *except* fsck, and there's rarely a need to run\n> > > fsck, so this problem can hide for a long time.\n> > \n> > So, does the error matter or not? If it doesn't matter, then shouldn't\n> > Jgit stop whining? If it does, then whatever-it-is needs to be fixed.\n> \n> Its less harmful than other types of corruption. But its quite\n> wrong from a format perspective. The hash of the tree differs even\n> though there is no semantic difference in the tree content.\n> \n> Given that GitHub has blessed the world with this corruption,\n> we may need to modify JGit to accept it.\n\nShould we?\n\nThis is going to screw up pack v4 (yes, someday I'll have the time to \nmake it real).\n\n\nNicolas\n"},{"id":"137927","messageId":"20100327013443.GE10910@spearce.org","threadId":"23193","inReplyTo":"alpine.LFD.2.00.1003262125120.694@xanadu.home","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-03-27T01:34:43Z","receivedAt":"2010-03-27T01:34:43Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Nicolas Pitre wrote:\n> On Fri, 26 Mar 2010, Shawn O. Pearce wrote:\n> > \"Mike.lifeguard\" wrote:\n> > > On 10-03-26 08:56 PM, Avery Pennarun wrote:\n> > > > The problem is that everything in git works perfectly with these\n> > > > invalid file modes *except* fsck, and there's rarely a need to run\n> > > > fsck, so this problem can hide for a long time.\n> > > \n> > > So, does the error matter or not? If it doesn't matter, then shouldn't\n> > > Jgit stop whining? If it does, then whatever-it-is needs to be fixed.\n> > \n> > Its less harmful than other types of corruption. But its quite\n> > wrong from a format perspective. The hash of the tree differs even\n> > though there is no semantic difference in the tree content.\n> > \n> > Given that GitHub has blessed the world with this corruption,\n> > we may need to modify JGit to accept it.\n> \n> Should we?\n> \n> This is going to screw up pack v4 (yes, someday I'll have the time to \n> make it real).\n\nExactly. I *really* don't want to permit this sort of corruption\nin a Git repository.\n\nBut GitHub's approach here seems to be \"Meh, its fine, don't worry\nabout it\".\n\nIts *NOT* fine. But Avery and Junio might disagree with me. :-)\n\n\nThough, FWIW, it might not screw up pack v4. IIRC from our\ndiscussions long ago on pack v4, we store \"$mode $name\" pairs in\nan indexed list, preciously because we needed to support odd modes\nlike 10664 from ancient Git binaries. If we continue to allow this\ncorruption, it means we have to ensure $mode is the octal string\nand not the binary value.\n\n-- \nShawn.\n"},{"id":"137928","messageId":"alpine.LFD.2.00.1003262142121.694@xanadu.home","threadId":"23193","inReplyTo":"20100327013443.GE10910@spearce.org","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-03-27T01:56:59Z","receivedAt":"2010-03-27T01:56:59Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Fri, 26 Mar 2010, Shawn O. Pearce wrote:\n\n> Nicolas Pitre wrote:\n> > On Fri, 26 Mar 2010, Shawn O. Pearce wrote:\n> > > Given that GitHub has blessed the world with this corruption,\n> > > we may need to modify JGit to accept it.\n> > \n> > Should we?\n> > \n> > This is going to screw up pack v4 (yes, someday I'll have the time to \n> > make it real).\n> \n> Exactly. I *really* don't want to permit this sort of corruption\n> in a Git repository.\n> \n> But GitHub's approach here seems to be \"Meh, its fine, don't worry\n> about it\".\n\nIt's up to GitHub to fork Git then, and while at it stop calling it Git \ncompatible. Really. If we start to get slack about the pack format \nlike this then every Git reimplementation du jour will make similar \ndeviations except in different directions and we'll end up with a mess \nto support.\n\nAnd in this case there is _no_ excuse as 'git fsck' is actually \ncomplaining.\n\nMy stance has always been that the C Git is authoritative with regards to \nformats and protocols. It's up to Github to fix their screw-up.\n\n> Its *NOT* fine. But Avery and Junio might disagree with me. :-)\n\nFWIW I agree with you.\n\n> Though, FWIW, it might not screw up pack v4. IIRC from our\n> discussions long ago on pack v4, we store \"$mode $name\" pairs in\n> an indexed list, preciously because we needed to support odd modes\n> like 10664 from ancient Git binaries. If we continue to allow this\n> corruption, it means we have to ensure $mode is the octal string\n> and not the binary value.\n\nWhich is a real pity.\n\nIn fact, my position is that pack v4 would simply refuse to optimize the \nencoding for such tree objects, period. Only the non ambiguously \nencoded tree objects would benefit from the v4 improvements.\n\n\nNicolas\n"},{"id":"137930","messageId":"32541b131003261933m940ad70g19b3961d20f5a165@mail.gmail.com","threadId":"23193","inReplyTo":"alpine.LFD.2.00.1003262142121.694@xanadu.home","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-03-27T02:33:44Z","receivedAt":"2010-03-27T02:33:44Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Fri, Mar 26, 2010 at 9:56 PM, Nicolas Pitre wrote:\n> On Fri, 26 Mar 2010, Shawn O. Pearce wrote:\n>> Its *NOT* fine.  But Avery and Junio might disagree with me.  :-)\n>\n> FWIW I agree with you.\n\nI would also like to remove my name from the \"disagree\" list. :)\n\nProducing nonstandard output isn't fine at all - I mentioned Postel's\nLaw, but the neglected half of that law is that you're supposed to\nproduce valid data in the first place. This is why (as I mentioned\nearlier) bup's automated tests now run 'git fsck' explicitly to verify\nthat it gets it right. It was only the very first versions of bup,\nwhich thankfully nobody used for anything important, that screwed this\nup. Barring any new and improved screw-ups, anyway.\n\nI only brought it up to say that it's actually easy to make this\nmistake undetected. Very few people run git fsck nowadays. The world\nmight benefit if git complained (albeit non-fatally) *whenever* it saw\nsuch an incorrect tree.\n\n> In fact, my position is that pack v4 would simply refuse to optimize the\n> encoding for such tree objects, period.  Only the non ambiguously\n> encoded tree objects would benefit from the v4 improvements.\n\nThis sounds very wise to me.\n\nHave fun,\n\nAvery\n"},{"id":"137932","messageId":"7vvdci2vk8.fsf@alter.siamese.dyndns.org","threadId":"23193","inReplyTo":"20100327013443.GE10910@spearce.org","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-03-27T05:16:39Z","receivedAt":"2010-03-27T05:16:39Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> But GitHub's approach here seems to be \"Meh, its fine, don't worry\n> about it\".\n>\n> Its *NOT* fine. But Avery and Junio might disagree with me. :-)\n\nDid I ever say it is _fine_? I thought I said \"complain loudly\".\n\nThat would at least give poor jgit users who have hit such a corrupted\nobject a chance to get a controlled notice and ask for help (and get an\ninsn to recover with filter-branch that appeared in this thread).\n"},{"id":"137944","messageId":"d411cc4a1003270544l43f2f93dq5006efb737aa7bbc@mail.gmail.com","threadId":"23193","inReplyTo":"alpine.LFD.2.00.1003262142121.694@xanadu.home","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"Scott Chacon","fromEmail":"schacon@gmail.com","sentAt":"2010-03-27T12:44:12Z","receivedAt":"2010-03-27T12:44:12Z","isPatch":false,"sender":{"key":"schacon@gmail.com","avatar":"https://gravatar.com/avatar/9b13a8a078e1dcf8588c4eea9554445d51ebed6c41b51f56f4d96738130b05c6?d=mp&s=160"},"body":"Hey,\n\nSorry it's taken me a bit - I'm traveling right now.\n\nOn Fri, Mar 26, 2010 at 6:56 PM, Nicolas Pitre wrote:\n>> > > Given that GitHub has blessed the world with this corruption,\n>> > > we may need to modify JGit to accept it.\n\nWell, shouldn't it accept it just because CGit accepts it? Isn't that\nan incompatibility in implementation?\n\n>> But GitHub's approach here seems to be \"Meh, its fine, don't worry\n>> about it\".\n\nThat isn't really my approach, I actually thought I had fixed this a\nwhile ago. It seems to be a pretty understandable mistake, since\nls-tree and cat-file -p both output zero padded modes and it is only\nan issue on trees with subtrees, obviously, so we don't see it all the\ntime at GitHub. I have fixed this and it's in the queue for\ndeployment which should be in the next few days (I gotta get home\nfirst).\n\n> It's up to GitHub to fork Git then, and while at it stop calling it Git\n> compatible.  Really.  If we start to get slack about the pack format\n> like this then every Git reimplementation du jour will make similar\n> deviations except in different directions and we'll end up with a mess\n> to support.\n\nReally? It's not the pack format - we use stock Git servers and\nalmost always have. It's the tree writing when someone edits a file\ninline - I was writing out zero-padded trees. And, it _is_ Git\ncompatible - CGit only issues a warning, and that only if the\ncircumstances align such that we write a tree with a subtree, which\nagain is pretty rare. There are only a handful of projects like this\nand in all CGit circumstances makes no practical difference.\n\n> My stance has always been that the C Git is authoritative with regards to\n> formats and protocols.  It's up to Github to fix their screw-up.\n\nIt is fixed and will be deployed soon, but really, there is no reason\nto be snippy. It is a simple and minor mistake effecting very few\nrepositories (maybe 100 out of 730k), and the only reason it's an\nissue at all is that JGit is not following the authoritative CGit\nimplementation of basically ignoring it.\n\nAlso, if we're all concerned about \"Git reimplementation du jour\"\ndeviations, then we need to focus on libifying Git so there isn't a\nneed for such re-implementations. I'm hoping to help with a possible\nGSoC project on libgit2, but the lack of a linkable library will\nensure that re-implementations in nearly every useful language will\ncontinue.\n\nScott\n"},{"id":"137950","messageId":"alpine.LFD.2.00.1003270959110.694@xanadu.home","threadId":"23193","inReplyTo":"d411cc4a1003270544l43f2f93dq5006efb737aa7bbc@mail.gmail.com","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"Nicolas Pitre","fromEmail":"nico@fluxnic.net","sentAt":"2010-03-27T14:21:30Z","receivedAt":"2010-03-27T14:21:30Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sat, 27 Mar 2010, Scott Chacon wrote:\n\n> Hey,\n> \n> Sorry it's taken me a bit - I'm traveling right now.\n> \n> On Fri, Mar 26, 2010 at 6:56 PM, Nicolas Pitre wrote:\n> >> > > Given that GitHub has blessed the world with this corruption,\n> >> > > we may need to modify JGit to accept it.\n> \n> Well, shouldn't it accept it just because CGit accepts it? Isn't that\n> an incompatibility in implementation?\n\nCGit fsck complains about it. This should be sufficient a clue to \navoid such things.\n\n> >> But GitHub's approach here seems to be \"Meh, its fine, don't worry\n> >> about it\".\n> \n> That isn't really my approach, I actually thought I had fixed this a\n> while ago. It seems to be a pretty understandable mistake, since\n> ls-tree and cat-file -p both output zero padded modes and it is only\n> an issue on trees with subtrees, obviously, so we don't see it all the\n> time at GitHub. I have fixed this and it's in the queue for\n> deployment which should be in the next few days (I gotta get home\n> first).\n\nThanks.\n\n> > It's up to GitHub to fork Git then, and while at it stop calling it Git\n> > compatible.  Really.  If we start to get slack about the pack format\n> > like this then every Git reimplementation du jour will make similar\n> > deviations except in different directions and we'll end up with a mess\n> > to support.\n> \n> Really? It's not the pack format - we use stock Git servers and\n> almost always have. It's the tree writing when someone edits a file\n> inline - I was writing out zero-padded trees. And, it _is_ Git\n> compatible - CGit only issues a warning, and that only if the\n> circumstances align such that we write a tree with a subtree, which\n> again is pretty rare. There are only a handful of projects like this\n> and in all CGit circumstances makes no practical difference.\n\nIt is still damn important to those with an interest in pack format \nimprovements that only one way of creating a tree object exists, \nespecially as we stamp a SHA1 hash on it. Whatever we do with the tree \nencoding in the future, it is essential that the canonical expression of \nany tree object be unambiguous and always produce the same hash.\n\n> > My stance has always been that the C Git is authoritative with regards to\n> > formats and protocols.  It's up to Github to fix their screw-up.\n> \n> It is fixed and will be deployed soon, but really, there is no reason\n> to be snippy. It is a simple and minor mistake effecting very few\n> repositories (maybe 100 out of 730k), and the only reason it's an\n> issue at all is that JGit is not following the authoritative CGit\n> implementation of basically ignoring it.\n\nBut again CGit's fsck is not ignoring this discrepancy. And if the CGit \ncore is otherwise silently accepting it then it is a mistake.\n\n> Also, if we're all concerned about \"Git reimplementation du jour\"\n> deviations, then we need to focus on libifying Git so there isn't a\n> need for such re-implementations. I'm hoping to help with a possible\n> GSoC project on libgit2, but the lack of a linkable library will\n> ensure that re-implementations in nearly every useful language will\n> continue.\n\nDon't get me wrong. I'm not against Git reimplementations per se, as \nlong as they rigorously implement the exact format and protocol from \nCGit. In that sense it is important that the CGit fsck and verify-pack \ntools be exploited on objects/packs produced by alternate Git \nimplementation systematically to find such issues.\n\n\nNicolas\n"},{"id":"137959","messageId":"20100327191405.GF10910@spearce.org","threadId":"23193","inReplyTo":"alpine.LFD.2.00.1003270959110.694@xanadu.home","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-03-27T19:14:05Z","receivedAt":"2010-03-27T19:14:05Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Nicolas Pitre wrote:\n> On Sat, 27 Mar 2010, Scott Chacon wrote:\n> > > My stance has always been that the C Git is authoritative with regards to\n> > > formats and protocols. ??It's up to Github to fix their screw-up.\n> > \n> > It is fixed and will be deployed soon, but really, there is no reason\n> > to be snippy. It is a simple and minor mistake effecting very few\n> > repositories (maybe 100 out of 730k)\n\nWhat is the C Git stance on these 100 repositories then? Are they\nnow considered corrupt? Or is 100 enough in the wild that we have\nto accept the problem, just like we accept the 10664 mode issue from\n\"ancient\" Linux?\n\nI would love to say \"those are corrupt, sorry, fix your repository\".\n\nBut we have traditionally tried to help our users, and not cause\nthem pain. Forcing a rewrite on these 100 projects to fix up the\ncorruption is going to be painful for them. \n\n> > , and the only reason it's an\n> > issue at all is that JGit is not following the authoritative CGit\n> > implementation of basically ignoring it.\n> \n> But again CGit's fsck is not ignoring this discrepancy. And if the CGit \n> core is otherwise silently accepting it then it is a mistake.\n\nRight. I tend to agree. CGit was too lax here, fsck shouldn't\nbe issuing a warning, it should be a fatal error. Both CGit and\nJGit are too lax by not failing when reading that tree during\nnormal processing.\n \n> > Also, if we're all concerned about \"Git reimplementation du jour\"\n> > deviations, then we need to focus on libifying Git so there isn't a\n> > need for such re-implementations. I'm hoping to help with a possible\n> > GSoC project on libgit2, but the lack of a linkable library will\n> > ensure that re-implementations in nearly every useful language will\n> > continue.\n> \n> Don't get me wrong. I'm not against Git reimplementations per se, as \n> long as they rigorously implement the exact format and protocol from \n> CGit. In that sense it is important that the CGit fsck and verify-pack \n> tools be exploited on objects/packs produced by alternate Git \n> implementation systematically to find such issues.\n\nWhen JGit had the tree sort order wrong, JGit was in the wrong,\nand any repository which contained those corrupt trees had to be\nfixed by rewriting them. IIRC it was only the JGit repository\nitself that had this problem in the wild. But we fixed our code.\n\nIMHO, this leading '0' thing is a similar breakage. We shouldn't\nrelax CGit or JGit to accept it just because the Ruby implementation\nof Git got the tree encoding wrong. If anything, we should teach\nthese implementations to catch these sorts of problems earlier.\n\n-- \nShawn.\n"},{"id":"137961","messageId":"20100327192018.GG10910@spearce.org","threadId":"23193","inReplyTo":"7vvdci2vk8.fsf@alter.siamese.dyndns.org","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-03-27T19:20:18Z","receivedAt":"2010-03-27T19:20:18Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Junio C Hamano wrote:\n> \"Shawn O. Pearce\" writes:\n> \n> > But GitHub's approach here seems to be \"Meh, its fine, don't worry\n> > about it\".\n> >\n> > Its *NOT* fine. But Avery and Junio might disagree with me. :-)\n> \n> Did I ever say it is _fine_? I thought I said \"complain loudly\".\n\nI apologize if I misrepresented you above.\n \n> That would at least give poor jgit users who have hit such a corrupted\n> object a chance to get a controlled notice and ask for help (and get an\n> insn to recover with filter-branch that appeared in this thread).\n\nWell, there is \"complain loudly but do it anyway\" and \"hard stop\".\n\nJGit currently has the leading '0' be a \"hard stop\". Because this is\nthe fsck code running inside of the receive-pack service, validating\nwhat the user sent is isn't malformed. Its clearly malformed.\n\nThis only got discovered because Mike tried to take a repository\nfrom GitHub and push it into Gerrit Code Review, where JGit's fsck\nroutine cannot be bypassed during receive-pack.\n\nAre you suggesting JGit should change its behavior to be \"complain\nloudly but do it anyway\"? I'm open to making the code change there\nif that is how you think a Git implementation should behave in\nthis case. But I don't want to do it just to match CGit's behavior,\nsometimes CGit can be wrong. :-)\n\n-- \nShawn.\n"},{"id":"137964","messageId":"4BAE5CB9.6020905@gmail.com","threadId":"23193","inReplyTo":"20100327191405.GF10910@spearce.org","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2010-03-27T19:30:01Z","receivedAt":"2010-03-27T19:30:01Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Shawn O. Pearce wrote:\n> Nicolas Pitre wrote:\n>> On Sat, 27 Mar 2010, Scott Chacon wrote:\n>>>> My stance has always been that the C Git is authoritative with regards to\n>>>> formats and protocols. ??It's up to Github to fix their screw-up.\n>>> It is fixed and will be deployed soon, but really, there is no reason\n>>> to be snippy. It is a simple and minor mistake effecting very few\n>>> repositories (maybe 100 out of 730k)\n> \n> What is the C Git stance on these 100 repositories then? Are they\n> now considered corrupt? Or is 100 enough in the wild that we have\n> to accept the problem, just like we accept the 10664 mode issue from\n> \"ancient\" Linux?\n> \n> I would love to say \"those are corrupt, sorry, fix your repository\".\n> \n> But we have traditionally tried to help our users, and not cause\n> them pain. Forcing a rewrite on these 100 projects to fix up the\n> corruption is going to be painful for them. \n> \n>>> , and the only reason it's an\n>>> issue at all is that JGit is not following the authoritative CGit\n>>> implementation of basically ignoring it.\n>> But again CGit's fsck is not ignoring this discrepancy. And if the CGit \n>> core is otherwise silently accepting it then it is a mistake.\n> \n> Right. I tend to agree. CGit was too lax here, fsck shouldn't\n> be issuing a warning, it should be a fatal error. Both CGit and\n> JGit are too lax by not failing when reading that tree during\n> normal processing.\n> \n>>> Also, if we're all concerned about \"Git reimplementation du jour\"\n>>> deviations, then we need to focus on libifying Git so there isn't a\n>>> need for such re-implementations. I'm hoping to help with a possible\n>>> GSoC project on libgit2, but the lack of a linkable library will\n>>> ensure that re-implementations in nearly every useful language will\n>>> continue.\n>> Don't get me wrong. I'm not against Git reimplementations per se, as \n>> long as they rigorously implement the exact format and protocol from \n>> CGit. In that sense it is important that the CGit fsck and verify-pack \n>> tools be exploited on objects/packs produced by alternate Git \n>> implementation systematically to find such issues.\n> \n> When JGit had the tree sort order wrong, JGit was in the wrong,\n> and any repository which contained those corrupt trees had to be\n> fixed by rewriting them. IIRC it was only the JGit repository\n> itself that had this problem in the wild. But we fixed our code.\n> \n> IMHO, this leading '0' thing is a similar breakage. We shouldn't\n> relax CGit or JGit to accept it just because the Ruby implementation\n> of Git got the tree encoding wrong. If anything, we should teach\n> these implementations to catch these sorts of problems earlier.\n> \n\nJust add an additional data point, it looks like up to 16 of these trees \nwith zero-padded file modes are reachable from Linus' kernel master ref.\n"},{"id":"137965","messageId":"20100327193222.GI10910@spearce.org","threadId":"23193","inReplyTo":"4BAE5CB9.6020905@gmail.com","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2010-03-27T19:32:22Z","receivedAt":"2010-03-27T19:32:22Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"A Large Angry SCM wrote:\n> Shawn O. Pearce wrote:\n>>\n>> IMHO, this leading '0' thing is a similar breakage. We shouldn't\n>> relax CGit or JGit to accept it just because the Ruby implementation\n>> of Git got the tree encoding wrong. If anything, we should teach\n>> these implementations to catch these sorts of problems earlier.\n>\n> Just add an additional data point, it looks like up to 16 of these trees \n> with zero-padded file modes are reachable from Linus' kernel master ref.\n\nFrell.\n\nWe can't ask Linus to rewrite his history to repair this breakage.\nThe fact that its made it into the kernel history means we have\nto accept this. The kernel project is simply too large and move\ntoo fast for us to ask them to fix their repository history.\nSmaller projects of 1-2 people, we could have gotten away with\nasking them to fix their history.\n\nI guess that answers the questions then. CGit permits this with\na warning, and must always continue to do that. And JGit needs to\nfix itself to do the same.\n\n-- \nShawn.\n"},{"id":"137966","messageId":"4BAE5EEC.5090804@gmail.com","threadId":"23193","inReplyTo":"20100327193222.GI10910@spearce.org","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2010-03-27T19:39:24Z","receivedAt":"2010-03-27T19:39:24Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Shawn O. Pearce wrote:\n> A Large Angry SCM wrote:\n>> Shawn O. Pearce wrote:\n>>> IMHO, this leading '0' thing is a similar breakage. We shouldn't\n>>> relax CGit or JGit to accept it just because the Ruby implementation\n>>> of Git got the tree encoding wrong. If anything, we should teach\n>>> these implementations to catch these sorts of problems earlier.\n>> Just add an additional data point, it looks like up to 16 of these trees \n>> with zero-padded file modes are reachable from Linus' kernel master ref.\n> \n> Frell.\n> \n> We can't ask Linus to rewrite his history to repair this breakage.\n> The fact that its made it into the kernel history means we have\n> to accept this. The kernel project is simply too large and move\n> too fast for us to ask them to fix their repository history.\n> Smaller projects of 1-2 people, we could have gotten away with\n> asking them to fix their history.\n> \n> I guess that answers the questions then. CGit permits this with\n> a warning, and must always continue to do that. And JGit needs to\n> fix itself to do the same.\n> \n\nWait a minute, something strange is going on here.\n\nMy combined kernel repository has 16 of these things according to \ngit-fsck. And when I do 'git-fsck torvalds/linux-2.6/master' I get the \nsame 16 BUT when I 'git-rev-list --objects torvalds/linux-2.6/master' \nthey do not appear in the output.\n"},{"id":"137967","messageId":"4BAE601D.6010205@gmail.com","threadId":"23193","inReplyTo":"4BAE5EEC.5090804@gmail.com","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2010-03-27T19:44:29Z","receivedAt":"2010-03-27T19:44:29Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"A Large Angry SCM wrote:\n> Shawn O. Pearce wrote:\n>> A Large Angry SCM wrote:\n>>> Shawn O. Pearce wrote:\n>>>> IMHO, this leading '0' thing is a similar breakage. We shouldn't\n>>>> relax CGit or JGit to accept it just because the Ruby implementation\n>>>> of Git got the tree encoding wrong. If anything, we should teach\n>>>> these implementations to catch these sorts of problems earlier.\n>>> Just add an additional data point, it looks like up to 16 of these \n>>> trees with zero-padded file modes are reachable from Linus' kernel \n>>> master ref.\n>>\n>> Frell.\n>>\n>> We can't ask Linus to rewrite his history to repair this breakage.\n>> The fact that its made it into the kernel history means we have\n>> to accept this. The kernel project is simply too large and move\n>> too fast for us to ask them to fix their repository history.\n>> Smaller projects of 1-2 people, we could have gotten away with\n>> asking them to fix their history.\n>>\n>> I guess that answers the questions then. CGit permits this with\n>> a warning, and must always continue to do that. And JGit needs to\n>> fix itself to do the same.\n>>\n> \n> Wait a minute, something strange is going on here.\n> \n> My combined kernel repository has 16 of these things according to \n> git-fsck. And when I do 'git-fsck torvalds/linux-2.6/master' I get the \n> same 16 BUT when I 'git-rev-list --objects torvalds/linux-2.6/master' \n> they do not appear in the output.\n> \n\nIgnore my noise until some more checking is done!\n\nMy combined repository also includes Scott's progit book and examples \nrepositories. I'm guessing that git-fsck did not limit itself to just \nthe object reachable from the torvalds/linux-2.6/master ref.\n"},{"id":"137968","messageId":"4BAE6331.9020000@gmail.com","threadId":"23193","inReplyTo":"4BAE601D.6010205@gmail.com","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2010-03-27T19:57:37Z","receivedAt":"2010-03-27T19:57:37Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"A Large Angry SCM wrote:\n> A Large Angry SCM wrote:\n>> Shawn O. Pearce wrote:\n>>> A Large Angry SCM wrote:\n>>>> Shawn O. Pearce wrote:\n>>>>> IMHO, this leading '0' thing is a similar breakage. We shouldn't\n>>>>> relax CGit or JGit to accept it just because the Ruby implementation\n>>>>> of Git got the tree encoding wrong. If anything, we should teach\n>>>>> these implementations to catch these sorts of problems earlier.\n>>>> Just add an additional data point, it looks like up to 16 of these \n>>>> trees with zero-padded file modes are reachable from Linus' kernel \n>>>> master ref.\n>>>\n>>> Frell.\n>>>\n>>> We can't ask Linus to rewrite his history to repair this breakage.\n>>> The fact that its made it into the kernel history means we have\n>>> to accept this. The kernel project is simply too large and move\n>>> too fast for us to ask them to fix their repository history.\n>>> Smaller projects of 1-2 people, we could have gotten away with\n>>> asking them to fix their history.\n>>>\n>>> I guess that answers the questions then. CGit permits this with\n>>> a warning, and must always continue to do that. And JGit needs to\n>>> fix itself to do the same.\n>>>\n>>\n>> Wait a minute, something strange is going on here.\n>>\n>> My combined kernel repository has 16 of these things according to \n>> git-fsck. And when I do 'git-fsck torvalds/linux-2.6/master' I get the \n>> same 16 BUT when I 'git-rev-list --objects torvalds/linux-2.6/master' \n>> they do not appear in the output.\n>>\n> \n> Ignore my noise until some more checking is done!\n> \n> My combined repository also includes Scott's progit book and examples \n> repositories. I'm guessing that git-fsck did not limit itself to just \n> the object reachable from the torvalds/linux-2.6/master ref.\n> \n\nThe 16 offending trees are all git-rev-list reachable from Scott's \nprogit book master ref and _NOT_ from Linus' Linux master ref.\n\nSorry about the confusion!!!\n"},{"id":"137969","messageId":"7v7hoxpm3o.fsf@alter.siamese.dyndns.org","threadId":"23193","inReplyTo":"20100327192018.GG10910@spearce.org","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-03-27T20:04:43Z","receivedAt":"2010-03-27T20:04:43Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> JGit currently has the leading '0' be a \"hard stop\". Because this is\n> the fsck code running inside of the receive-pack service, validating\n> what the user sent is isn't malformed. Its clearly malformed.\n\nThe \"complain loudly to let the user know about the need to fix the\ncorrupt repository\" comment was meant for the \"git fsck\" equivalent of\njgit (if there such a thing).\n\nI think \"hard stop to prevent corruption from getting propagated\" is\nactually something we should do in receive-pack in the reference\nimplementation if we don't do so already.\n"},{"id":"137970","messageId":"4BAE6704.8030101@gmail.com","threadId":"23193","inReplyTo":"20100327191405.GF10910@spearce.org","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2010-03-27T20:13:56Z","receivedAt":"2010-03-27T20:13:56Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Shawn O. Pearce wrote:\n> Nicolas Pitre wrote:\n>> On Sat, 27 Mar 2010, Scott Chacon wrote:\n>>>> My stance has always been that the C Git is authoritative with regards to\n>>>> formats and protocols. ??It's up to Github to fix their screw-up.\n>>> It is fixed and will be deployed soon, but really, there is no reason\n>>> to be snippy. It is a simple and minor mistake effecting very few\n>>> repositories (maybe 100 out of 730k)\n> \n> What is the C Git stance on these 100 repositories then? Are they\n> now considered corrupt? Or is 100 enough in the wild that we have\n> to accept the problem, just like we accept the 10664 mode issue from\n> \"ancient\" Linux?\n> \n> I would love to say \"those are corrupt, sorry, fix your repository\".\n> \n> But we have traditionally tried to help our users, and not cause\n> them pain. Forcing a rewrite on these 100 projects to fix up the\n> corruption is going to be painful for them. \n> \n>>> , and the only reason it's an\n>>> issue at all is that JGit is not following the authoritative CGit\n>>> implementation of basically ignoring it.\n>> But again CGit's fsck is not ignoring this discrepancy. And if the CGit \n>> core is otherwise silently accepting it then it is a mistake.\n> \n> Right. I tend to agree. CGit was too lax here, fsck shouldn't\n> be issuing a warning, it should be a fatal error. Both CGit and\n> JGit are too lax by not failing when reading that tree during\n> normal processing.\n\nCGit should treat the object as corrupt, output a message to that \neffect, and continue checking the rest of the objects. Everything else \nthat traverses graph should exit with an error as soon as it tries \ndetects a corrupt object.\n\nThis would allow someone to use git-for-each-ref and git-rev-list to \nprune the graph by deleting refs without trashing the entire repository.\n\n>>> Also, if we're all concerned about \"Git reimplementation du jour\"\n>>> deviations, then we need to focus on libifying Git so there isn't a\n>>> need for such re-implementations. I'm hoping to help with a possible\n>>> GSoC project on libgit2, but the lack of a linkable library will\n>>> ensure that re-implementations in nearly every useful language will\n>>> continue.\n>> Don't get me wrong. I'm not against Git reimplementations per se, as \n>> long as they rigorously implement the exact format and protocol from \n>> CGit. In that sense it is important that the CGit fsck and verify-pack \n>> tools be exploited on objects/packs produced by alternate Git \n>> implementation systematically to find such issues.\n> \n> When JGit had the tree sort order wrong, JGit was in the wrong,\n> and any repository which contained those corrupt trees had to be\n> fixed by rewriting them. IIRC it was only the JGit repository\n> itself that had this problem in the wild. But we fixed our code.\n> \n> IMHO, this leading '0' thing is a similar breakage. We shouldn't\n> relax CGit or JGit to accept it just because the Ruby implementation\n> of Git got the tree encoding wrong. If anything, we should teach\n> these implementations to catch these sorts of problems earlier.\n> \n\nI agree. Now how can the git community help them help themselves?\n"},{"id":"137971","messageId":"7vk4sxo6zr.fsf@alter.siamese.dyndns.org","threadId":"23193","inReplyTo":"20100327191405.GF10910@spearce.org","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2010-03-27T20:16:24Z","receivedAt":"2010-03-27T20:16:24Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Shawn O. Pearce\" writes:\n\n> Nicolas Pitre wrote:\n>> On Sat, 27 Mar 2010, Scott Chacon wrote:\n>> > > My stance has always been that the C Git is authoritative with regards to\n>> > > formats and protocols. ??It's up to Github to fix their screw-up.\n>> > \n>> > It is fixed and will be deployed soon, but really, there is no reason\n>> > to be snippy. It is a simple and minor mistake effecting very few\n>> > repositories (maybe 100 out of 730k)\n>\n> What is the C Git stance on these 100 repositories then? Are they\n> now considered corrupt? Or is 100 enough in the wild that we have\n> to accept the problem, just like we accept the 10664 mode issue from\n> \"ancient\" Linux?\n>\n> I would love to say \"those are corrupt, sorry, fix your repository\".\n\nThis is why I first asked how widespread the copies of the implementation\nof that broken tool are. If it is only 100 and all breakages are confined\nto objects created at GitHub installation, and the owners of these 100\nrepositories are not locally creating corrupt objects with copies of\nbroken reimplementation of git they have, I would say that we tell them to\nfix it, and GitHub can hopefully help them as their hosting site.\n"},{"id":"137976","messageId":"32541b131003271516l3a41ad6dnb75eacb0ad7a8850@mail.gmail.com","threadId":"23193","inReplyTo":"20100327191405.GF10910@spearce.org","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"Avery Pennarun","fromEmail":"apenwarr@gmail.com","sentAt":"2010-03-27T22:16:03Z","receivedAt":"2010-03-27T22:16:03Z","isPatch":false,"sender":{"key":"apenwarr@gmail.com","avatar":"https://avatars.githubusercontent.com/u/20592?v=4"},"body":"On Sat, Mar 27, 2010 at 3:14 PM, Shawn O. Pearce wrote:\n> Nicolas Pitre wrote:\n>> On Sat, 27 Mar 2010, Scott Chacon wrote:\n>> > , and the only reason it's an\n>> > issue at all is that JGit is not following the authoritative CGit\n>> > implementation of basically ignoring it.\n>>\n>> But again CGit's fsck is not ignoring this discrepancy.  And if the CGit\n>> core is otherwise silently accepting it then it is a mistake.\n>\n> Right.  I tend to agree.  CGit was too lax here, fsck shouldn't\n> be issuing a warning, it should be a fatal error.  Both CGit and\n> JGit are too lax by not failing when reading that tree during\n> normal processing.\n\nGah, no! Why would you want to make CGit work *less*? Print a\nwarning, sure, but since the tree is perfectly readable, there's no\nreason to refuse to read it. That's just rude.\n\nSimilarly, there should be no reason for fsck to treat *any*\nrecoverable error as fatal. If it drops dead, it then misses the\nchance to diagnose later problems. When I run fsck, I want to see all\nthe problems, not just the first one, especially if the first one can\nbe fixed by filter-branch. Bonus points if (like e2fsck) it offers to\nfix it for me, though that's probably not worth implementing here.\n\nCGit works fine already. The only problem it has is that it works so\nwell that Scott didn't notice his bug. This can be fixed by adding a\nsimple warning.\n\nHave fun,\n\nAvery\n"},{"id":"138021","messageId":"2e24e5b91003281038u486cd966w1d9263b897e7c9b9@mail.gmail.com","threadId":"23193","inReplyTo":"4BAE601D.6010205@gmail.com","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"Sitaram Chamarty","fromEmail":"sitaramc@gmail.com","sentAt":"2010-03-28T17:38:09Z","receivedAt":"2010-03-28T17:38:09Z","isPatch":false,"sender":{"key":"sitaramc@gmail.com","avatar":"https://avatars.githubusercontent.com/u/43316?v=4"},"body":"On Sun, Mar 28, 2010 at 1:14 AM, A Large Angry SCM wrote:\n\n> My combined repository also includes Scott's progit book and examples\n> repositories. I'm guessing that git-fsck did not limit itself to just the\n> object reachable from the torvalds/linux-2.6/master ref.\n\n\n\nYou have *one* repo containing both Scott's book and the Linux kernel\ntree? \"large angry SCM\" is probably an understatement then... any SCM\nhas the right to be angry mixing up things that are so unrelated!\n\nOr did I totally, *totally* misunderstand...?\n"},{"id":"138036","messageId":"4BAFE61C.7060604@gmail.com","threadId":"23193","inReplyTo":"2e24e5b91003281038u486cd966w1d9263b897e7c9b9@mail.gmail.com","subject":"Re: Tree with leading '0' modes in 1.7.0.3","fromName":"A Large Angry SCM","fromEmail":"gitzilla@gmail.com","sentAt":"2010-03-28T23:28:28Z","receivedAt":"2010-03-28T23:28:28Z","isPatch":false,"sender":{"key":"gitzilla@gmail.com","avatar":"https://gravatar.com/avatar/354625c442439908ff3dd99757dee330e29e9df7847472384faf7a00add247fb?d=mp&s=160"},"body":"Sitaram Chamarty wrote:\n> On Sun, Mar 28, 2010 at 1:14 AM, A Large Angry SCM wrote:\n> \n>> My combined repository also includes Scott's progit book and examples\n>> repositories. I'm guessing that git-fsck did not limit itself to just the\n>> object reachable from the torvalds/linux-2.6/master ref.\n> \n> \n> \n> You have *one* repo containing both Scott's book and the Linux kernel\n> tree? \"large angry SCM\" is probably an understatement then... any SCM\n> has the right to be angry mixing up things that are so unrelated!\n> \n> Or did I totally, *totally* misunderstand...?\n> \n\nYou understood correctly. I use that repository to track activities in a \nnumber of different but related projects/communities. Scott's stuff has \nsince been purged until he fixes the corruption.\n"}]}