{"thread":{"id":"40784","subject":"Fw: [git-users] git fsck error - duplicate file entries - different then existing stackoverflow scenarios","startedAt":"2015-11-12T11:02:10Z","lastAt":"2015-11-13T05:33:10Z","messageCount":2,"participants":["Konstantin Khomoutov","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"273243","messageId":"20151112140210.ef05a31c401dd49992e9674e@domain007.com","threadId":"40784","inReplyTo":null,"subject":"Fw: [git-users] git fsck error - duplicate file entries - different then existing stackoverflow scenarios","fromName":"Konstantin Khomoutov","fromEmail":"kostix+git@007spb.ru","sentAt":"2015-11-12T11:02:10Z","receivedAt":"2015-11-12T11:02:10Z","isPatch":false,"sender":{"key":"kostix+git@007spb.ru","avatar":null},"body":"A user recently asked an interesting question on the git-users list.\nI think it warrants attentions of a specialists more hard-core than\nwe're there over at git-users.\n\nSo I'd like to solicit help if those knowledgeable, if possible.\n\nBegin forwarded message:\n\nDate: Wed, 11 Nov 2015 14:30:40 -0800 (PST)\nFrom: Wind Over Water <wndovrwtr@g...>\nTo: Git for human beings <git-users@g...>\nSubject: [git-users] git fsck error - duplicate file entries -\ndifferent then existing stackoverflow scenarios\n\n\nHi all,\n\nI have a repo that is giving a 'git fsck --full' error that seems to be \ndifferent from the existing questions and answers on stackoverflow on\nthis topic.  For example, in our fsck error it is not obvious which\nfile is actually duplicated and how/where.  And there is no commit sha\ninvolved - apparently only blob and tree sha's.  But then finding good\ndocumentation on this is challenging.\n\nMight anyone have a pointer as to what to read to help figure out a \nsolution/fix to the below?  Or know of a solution outright?\n\nThanks much in advance!\n\n-sandy\n\n$ git fsck --full\n\nChecking object directories: 100% (256/256), done.\n\nerror in tree df79068051fa8702eae7e91635cca7eee1339002: contains\nduplicate file entries\n\nerror in tree c2d09540a3c3f44c42be1dc8a2b0afa73a35f861: contains\nduplicate file entries\n\nChecking objects: 100% (623704/623704), done.\n\nChecking connectivity: 623532, done.\n\ndangling commit 4d1402c8c74c9f4de6172d7dbd5a14c41683c9e8\n\n\n$ git ls-tree df79068051fa8702eae7e91635cca7eee1339002\n\n100644 blob 14d6d1a6a2f4a7db4e410583c2893d24cb587766 build.gradle\n\n120000 blob cd70e37500a35663957cf60f011f81703be5d032 msrc\n\n040000 tree 658c892e15fbe0d3ea6b8490d9d54c5f2e658fc9 msrc\n\n100644 blob f623819c94a08252298220871ac0ba1118372e59 pom.xml\n\n100644 blob 9223cc2fddb138f691312c1ea2656b9dc17612d2 settings.gradle\n\n040000 tree c3bac1d92722bdee9588a27747b164baa275201f src\n\n\n$ git ls-tree c2d09540a3c3f44c42be1dc8a2b0afa73a35f861\n\n100644 blob 14d6d1a6a2f4a7db4e410583c2893d24cb587766 build.gradle\n\n120000 blob cd70e37500a35663957cf60f011f81703be5d032 msrc\n\n040000 tree 658c892e15fbe0d3ea6b8490d9d54c5f2e658fc9 msrc\n\n100644 blob f623819c94a08252298220871ac0ba1118372e59 pom.xml\n\n100644 blob 9223cc2fddb138f691312c1ea2656b9dc17612d2 settings.gradle\n\n040000 tree a5aa6758a25fee779cbb8c9717d744297071ea79 src\n\n\n$ git show cd70e37500a35663957cf60f011f81703be5d032\n\nsrc/main/java/com/foo/bar/baz/common/\n\n\n$ git show 658c892e15fbe0d3ea6b8490d9d54c5f2e658fc9\n\ntree 658c892e15fbe0d3ea6b8490d9d54c5f2e658fc9\n\n\nBillingAggregator.java\n\nBillingDataThriftAdapter.java\n\n[...]\n\nMetricsProcessor.java\n"},{"id":"273263","messageId":"20151113053310.GC29708@sigill.intra.peff.net","threadId":"40784","inReplyTo":"20151112140210.ef05a31c401dd49992e9674e@domain007.com","subject":"Re: Fw: [git-users] git fsck error - duplicate file entries - different then existing stackoverflow scenarios","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2015-11-13T05:33:10Z","receivedAt":"2015-11-13T05:33:10Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Nov 12, 2015 at 02:02:10PM +0300, Konstantin Khomoutov wrote:\n\n> A user recently asked an interesting question on the git-users list.\n> I think it warrants attentions of a specialists more hard-core than\n> we're there over at git-users.\n> \n> So I'd like to solicit help if those knowledgeable, if possible.\n\nThanks. Curating user questions and forwarding the hard ones here is\nappreciated.\n\n> I have a repo that is giving a 'git fsck --full' error that seems to be \n> different from the existing questions and answers on stackoverflow on\n> this topic.  For example, in our fsck error it is not obvious which\n> file is actually duplicated and how/where.  And there is no commit sha\n> involved - apparently only blob and tree sha's.  But then finding good\n> documentation on this is challenging.\n\nYes, fsck does not traverse the graph in order. So it sees a problem\nwith a particular tree, but cannot know where that tree is within the\nwhole project tree, or which commits reference it. In fact, an arbitrary\nnumber of commits might reference it.\n\nThe most useful thing is sometimes to ask which commit introduced the\ntree (which can _also_ have multiple answers, but usually just one). You\ncan do that by walking the history, like this:\n\n  tree=df79068051fa8702eae7e91635cca7eee1339002\n  git log --all --format=raw --raw -t --no-abbrev | less +/$tree\n\nThat will visit each commit. The options are:\n\n  - we visit commits reachable from all branches and tags (--all)\n\n  - we include the sha1 of the root tree (due to --format=raw)\n\n  - adding --raw shows the raw diff, which includes the sha1 of each\n    file touched by the commit\n\n  - using \"-t\" includes the raw diff for trees, rather than just blobs\n\n  - using \"--no-abbrev\" gives full 40-hex sha1s\n\nAnd then \"less +/$tree\" will open the pager and immediately jump to the\nfirst instance of the sha1 in question.\n\nBut of course that doesn't tell you how to fix it. It might tell you how\nthe bogus object came about (and it is a bogus object; a bug-free git\nimplementation should _never_ produce a tree with duplicate entries.\nAFAIK we have never had such a bug in Git itself, but I have\noccasionally come across problematic entries that I suspect were created\nwith very old versions of JGit).\n\n> error in tree df79068051fa8702eae7e91635cca7eee1339002: contains\n> duplicate file entries\n> [...]\n> $ git ls-tree df79068051fa8702eae7e91635cca7eee1339002\n> \n> 100644 blob 14d6d1a6a2f4a7db4e410583c2893d24cb587766 build.gradle\n> \n> 120000 blob cd70e37500a35663957cf60f011f81703be5d032 msrc\n> \n> 040000 tree 658c892e15fbe0d3ea6b8490d9d54c5f2e658fc9 msrc\n> \n> 100644 blob f623819c94a08252298220871ac0ba1118372e59 pom.xml\n> \n> 100644 blob 9223cc2fddb138f691312c1ea2656b9dc17612d2 settings.gradle\n> \n> 040000 tree c3bac1d92722bdee9588a27747b164baa275201f src\n\nLooks like \"msrc\" is your duplicate entry (even though the sha1s of the\nsub-entries are different, the tree cannot have two entries with the\nsame name). You can use the \"log\" trick above to find the full path to it.\n\nThe fact that one is a symlink (mode 120000) and one is a tree means\nthat whatever git implementation created this presumably has a bug\nrelated to symlinks.\n\nThe only way to fix it is to rewrite the history mentioning the tree\n(because once the tree is fixed, it will get a new sha1, and then any\ncommit referencing it will get a new sha1, and commits built on that,\nand so forth).\n\nYou can use \"git filter-branch\" to do so. There is a sample command\nhere:\n\n  http://stackoverflow.com/questions/32577974/duplicate-file-error-while-pushing-mirror-into-git-repository/\n\nthat just rewrites each tree via a round-trip to the index (so it's not\nclear which of the duplicate entries it will discard). You could also\nwrite a more clever index-filter snippet to use git-update-index to\ninsert the entry you want.\n\n-Peff\n"}]}