{"thread":{"id":"17560","subject":"Question about --tree-filter","startedAt":"2009-02-04T16:08:05Z","lastAt":"2009-02-05T13:57:03Z","messageCount":6,"participants":["Sergio Callegari","Johannes Sixt"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"103159","messageId":"loom.20090204T155824-858@post.gmane.org","threadId":"17560","inReplyTo":null,"subject":"Question about --tree-filter","fromName":"Sergio Callegari","fromEmail":"sergio.callegari@gmail.com","sentAt":"2009-02-04T16:08:05Z","receivedAt":"2009-02-04T16:08:05Z","isPatch":false,"sender":{"key":"sergio.callegari@gmail.com","avatar":"https://gravatar.com/avatar/c98f41317e0422c1e630385de0e3970227b8e5ad15f35ba8586066467cc833bc?d=mp&s=160"},"body":"Hi,\n\nin working with the \"rezip\" filter for the efficient git management of\nopenoffice, zip and docx files, I am encountering the following problem.\n\nSuppose that you have an existing repository and that you want to convert it\ninto a repository using the rezip filters: git filter-branch should be the tool\nto do the conversion.\n\nInitially I believed that once set up the appropriate .git/config filter entries\nand a .git/info/attributes file tying the filter to the appropriate file types,\nit would have been enough to\n\ngit filter-branch --tree-filter true tag-name-filter cat\n\nto do the conversion.\nThis is also what I suggested in my original post about the rezip script.\n\nUnfortunately, this does not seem to work as expected.  Not all files get\nrewritten as filtered blobs.  The only way to do the right job seems to use a\ntree-filter that touches every single file in the project.\n\nAny idea why it is so?\n\nAlso this is not very nice, because it makes the filter-branch result in a huge\namount of work. In other terms, the rezip blob rewriting gets called many many\ntimes more than needed with this technique.\n\nDoes anybody have some suggestion of a tree filter that would be both \"safe\" and\n\"efficient\" ?\n\nThanks\n\nSergio \n"},{"id":"103167","messageId":"4989C437.4070401@viscovery.net","threadId":"17560","inReplyTo":"loom.20090204T155824-858@post.gmane.org","subject":"Re: Question about --tree-filter","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2009-02-04T16:37:11Z","receivedAt":"2009-02-04T16:37:11Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Sergio Callegari schrieb:\n> in working with the \"rezip\" filter for the efficient git management of\n> openoffice, zip and docx files, I am encountering the following problem.\n> \n> Suppose that you have an existing repository and that you want to convert it\n> into a repository using the rezip filters: git filter-branch should be the tool\n> to do the conversion.\n> \n> Initially I believed that once set up the appropriate .git/config filter entries\n> and a .git/info/attributes file tying the filter to the appropriate file types,\n> it would have been enough to\n> \n> git filter-branch --tree-filter true tag-name-filter cat\n> \n> to do the conversion.\n> This is also what I suggested in my original post about the rezip script.\n> \n> Unfortunately, this does not seem to work as expected.  Not all files get\n> rewritten as filtered blobs.\n\nBefore the tree-filter runs, the files are checked out (and smudged by\nrezip). But they are marked as unchanged (because they were checked out\nmoments ago). Since your tree-filter doesn't do anything, no new blobs are\nadded to the index, and none of your files are cleaned by rezip.\n\nI think your brute-force tree-filter should be\n\n   rm -f \"$GIT_INDEX_FILE\"\n\nassuming that a .gitattributes file is already in all revisions.\n\n-- Hannes\n"},{"id":"103200","messageId":"4989FDC6.2080404@gmail.com","threadId":"17560","inReplyTo":"4989C437.4070401@viscovery.net","subject":"Re: Question about --tree-filter","fromName":"Sergio Callegari","fromEmail":"sergio.callegari@gmail.com","sentAt":"2009-02-04T20:42:46Z","receivedAt":"2009-02-04T20:42:46Z","isPatch":false,"sender":{"key":"sergio.callegari@gmail.com","avatar":"https://gravatar.com/avatar/c98f41317e0422c1e630385de0e3970227b8e5ad15f35ba8586066467cc833bc?d=mp&s=160"},"body":"Johannes Sixt wrote:\n> Sergio Callegari schrieb:\n>   \n>> in working with the \"rezip\" filter for the efficient git management of\n>> openoffice, zip and docx files, I am encountering the following problem.\n>>\n>> Suppose that you have an existing repository and that you want to convert it\n>> into a repository using the rezip filters: git filter-branch should be the tool\n>> to do the conversion.\n>>\n>> Initially I believed that once set up the appropriate .git/config filter entries\n>> and a .git/info/attributes file tying the filter to the appropriate file types,\n>> it would have been enough to\n>>\n>> git filter-branch --tree-filter true tag-name-filter cat\n>>\n>> to do the conversion.\n>> This is also what I suggested in my original post about the rezip script.\n>>\n>> Unfortunately, this does not seem to work as expected.  Not all files get\n>> rewritten as filtered blobs.\n>>     \n>\n> Before the tree-filter runs, the files are checked out (and smudged by\n> rezip). But they are marked as unchanged (because they were checked out\n> moments ago). Since your tree-filter doesn't do anything, no new blobs are\n> added to the index, and none of your files are cleaned by rezip.\n>\n> I think your brute-force tree-filter should be\n>\n>    rm -f \"$GIT_INDEX_FILE\"\n>\n> assuming that a .gitattributes file is already in all revisions.\n>\n> -- Hannes\n>   \nSorry, it still is not completely clear to me... I would be very glad if\nyou could detail better what happens when I tree-filter.  From what you\nsay I get the impression that no file should get a new blob.  As a\nmatter of fact, most do (and that is why at the very beginning I thought\nthat --tree-filter true would have been sufficient)... only a few do not\nget the new blob.\n\nAnd if I experiment filter-branch again, with exactly the same\nparameters, apparently some of the files that did not get the new blob\nin the beginning do...  which looks completely weird.\n\nThe attributes are in the info subdir of .git, so the brute force\napproach should be fine.  I guess that it does not make any difference\nwrt to a\n\n    find ./ -type f -exec touch \\{\\} \\;\n\napart from looking slightly more aggressive to the index (and faster) or\ndoes it?\n\nSergio\n"},{"id":"103307","messageId":"498AA41D.3000609@viscovery.net","threadId":"17560","inReplyTo":"4989FDC6.2080404@gmail.com","subject":"Re: Question about --tree-filter","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2009-02-05T08:32:29Z","receivedAt":"2009-02-05T08:32:29Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"[ Don't cull Cc list, please! ]\n\nSergio Callegari schrieb:\n> And if I experiment filter-branch again, with exactly the same\n> parameters, apparently some of the files that did not get the new blob\n> in the beginning do...  which looks completely weird.\n\nI think I know what's going on. filter-branch has this code where the\ntree-filter is applied:\n\n\tgit checkout-index -f -u -a ||\n\t\tdie \"Could not checkout the index\"\n\nThis command may take a while to complete, and at the end it writes the\nindex file. At this point:\n\n(=) Some files may have the same timestamp as the index file.\n\n(<) Others have an earlier timestamp.\n\nLater we have this code:\n\n\t(\n\t\tgit diff-index -r --name-only $commit\n\t\tgit ls-files --others\n\t) |\n\tgit update-index --add --replace --remove --stdin\n\nThe files (=) are racily-clean, and are added to the database; they pass\nthrough the clean filter (rezip). The files (<) are regarded as unchanged,\nand are not added again, and are not rezipped.\n\n>    find ./ -type f -exec touch \\{\\} \\;\n\nThis could help, too, because now all files are regarded as either\nracily-clean (same timestamp as index file) or as changed (newer timestamp).\n\n-- Hannes\n"},{"id":"103327","messageId":"loom.20090205T123832-998@post.gmane.org","threadId":"17560","inReplyTo":"498AA41D.3000609@viscovery.net","subject":"Re: Question about --tree-filter","fromName":"Sergio Callegari","fromEmail":"sergio.callegari@gmail.com","sentAt":"2009-02-05T13:13:54Z","receivedAt":"2009-02-05T13:13:54Z","isPatch":false,"sender":{"key":"sergio.callegari@gmail.com","avatar":"https://gravatar.com/avatar/c98f41317e0422c1e630385de0e3970227b8e5ad15f35ba8586066467cc833bc?d=mp&s=160"},"body":"Johannes Sixt <j.sixt <at> viscovery.net> writes:\n\n> \n> [ Don't cull Cc list, please! ]\n\nSorry... first made a reply ignoring the list and then tried to\nfix!\n \n> I think I know what's going on. filter-branch has this code where the\n> tree-filter is applied:\n> \n> \tgit checkout-index -f -u -a ||\n> \t\tdie \"Could not checkout the index\"\n> \n> This command may take a while to complete, and at the end it writes the\n> index file. At this point:\n> \n> (=) Some files may have the same timestamp as the index file.\n> \n> (<) Others have an earlier timestamp.\n> \n> Later we have this code:\n> \n> \t(\n> \t\tgit diff-index -r --name-only $commit\n> \t\tgit ls-files --others\n> \t) |\n> \tgit update-index --add --replace --remove --stdin\n> \n> The files (=) are racily-clean, and are added to the database; they pass\n> through the clean filter (rezip). The files (<) are regarded as unchanged,\n> and are not added again, and are not rezipped.\n\nOk it is because of a race... now I start understanding the non-consistent\nbehaviour between different runs!  Thanks a lot for the explanation.\n\nWhen you say \"at the end it updates the index file\" do you mean the effect of\nthe -u switch?\n\nAnd when you say \"Some files have the same timestamp as the index file\" do you\nmean that diff-index uses the stat info inside the index only if a file is \"<\"\nthan the index otherwise it is directly assumed that the file is changed wrt the\nindex content? If so, would it make sense to re-touch the index after the\ncheckout -u so that after the checkout the index is always > than every file it\ncontains and one always starts at a non-racy situation? With this, one could\nonly explicitly touch those files that need to get (re)filtered and gain in\nefficiency... or am I still missing something?\n\nSergio\n\n \n"},{"id":"103334","messageId":"498AF02F.4020609@viscovery.net","threadId":"17560","inReplyTo":"loom.20090205T123832-998@post.gmane.org","subject":"Re: Question about --tree-filter","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2009-02-05T13:57:03Z","receivedAt":"2009-02-05T13:57:03Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Sergio Callegari schrieb:\n> Johannes Sixt <j.sixt <at> viscovery.net> writes:\n>> [ Don't cull Cc list, please! ]\n\nBy this I actually meant that you should \"Reply to All\", not just the\nmailing list.\n\n> When you say \"at the end it updates the index file\" do you mean the effect of\n> the -u switch?\n\nI think so.\n\n> And when you say \"Some files have the same timestamp as the index file\" do you\n> mean that diff-index uses the stat info inside the index only if a file is \"<\"\n> than the index otherwise it is directly assumed that the file is changed wrt the\n> index content? If so, would it make sense to re-touch the index after the\n> checkout -u so that after the checkout the index is always > than every file it\n> contains and one always starts at a non-racy situation?\n\nNo. The \"racy\" situation is not something that is bad. It's merely a\nsituation where git cannot decide from stat information alone whether a\nfile was changed or not. So it plays safe, and looks also at the content.\n\nBut if you lie about the index's timestamp, then git will think that all\nfiles are up-to-date.\n\n> With this, one could\n> only explicitly touch those files that need to get (re)filtered and gain in\n> efficiency... or am I still missing something?\n\nNo, you cannot if you are on a \"fast\" machine, where the touch happens in\nthe same second that the index file was written. But you can wait one\nsecond before you touch the files. Depending on the volume of your total\ndata, this might actually be faster as long as you touch only selected files.\n\n-- Hannes\n"}]}