{"thread":{"id":"18999","subject":"A system administration use case for git","startedAt":"2009-04-22T08:33:12Z","lastAt":"2009-04-23T11:39:24Z","messageCount":8,"participants":["Jon Seymour","Martin Langhoff","Uwe Kleine-König","Johannes Sixt"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"111930","messageId":"2cfc40320904220133l5ab567f3q46608793b93f0e1f@mail.gmail.com","threadId":"18999","inReplyTo":null,"subject":"A system administration use case for git","fromName":"Jon Seymour","fromEmail":"jon.seymour@gmail.com","sentAt":"2009-04-22T08:33:12Z","receivedAt":"2009-04-22T08:33:12Z","isPatch":false,"sender":{"key":"jon.seymour@gmail.com","avatar":"https://avatars.githubusercontent.com/u/207131?v=4"},"body":"Hello all, it's been quite a while.\n\nAnd, no, I don't care that you ripped out git-rev-list --merge-order,\nit was fun while it lasted and it was still a cool  (if under\nappreciated)  algorithm :-)\n\nI've got a system administration use case that I know git does 99.9%\nof. I am wondering if the last 0.1%.\n\nIt'd be nice to do this on arbitrary (non-git-controlled) file system tree:\n\n* calculate the git hashes of the tree (without making copies of the\nfiles in the tree)\n* archive the tree hashes\n* rsync the tree hashes to another place\n* work out which files aren't available in the other place's git repo\n* rsync those files the the other place\n\nIs there a way to easily achieve this with git's existing tool set or\na higher layer?\n\njon.\n"},{"id":"111931","messageId":"46a038f90904220148i23f1a75n7036d9726870d5f5@mail.gmail.com","threadId":"18999","inReplyTo":"2cfc40320904220133l5ab567f3q46608793b93f0e1f@mail.gmail.com","subject":"Re: A system administration use case for git","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2009-04-22T08:48:23Z","receivedAt":"2009-04-22T08:48:23Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On Wed, Apr 22, 2009 at 10:33 AM, Jon Seymour <jon.seymour@gmail.com> wrote:\n> Hello all, it's been quite a while.\n> It'd be nice to do this on arbitrary (non-git-controlled) file system tree:\n>\n> * calculate the git hashes of the tree (without making copies of the\n> files in the tree)\n> * archive the tree hashes\n> * rsync the tree hashes to another place\n> * work out which files aren't available in the other place's git repo\n> * rsync those files the the other place\n\nSteps:\n\n1 - rsync to the \"other place\"\n2 - use the git repo in that \"other place\"\n3 - if the tree hashes are needed \"here\", copy them from the \"other\nplace\" git to here.\n\nrsync + git = awesomenesssssss\n\n\n\n\n-- \n martin.langhoff@gmail.com\n martin@laptop.org -- School Server Architect\n - ask interesting questions\n - don't get distracted with shiny stuff  - working code first\n - http://wiki.laptop.org/go/User:Martinlanghoff\n"},{"id":"111932","messageId":"2cfc40320904220222ub9b95f8o35afcddb0390fe1b@mail.gmail.com","threadId":"18999","inReplyTo":"2cfc40320904220208g5acc2200w6144668ba2da5a09@mail.gmail.com","subject":"A system administration use case for git","fromName":"Jon Seymour","fromEmail":"jon.seymour@gmail.com","sentAt":"2009-04-22T09:22:09Z","receivedAt":"2009-04-22T09:22:09Z","isPatch":false,"sender":{"key":"jon.seymour@gmail.com","avatar":"https://avatars.githubusercontent.com/u/207131?v=4"},"body":"Martin,\n\nOne disadvantage of that approach is if the file system is very large\nand only has a few deltas, then I effectively have to have two copies\nof the reference file system - one in the GIT repo and one that I can\ndamage on a regular basis with rsync for the purposes calculating the\ngit deltas.   I know I can then use git to repair the damage, but then\nthe reference has to be protected from concurrent access.\n\nI'd prefer not to have to maintain a copy of the tree that has to be\n\"damaged\" on a regular basis and also if I didn't have to maintain a\ncopy of the git objects for identical files.\n\nIn an ideal world, storage requirements at the other place would be\nthose of the reference file system + those of the various deltas, but\nno more.\n\njon seymour\n\nOn Wed, Apr 22, 2009 at 6:48 PM, Martin Langhoff\n<martin.langhoff@gmail.com> wrote:\n> On Wed, Apr 22, 2009 at 10:33 AM, Jon Seymour <jon.seymour@gmail.com> wrote:\n>> Hello all, it's been quite a while.\n>> It'd be nice to do this on arbitrary (non-git-controlled) file system tree:\n>>\n>> * calculate the git hashes of the tree (without making copies of the\n>> files in the tree)\n>> * archive the tree hashes\n>> * rsync the tree hashes to another place\n>> * work out which files aren't available in the other place's git repo\n>> * rsync those files the the other place\n>\n> Steps:\n>\n> 1 - rsync to the \"other place\"\n> 2 - use the git repo in that \"other place\"\n> 3 - if the tree hashes are needed \"here\", copy them from the \"other\n> place\" git to here.\n>\n> rsync + git = awesomenesssssss\n>\n>\n>\n>\n> --\n>  martin.langhoff@gmail.com\n>  martin@laptop.org -- School Server Architect\n>  - ask interesting questions\n>  - don't get distracted with shiny stuff  - working code first\n>  - http://wiki.laptop.org/go/User:Martinlanghoff\n>\n"},{"id":"111935","messageId":"46a038f90904220307h2ac16881qe29c77c5a0334160@mail.gmail.com","threadId":"18999","inReplyTo":"2cfc40320904220222ub9b95f8o35afcddb0390fe1b@mail.gmail.com","subject":"Re: A system administration use case for git","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2009-04-22T10:07:36Z","receivedAt":"2009-04-22T10:07:36Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On Wed, Apr 22, 2009 at 11:22 AM, Jon Seymour <jon.seymour@gmail.com> wrote:\n> One disadvantage of that approach is if the file system is very large\n> and only has a few deltas, then I effectively have to have two copies\n> of the reference file system - one in the GIT repo and one that I can\n\nYou could minimise the on-disk footprint -- and protect it from\nconcurrent access (concurrent change) by using a hardlinked tree on\nthe destination side. rsync knows to break hardlinks, etc.\n\nCurrently, you can't \"rsync into git\" which would save you that step.\nIt's a ton of work to do that -- if anyone is planning on working on\nsomething like that, perhaps writing directly into the fast-import\nprotocol is a good shortcut.\n\nI'd like to have something like that for my OLPC School Server, which\ncould benefit from using git as the backup backend -- it currently\nuses hardlinked directories.\n\n> In an ideal world, storage requirements at the other place would be\n> those of the reference file system + those of the various deltas, but\n> no more.\n\nrsync + hardlinked trees + git gets you quite close to that.\n\ncheers,\n\n\n\nm\n-- \n martin.langhoff@gmail.com\n martin@laptop.org -- School Server Architect\n - ask interesting questions\n - don't get distracted with shiny stuff  - working code first\n - http://wiki.laptop.org/go/User:Martinlanghoff\n"},{"id":"111936","messageId":"2cfc40320904220318ra5e8fcbh863f9a4bf9c82443@mail.gmail.com","threadId":"18999","inReplyTo":"46a038f90904220307h2ac16881qe29c77c5a0334160@mail.gmail.com","subject":"Re: A system administration use case for git","fromName":"Jon Seymour","fromEmail":"jon.seymour@gmail.com","sentAt":"2009-04-22T10:18:26Z","receivedAt":"2009-04-22T10:18:26Z","isPatch":false,"sender":{"key":"jon.seymour@gmail.com","avatar":"https://avatars.githubusercontent.com/u/207131?v=4"},"body":"Martin,\n\nThanks for the info about hard-linked trees...they may well do exactly\nwhat I need - thank you!\n\njon.\n\nOn Wed, Apr 22, 2009 at 8:07 PM, Martin Langhoff\n<martin.langhoff@gmail.com> wrote:\n> On Wed, Apr 22, 2009 at 11:22 AM, Jon Seymour <jon.seymour@gmail.com> wrote:\n>> One disadvantage of that approach is if the file system is very large\n>> and only has a few deltas, then I effectively have to have two copies\n>> of the reference file system - one in the GIT repo and one that I can\n>\n> You could minimise the on-disk footprint -- and protect it from\n> concurrent access (concurrent change) by using a hardlinked tree on\n> the destination side. rsync knows to break hardlinks, etc.\n>\n> Currently, you can't \"rsync into git\" which would save you that step.\n> It's a ton of work to do that -- if anyone is planning on working on\n> something like that, perhaps writing directly into the fast-import\n> protocol is a good shortcut.\n>\n> I'd like to have something like that for my OLPC School Server, which\n> could benefit from using git as the backup backend -- it currently\n> uses hardlinked directories.\n>\n>> In an ideal world, storage requirements at the other place would be\n>> those of the reference file system + those of the various deltas, but\n>> no more.\n>\n> rsync + hardlinked trees + git gets you quite close to that.\n>\n> cheers,\n>\n>\n>\n> m\n> --\n>  martin.langhoff@gmail.com\n>  martin@laptop.org -- School Server Architect\n>  - ask interesting questions\n>  - don't get distracted with shiny stuff  - working code first\n>  - http://wiki.laptop.org/go/User:Martinlanghoff\n>\n"},{"id":"112073","messageId":"20090423095533.GE13989@pengutronix.de","threadId":"18999","inReplyTo":"2cfc40320904220133l5ab567f3q46608793b93f0e1f@mail.gmail.com","subject":"Re: A system administration use case for git","fromName":"Uwe Kleine-König","fromEmail":"u.kleine-koenig@pengutronix.de","sentAt":"2009-04-23T09:55:33Z","receivedAt":"2009-04-23T09:55:33Z","isPatch":false,"sender":{"key":"u.kleine-koenig@pengutronix.de","avatar":"https://gravatar.com/avatar/354b5e3ceb2806a2f1e1e382ac29ddbdad18288654da62b61eb13583a857eee7?d=mp&s=160"},"body":"Hello Jon,\n\nOn Wed, Apr 22, 2009 at 06:33:12PM +1000, Jon Seymour wrote:\n> * calculate the git hashes of the tree (without making copies of the\n> files in the tree)\nYou can hand-craft a tree object:\n\n\tfor f in $filelist; do\n\t\tprintf \"100644 %s\\x00\" $f;\n\n\t\tgit hash-object $f |\n\t\t\tperl -n -e 'chomp; for $c (split(/(.{2})/)) { printf(\"%c\", hex($c)) if $c }';\n\tdone |\n\t\tgit hash-object -t tree --stdin\n\n\nYou get some extra points for directories that include subdirs :-)\n\nThere is a problem though, I think it has to do with the order of the\nfiles in the tree object.  Here is a real-life example from the Linux\nkernel (I choosed the usr subdirectory, because it's small and doesn't\ncontain subdirs):\n\n\tukleinek@cepheus:~/gsrc/linux-2.6/usr$ for f in .gitignore Kconfig Makefile gen_init_cpio.c initramfs_data.S initramfs_data.bz2.S initramfs_data.gz.S initramfs_data.lzma.S; do printf \"100644 %s\\x00\" $f; git hash-object $f | perl -n -e 'chomp; for $c (split(/(.{2})/)) { printf(\"%c\", hex($c)) if $c }'; done | git hash-object -t tree --stdin\n\t64f2ca854bc19fd29479b198be11beba56a26b1e\n\n\tukleinek@cepheus:~/gsrc/linux-2.6/usr$ git rev-parse HEAD:usr\n\t64f2ca854bc19fd29479b198be11beba56a26b1e\n\nThere is a practical problem though:  The filelist has to be sorted in a\nway that is not provided by ls, so:\n\n\tukleinek@cepheus:~/gsrc/linux-2.6/usr$ for f in $(ls -A); do printf \"100644 %s\\x00\" $f; git hash-object $f | perl -n -e 'chomp; for $c (split(/(.{2})/)) { printf(\"%c\", hex($c)) if $c }'; done | git hash-object -t tree -w --stdin\n\ta0a6efb3f1de956badc7607c7d372cc325a18846\n\n\tukleinek@cepheus:~/gsrc/linux-2.6/usr$ git ls-tree a0a6efb3f1de956badc7607c7d372cc325a18846 | wc -l\n\t0\n\ndoesn't work :-/\n\nThis would make a nice plumbing:\n\n\tgit hash-tree $directory\n\nBest regards\nUwe\n\n-- \nPengutronix e.K.                              | Uwe Kleine-König            |\nIndustrial Linux Solutions                    | http://www.pengutronix.de/  |\n"},{"id":"112077","messageId":"49F04511.3070601@viscovery.net","threadId":"18999","inReplyTo":"20090423095533.GE13989@pengutronix.de","subject":"Re: A system administration use case for git","fromName":"Johannes Sixt","fromEmail":"j.sixt@viscovery.net","sentAt":"2009-04-23T10:38:09Z","receivedAt":"2009-04-23T10:38:09Z","isPatch":false,"sender":{"key":"j6t@kdbg.org","avatar":"https://avatars.githubusercontent.com/u/14810926?v=4"},"body":"Uwe Kleine-König schrieb:\n> There is a practical problem though:  The filelist has to be sorted in a\n> way that is not provided by ls, so:\n> \n> \tukleinek@cepheus:~/gsrc/linux-2.6/usr$ for f in $(ls -A); do printf \"100644 %s\\x00\" $f; git hash-object $f | perl -n -e 'chomp; for $c (split(/(.{2})/)) { printf(\"%c\", hex($c)) if $c }'; done | git hash-object -t tree -w --stdin\n> \ta0a6efb3f1de956badc7607c7d372cc325a18846\n\nDoes ... $(LANG=C ls -A) ... make a difference for you?\n\nBut note that this is still not the whole story because, IIUC, in tree\nobjects directories are sorted as if they had a slash appended; i.e.\ndirectory \"foo\" is sorted *after* file \"foo.c\", but *before* file \"foo0\".\n\n-- Hannes\n"},{"id":"112079","messageId":"20090423113924.GA22915@pengutronix.de","threadId":"18999","inReplyTo":"49F04511.3070601@viscovery.net","subject":"Re: A system administration use case for git","fromName":"Uwe Kleine-König","fromEmail":"u.kleine-koenig@pengutronix.de","sentAt":"2009-04-23T11:39:24Z","receivedAt":"2009-04-23T11:39:24Z","isPatch":false,"sender":{"key":"u.kleine-koenig@pengutronix.de","avatar":"https://gravatar.com/avatar/354b5e3ceb2806a2f1e1e382ac29ddbdad18288654da62b61eb13583a857eee7?d=mp&s=160"},"body":"Hi Hannes,\n\nOn Thu, Apr 23, 2009 at 12:38:09PM +0200, Johannes Sixt wrote:\n> Uwe Kleine-König schrieb:\n> > There is a practical problem though:  The filelist has to be sorted in a\n> > way that is not provided by ls, so:\n> > \n> > \tukleinek@cepheus:~/gsrc/linux-2.6/usr$ for f in $(ls -A); do printf \"100644 %s\\x00\" $f; git hash-object $f | perl -n -e 'chomp; for $c (split(/(.{2})/)) { printf(\"%c\", hex($c)) if $c }'; done | git hash-object -t tree -w --stdin\n> > \ta0a6efb3f1de956badc7607c7d372cc325a18846\n> \n> Does ... $(LANG=C ls -A) ... make a difference for you?\noh, up to now I thought that C and en_US.UTF-8 use the same sorting.  So\nyes, it does it right then.\n\nBest regards and thanks\nUwe\n\n-- \nPengutronix e.K.                              | Uwe Kleine-König            |\nIndustrial Linux Solutions                    | http://www.pengutronix.de/  |\n"}]}