{"thread":{"id":"9334","subject":"Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","startedAt":"2007-08-01T00:16:59Z","lastAt":"2007-08-03T08:20:43Z","messageCount":29,"participants":["Jakub Narebski","Linus Torvalds","Shawn O. Pearce","Junio C Hamano","David Kastrup","Theodore Tso","Alex Riesen","Carl Worth","Brandon Casey","Allan Wind","Florian Weimer","Ramsay Jones","Johan Herland"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"49254","messageId":"200708010216.59750.jnareb@gmail.com","threadId":"9334","inReplyTo":null,"subject":"Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2007-08-01T00:16:59Z","receivedAt":"2007-08-01T00:16:59Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"I have lately added new Git speed benchmark, from Bryan Murdock blog. \nThe repository is bit untypical:\n\n \n By performance, I mean that I used the UNIX time command to see how\n long various basic operations took. Performing the various basic\n operations gave me some insight into the usability of each as well.\n For this test I used a directory with 266 MB of files, 258 KB of which\n were text files, with the rest being image files. I know, kind of\n weird to version all those binary files, but that was the project I\n was interested in testing this out on. Your mileage may vary and all\n that. Here’s a table summarizing the real times reported by time(1):\n
\n\nIf I remember correctly there were some patches to git which tried to \nbetter deal with large blobs. In this simple benchmark git was \noutperformed by Mercurial and even Bazaar-NG a bit.\n\nhttp://git.or.cz/gitwiki/GitBenchmarks#head-5657b8361895b5a02c0de39337c410e4d8dcdbce\nhttp://bryan-murdock.blogspot.com/2007/03/cutting-edge-revision-control.html\n-- \nJakub Narebski\nPoland\n"},{"id":"49268","messageId":"alpine.LFD.0.999.0707311850220.4161@woody.linux-foundation.org","threadId":"9334","inReplyTo":"200708010216.59750.jnareb@gmail.com","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-08-01T02:14:23Z","receivedAt":"2007-08-01T02:14:23Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 1 Aug 2007, Jakub Narebski wrote:\n> \n> If I remember correctly there were some patches to git which tried to \n> better deal with large blobs. In this simple benchmark git was \n> outperformed by Mercurial and even Bazaar-NG a bit.\n\nIt's almost certainly not the binary blobs.\n\nI think almost all the difference is from the cloning, without repacking \nthe souce or using a local clone.\n\nThe default action for a git clone is to create a pack-file, and do a \nlocal clone as if you did it over the network. That is obviously much \nslower than using the \"-l\" flag for the _clone_ action, but it tends to be \nbetter for the end result - since you get a nice packed starting point, \nand none of the confusion with hardlinks etc.\n\n[ Maybe I'm just a worry-wart, but hardlinking two repos still makes me \n worried. Even though we never modify the object files. \n\n Quite frankly, I almost wish we hadn't ever done \"-l\" at all, and I \n cannot really suggest using it. Either use \"-s\" for the truly shared \n repository, or use the default pack-generating one. The hardlinking one \n was simple and made sense, but it's really not very nice.\n\n But that aversion to \"git clone -l\" is really totally illogical. The way \n we do the object handling, hardlinking object files in git is just about \n the most safe operation you can think of - and I *still* shudder at it ]\n\nNow, I think the \"always act as if you were network transparent\" by \ndefault is great, but especially if you have never run \"git gc\" to \ngenerate a pack to begin with, it's going to be a very costly thing. And I \nthink that's what the numbers show. That's the only op we do a *lot* worse \non than we should.\n\n(The \"nonconflicting merge\" is probably - once more - the diffstat \ngeneration that bites us. That's generally the most costly thing of the \nwhole merge, but I *love* the diffstat).\n\nThat said, even if he had done a \"git gc\", to be fair he would have had to \ninclude the cost of that first garbage collect in the \"initial import\", so \nthe end result would have been exactly the same. Git _does_ end up having \na very odd performance profile, and while it's optimized for certain \nthing, the \"initial import\" is not one of them.\n\n(Which admittedly is a bit odd. The reason I didn't ever seriously even \nconsider monotone was that the initial import was so *incredibly* sucky, \nand took hours for the kernel. So use \"-l\" for benchmarks, and damn my \n\"I hate hardlinking repos\" idiocy).\n\nSo the only way to truly do a fast initial import *and* get a reasonably \ngood initial clone is likely one of:\n\n - take full advantage of git, and use local branches, instead of \n bothering with lots of clones.\n\n I think that this is often the right thing to do, but it's obviously \n not fair for comparisons, since it's really something different from \n what's likely available in the other SCM's. But it's the \"git way\".\n\n - use \"git clone -s\" (or \"-l\").\n\n I think the hg numbers are the result of hg defaulting to \"-l\" \n behaviour. Which makes sense for hg, since people need to clone more \n (in git, you'd generally work with local branches instead).\n\n - or the initial import would be done with some \"git fast-import\" thing, \n rather than \"git add .\" We don't do it now, and the resulting pack-file \n wouldn't be optimal, but it would be reasonable. It would at least cut \n down a _bit_ on the clone cost.\n\nThe other reaction I took away from that (quite reasonable, I think) \ncomparison is that I think Murdock would have been much happier if git \ndiff defaulted to \"-C\". We don't do that (for the best of reasons: \ninteroperability), but maybe we should document the \"-M/-C\" options more. \n\nThe options do show up in the man-page, but apparently not \nobviously enough, since he hadn't noticed.\n\n\t\t\tLinus\n"},{"id":"49269","messageId":"20070801021752.GF20052@spearce.org","threadId":"9334","inReplyTo":"200708010216.59750.jnareb@gmail.com","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-08-01T02:17:52Z","receivedAt":"2007-08-01T02:17:52Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jakub Narebski wrote:\n> I have lately added new Git speed benchmark, from Bryan Murdock blog. \n> The repository is bit untypical:\n> \n> \n> By performance, I mean that I used the UNIX time command to see how\n> long various basic operations took. Performing the various basic\n> operations gave me some insight into the usability of each as well.\n> For this test I used a directory with 266 MB of files, 258 KB of which\n> were text files, with the rest being image files. I know, kind of\n> weird to version all those binary files, but that was the project I\n> was interested in testing this out on. Your mileage may vary and all\n> that. Here’s a table summarizing the real times reported by time(1):\n>
\n> \n> If I remember correctly there were some patches to git which tried to \n> better deal with large blobs. In this simple benchmark git was \n> outperformed by Mercurial and even Bazaar-NG a bit.\n\nYes. And we backed them out more recently. :-(\n\nA while ago someone had issues with large binary blobs being added to\nthe repository as loose objects (e.g. by git-add/git-update-index).\nRepacking that repository (for just git-gc or for transport/clone)\nwas ugly as the large binary blob had to be deflated then\nreinflated to encode it in the packfile. The solution was the\ncore.legacyheaders = false configuration setting, which used\npackfile encoding for loose objects, thereby allowing the packer\nto just copy the already compressed data into the output packfile.\n\nUnfortunately we backed that out recently to \"simplify the code\".\nWe can still read that loose object format, but we cannot create\nit and during packing we don't copy the data (we deflate/inflate\nanyway). So we're back to the horrible deflate/inflate problem.\nThat probably explains the large clone time seen by the author.\n\nI wonder if hg realizes that the two repositories are on the\nsame filesystem and automatically uses hardlinks if possible (aka\ngit clone -l). That would easily explain how they can clone so\ndang fast. Maybe we should do the same in git-clone, its a pretty\nsimple thing to do.\n\n\nI do have to question the author's timing method. I don't know if\nthis was hot-cache or not, and he doesn't say. I don't know if the\nsystem was 100% idle when running these times, or the times were\naveraged over a few runs. Usually the first run of anything can\ngive inaccurate timings, as for example the executable code may\nnot be paged in from disk. One of the tools may have had a bias\nas maybe he poked around with that tool first, before starting the\ntimings, so its executables were still hot in cache. Etc.\n\nHowever assuming everything was actually done in a way that the\ntimings can be accurately relied upon...\n\nRegarding the initial file import it looks like we about broke even\nwith bzr if you add the \"initial file import\" and \"initial commit\"\ntimes together. Remember we have to hash and compress the data\nduring git-add; bzr probably delayed their equivilant operation(s)\nuntil the commit operation. Summing these two times is probably\nneeded to really compare them.\n\nWe were also rather close to hg if you again sum the times up.\nBut we do appear to be slower, by about 27s. I guess I find that\nhard to believe, but sure, maybe hg somehow has a faster codepath\nfor their file revision disk IO than we do. Maybe its because hg\nis streaming data and we're loading it all in-core first; maybe the\nauthor's system had to swap get enough virtual memory for git-add.\nMaybe it is just because the author's testing methodology was not\nvery good and one or more of these numbers are just bunk.\n\n\nOur merge time is pretty respectible giving the competition.\nIts probably within the margin of error of the author's testing\nmethodology.\n \n-- \nShawn.\n"},{"id":"49283","messageId":"7vodhrby6f.fsf@assigned-by-dhcp.cox.net","threadId":"9334","inReplyTo":"alpine.LFD.0.999.0707311850220.4161@woody.linux-foundation.org","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2007-08-01T05:50:48Z","receivedAt":"2007-08-01T05:50:48Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds writes:\n\n> (Which admittedly is a bit odd. The reason I didn't ever seriously even \n> consider monotone was that the initial import was so *incredibly* sucky, \n> and took hours for the kernel. So use \"-l\" for benchmarks, and damn my \n> \"I hate hardlinking repos\" idiocy).\n\nI would call aversion to -l a superstition, while aversion to -s\nhas a sound technical reasons. The latter means you need to know\nwhat you are doing --- namely, you are making the clone still\ndependent on the original.\n"},{"id":"49287","messageId":"200708011033.00873.jnareb@gmail.com","threadId":"9334","inReplyTo":"alpine.LFD.0.999.0707311850220.4161@woody.linux-foundation.org","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2007-08-01T08:33:00Z","receivedAt":"2007-08-01T08:33:00Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Linus Torvalds wrote:\n\n> (The \"nonconflicting merge\" is probably - once more - the diffstat \n> generation that bites us. That's generally the most costly thing of the \n> whole merge, but I *love* the diffstat).\n\nhttp://bryan-murdock.blogspot.com/2007/03/cutting-edge-revision-control.html\ndoesn't tell what is the directory structure of imported files.\nIf it is flat, then git does not use advantage of hierarchical tree\nstructure.\n\nBy the way, I guess that \"nonconflicting merge\" is trivial tree-level\nmerge, as \"no changes\" merge should be faster (or fast-forward).\n\nAbout clone: there was \"pack loose, copy existing packs\" idea. I don't\nremember what happened with it. At least for local clone it would be\nnice.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"49289","messageId":"7vzm1b7i8v.fsf@assigned-by-dhcp.cox.net","threadId":"9334","inReplyTo":"200708011033.00873.jnareb@gmail.com","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2007-08-01T08:48:32Z","receivedAt":"2007-08-01T08:48:32Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jakub Narebski writes:\n\n> About clone: there was \"pack loose, copy existing packs\" idea.\n\nCan you give more details --- I do not recall such an \"idea\"\ndiscussed.\n"},{"id":"49290","messageId":"86myxbod1j.fsf@lola.quinscape.zz","threadId":"9334","inReplyTo":"7vodhrby6f.fsf@assigned-by-dhcp.cox.net","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"David Kastrup","fromEmail":"dak@gnu.org","sentAt":"2007-08-01T08:48:56Z","receivedAt":"2007-08-01T08:48:56Z","isPatch":false,"sender":{"key":"dak@gnu.org","avatar":"https://avatars.githubusercontent.com/u/52141349?v=4"},"body":"\nJunio C Hamano writes:\n\n> Linus Torvalds writes:\n>\n>> (Which admittedly is a bit odd. The reason I didn't ever seriously even \n>> consider monotone was that the initial import was so *incredibly* sucky, \n>> and took hours for the kernel. So use \"-l\" for benchmarks, and damn my \n>> \"I hate hardlinking repos\" idiocy).\n>\n> I would call aversion to -l a superstition, while aversion to -s\n> has a sound technical reasons. The latter means you need to know\n> what you are doing --- namely, you are making the clone still\n> dependent on the original.\n\nWell, I'd not call the -l aversy a complete superstition: it means\nthat cloning a repository won't provide any redundancy worth noting\nagainst file system corruption.\n\n-- \nDavid Kastrup\n"},{"id":"49295","messageId":"20070801092428.GB28106@thunk.org","threadId":"9334","inReplyTo":"7vodhrby6f.fsf@assigned-by-dhcp.cox.net","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"Theodore Tso","fromEmail":"tytso@mit.edu","sentAt":"2007-08-01T09:24:29Z","receivedAt":"2007-08-01T09:24:29Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Tue, Jul 31, 2007 at 10:50:48PM -0700, Junio C Hamano wrote:\n> I would call aversion to -l a superstition, while aversion to -s\n> has a sound technical reasons. The latter means you need to know\n> what you are doing --- namely, you are making the clone still\n> dependent on the original.\n\nSo would you accept a patch which adds a git-config variable which\nspecifies whether or not local clones should use hard links by default\n(defaulting to yes), and which adds a --no-hard-links option to\ngit-clone to override the config option?\n\nI could imagine a situation where if you are using a git repository\nexclusively on a local system, with no remote repositories to act as\nbackups, where you might want git clone to to make full copies to\nprovide backups in case of filesystem or disk induced corruption. But\nmost of the time there are enough copies of the the repo on other\nmachines that the need for making separate copies of the git\nobjects/packs isn't really needed.\n\n\t\t\t\t\t- Ted\n"},{"id":"49297","messageId":"7vr6mn5znm.fsf@assigned-by-dhcp.cox.net","threadId":"9334","inReplyTo":"20070801092428.GB28106@thunk.org","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2007-08-01T10:15:25Z","receivedAt":"2007-08-01T10:15:25Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Theodore Tso writes:\n\n> On Tue, Jul 31, 2007 at 10:50:48PM -0700, Junio C Hamano wrote:\n>> I would call aversion to -l a superstition, while aversion to -s\n>> has a sound technical reasons. The latter means you need to know\n>> what you are doing --- namely, you are making the clone still\n>> dependent on the original.\n>\n> So would you accept a patch which adds a git-config variable which\n> specifies whether or not local clones should use hard links by default\n> (defaulting to yes), and which adds a --no-hard-links option to\n> git-clone to override the config option?\n\nAre you suggesting to make -l the default for local, in other\nwords? I personally do not make local clone often enough that I\nam not disturbed having to type extra \" -l\" on the command line.\n\nBut giving a way to force \"copy not hardlink\" while still\navoiding \"the same as the networked case by doing pack transfer\"\noverhead may be a good thing to do.\n\nPerhaps if the destination is local,\n\n - if -s is given, just set up alternates, do nothing else;\n - by default, do \"always copy never hardlink\";\n - with -l, do \"hardlink if possible\";\n\nHmmmm...\n"},{"id":"49305","messageId":"81b0412b0708010620h6bf44aarb35da301db967855@mail.gmail.com","threadId":"9334","inReplyTo":"7vr6mn5znm.fsf@assigned-by-dhcp.cox.net","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"Alex Riesen","fromEmail":"raa.lkml@gmail.com","sentAt":"2007-08-01T13:20:11Z","receivedAt":"2007-08-01T13:20:11Z","isPatch":false,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"On 8/1/07, Junio C Hamano wrote:\n> Theodore Tso writes:\n> > So would you accept a patch which adds a git-config variable which\n> > specifies whether or not local clones should use hard links by default\n> > (defaulting to yes), and which adds a --no-hard-links option to\n> > git-clone to override the config option?\n>\n> Are you suggesting to make -l the default for local, in other\n> words? I personally do not make local clone often enough that I\n> am not disturbed having to type extra \" -l\" on the command line.\n\n...as long as the underlying filesystem _supports_ hardlinks.\n\nBTW, we need a warning when falling back to normal copy,\nif git-clone -l is used. The user _asked_ for a hard-linked\nclone, but silently got something else. Something like this:\n\ndiff --git a/git-clone.sh b/git-clone.sh\nindex 0922554..a744f5b 100755\n--- a/git-clone.sh\n+++ b/git-clone.sh\n@@ -266,6 +266,7 @@ yes,yes)\n \t l=\n \t if ln \"$repo/$sample_file\" \"$GIT_DIR/objects/sample\" 2>/dev/null\n \t then\n+\t\t echo >&2 \"Hardlinks not supported. Falling back to copy\"\n \t\t l=l\n \t fi &&\n \t rm -f \"$GIT_DIR/objects/sample\" &&\n"},{"id":"49306","messageId":"81b0412b0708010620g776fb89ei574ece00ac45bf30@mail.gmail.com","threadId":"9334","inReplyTo":"81b0412b0708010620h6bf44aarb35da301db967855@mail.gmail.com","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"Alex Riesen","fromEmail":"raa.lkml@gmail.com","sentAt":"2007-08-01T13:20:56Z","receivedAt":"2007-08-01T13:20:56Z","isPatch":false,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"On 8/1/07, Alex Riesen wrote:\n> if ln \"$repo/$sample_file\" \"$GIT_DIR/objects/sample\" 2>/dev/null\n> then\n> + echo >&2 \"Hardlinks not supported. Falling back to copy\"\n> l=l\n> fi &&\n\nErr, the other way around, of course.\n"},{"id":"49307","messageId":"81b0412b0708010623x5248612cm78db342c04a0e633@mail.gmail.com","threadId":"9334","inReplyTo":"81b0412b0708010620g776fb89ei574ece00ac45bf30@mail.gmail.com","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"Alex Riesen","fromEmail":"raa.lkml@gmail.com","sentAt":"2007-08-01T13:23:08Z","receivedAt":"2007-08-01T13:23:08Z","isPatch":false,"sender":{"key":"raa.lkml@gmail.com","avatar":"https://avatars.githubusercontent.com/u/324101?v=4"},"body":"On 8/1/07, Alex Riesen wrote:\n> On 8/1/07, Alex Riesen wrote:\n> > if ln \"$repo/$sample_file\" \"$GIT_DIR/objects/sample\" 2>/dev/null\n> > then\n> > + echo >&2 \"Hardlinks not supported. Falling back to copy\"\n> > l=l\n> > fi &&\n>\n> Err, the other way around, of course.\n>\n\ndiff --git a/git-clone.sh b/git-clone.sh\nindex 0922554..483b91d 100755\n--- a/git-clone.sh\n+++ b/git-clone.sh\n@@ -264,8 +264,10 @@ yes,yes)\n \t test -f \"$repo/$sample_file\" || exit\n\n \t l=\n-\t if ln \"$repo/$sample_file\" \"$GIT_DIR/objects/sample\" 2>/dev/null\n+\t if ! ln \"$repo/$sample_file\" \"$GIT_DIR/objects/sample\" 2>/dev/null\n \t then\n+\t\t echo >&2 \"Hardlinks not supported. Falling back to copy\"\n+\t else\n \t\t l=l\n \t fi &&\n \t rm -f \"$GIT_DIR/objects/sample\" &&\n"},{"id":"49320","messageId":"87tzrjfe5h.wl%cworth@cworth.org","threadId":"9334","inReplyTo":"7vr6mn5znm.fsf@assigned-by-dhcp.cox.net","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"Carl Worth","fromEmail":"cworth@cworth.org","sentAt":"2007-08-01T15:49:46Z","receivedAt":"2007-08-01T15:49:46Z","isPatch":false,"sender":{"key":"cworth@cworth.org","avatar":"https://gravatar.com/avatar/3746dc28cde609bdbd7f939058356e7e2bbd16d21e32274df0725eb3d998bc5b?d=mp&s=160"},"body":"On Wed, 01 Aug 2007 03:15:25 -0700, Junio C Hamano wrote:\n>\n> Are you suggesting to make -l the default for local, in other\n> words? I personally do not make local clone often enough that I\n> am not disturbed having to type extra \" -l\" on the command line.\n\nPersonally, I think it would be a great default.\n\nAnd I think the frequency with which you type this command is not a\ngood metric for deciding if a command-line option should be required.\n\nInstead, the focus should be on having good defaults for a good user\nexperience, (for example, the benchmarking that started this thread\nthat gave a bad first impression of git).\n\nSo, just making git-clone go as fast as possible when local, without\nrequiring any additional options from the user, would be a very good\nthing.\n\nAs for the concern that new users might do local clones in the hope to\nget some redundancy, hopefully the fact that the operation is\ninstantaneous will give plenty of clue to the user that no redundancy\nhas been provided. That should be enough to send the user looking for\nthe documentation to find the --no-hard-links option.\n\n-Carl\n"},{"id":"49325","messageId":"alpine.LFD.0.999.0708010937050.3582@woody.linux-foundation.org","threadId":"9334","inReplyTo":"87tzrjfe5h.wl%cworth@cworth.org","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-08-01T17:03:37Z","receivedAt":"2007-08-01T17:03:37Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 1 Aug 2007, Carl Worth wrote:\n>\n> On Wed, 01 Aug 2007 03:15:25 -0700, Junio C Hamano wrote:\n> >\n> > Are you suggesting to make -l the default for local, in other\n> > words? I personally do not make local clone often enough that I\n> > am not disturbed having to type extra \" -l\" on the command line.\n> \n> Personally, I think it would be a great default.\n\nI suspect it probably *would* make sense to default to \"-l\". Even if it \nmakes me get goose-bumps. I freely admit that my worries are totally \nillogical.\n\nWe might make it something like: \"if you use an url, we don't default to \nlocal\", so the difference would be that\n\n\tgit clone file:///directory/to/repo\n\nwould work the way it does now, but\n\n\tgit clone /directory/to/repo\n\nwould default to \"-l\" behaviour. That kind of would make sense (and should \nbe easy to implement: it would be a trivial fixup to \"connect.c\".\n\nSomething like this adds support for \"file://\". And then git-clone could \njust do something like\n\n\t# if the source is a local directory, default to local\n\tif [ -d \"$src\" ]; then\n\t\tuse_local=yes\n\tfi\n\nor similar.\n\n\t\tLinus\n\n---\n connect.c | 12 +++++++-----\n 1 files changed, 7 insertions(+), 5 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 715cdc0..ae49c5a 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -145,6 +145,8 @@ static enum protocol get_protocol(const char *name)\n \t\treturn PROTO_SSH;\n \tif (!strcmp(name, \"ssh+git\"))\n \t\treturn PROTO_SSH;\n+\tif (!strcmp(name, \"file\"))\n+\t\treturn PROTO_LOCAL;\n \tdie(\"I don't handle protocol '%s'\", name);\n }\n \n@@ -498,13 +500,13 @@ pid_t git_connect(int fd[2], char *url, const char *prog, int flags)\n \t\tend = host;\n \n \tpath = strchr(end, c);\n-\tif (c == ':') {\n-\t\tif (path) {\n+\tif (path) {\n+\t\tif (c == ':') {\n \t\t\tprotocol = PROTO_SSH;\n \t\t\t*path++ = '\\0';\n-\t\t} else\n-\t\t\tpath = host;\n-\t}\n+\t\t}\n+\t} else\n+\t\tpath = end;\n \n \tif (!path || !*path)\n \t\tdie(\"No path specified. See 'man git-pull' for valid url syntax\");\n"},{"id":"49333","messageId":"85vebzrufc.fsf@lola.goethe.zz","threadId":"9334","inReplyTo":"alpine.LFD.0.999.0708010937050.3582@woody.linux-foundation.org","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"David Kastrup","fromEmail":"dak@gnu.org","sentAt":"2007-08-01T18:17:27Z","receivedAt":"2007-08-01T18:17:27Z","isPatch":false,"sender":{"key":"dak@gnu.org","avatar":"https://avatars.githubusercontent.com/u/52141349?v=4"},"body":"Linus Torvalds writes:\n\n> I suspect it probably *would* make sense to default to \"-l\". Even if it \n> makes me get goose-bumps. I freely admit that my worries are totally \n> illogical.\n>\n> We might make it something like: \"if you use an url, we don't default to \n> local\",\n\nCouldn't git clone http://host/directory/to/repo tell the proxy that\nit should enter off-line mode and stop updating?\n\n-- \nDavid Kastrup, Kriemhildstr. 15, 44793 Bochum\n"},{"id":"49388","messageId":"87hcnj9el8.fsf@mid.deneb.enyo.de","threadId":"9334","inReplyTo":"85vebzrufc.fsf@lola.goethe.zz","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"Florian Weimer","fromEmail":"fw@deneb.enyo.de","sentAt":"2007-08-01T20:36:51Z","receivedAt":"2007-08-01T20:36:51Z","isPatch":false,"sender":{"key":"fw@deneb.enyo.de","avatar":null},"body":"* David Kastrup:\n\n> Couldn't git clone http://host/directory/to/repo tell the proxy that\n> it should enter off-line mode and stop updating?\n\nHuh? I don't see how this is relevant to the current thread.\n\nAnyway, I don't think the max-stale cache control mechanism is widely\nimplemented. If you want effective expiry controls, you need to\nimplement them on the server side.\n"},{"id":"49353","messageId":"20070801220350.GD28106@thunk.org","threadId":"9334","inReplyTo":"7vr6mn5znm.fsf@assigned-by-dhcp.cox.net","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"Theodore Tso","fromEmail":"tytso@mit.edu","sentAt":"2007-08-01T22:03:50Z","receivedAt":"2007-08-01T22:03:50Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Wed, Aug 01, 2007 at 03:15:25AM -0700, Junio C Hamano wrote:\n> > So would you accept a patch which adds a git-config variable which\n> > specifies whether or not local clones should use hard links by default\n> > (defaulting to yes), and which adds a --no-hard-links option to\n> > git-clone to override the config option?\n> \n> Are you suggesting to make -l the default for local, in other\n> words? I personally do not make local clone often enough that I\n> am not disturbed having to type extra \" -l\" on the command line.\n\nYeah, essentially, with a git-config option (and comand-line option)\nto override the default for those people who are \"squeamish\" about git\nclone -l. Linus's suggestion of using file:// as a way to indicate\nnon-local also makes a lot of sense to me.\n\n> Perhaps if the destination is local,\n> \n> - if -s is given, just set up alternates, do nothing else;\n\nAs I understand it, the main objection with making -s the default is\nsurprising result that could happen if you do a git-prune in the base\nrepository which causes objects which are borrowed from the base\nrepository via .git/objects/info/alternates, right?\n\nWhat if git clone -s appended the repository which is borrowing\nobjects via alternates to a file located in the base repository,\n.git/objects/info/shared-repos?\n\nThen git-prune could also use the refs marked in each of the\ndownstream repositories that are sharing objects with base repository\nand not make those objects go away. That way, git-gc --prune won't do\nanything surprising to any shared repositories, since it will scan\nthose shared repositories automatically. Would that be considered\nacceptable? That way you can get instant clones even on filesystems\nthat don't support hard links, such as Windows systems.\n\nThe way I would suggest doing it is once we implement support for\n.git/objects/info/shared-repos is to do the following with git-clone\nby default:\n\n \t* If the source repo is specified via a file:// URL, per Linus's\n suggestion, do the clone via copying.\n\n\t* If the source repo is specified via a local pathname, and\n .git/objects/info/shared-repos in the source repository is\n writeable by the user who is doing the clone, then do a\n clone -s.\n\n\t* If the above fails, try clone -l\n\n\t* If the above fails, do a clone via copying over a new pack\n\nWould this be acceptable? Patches to do the following should be\nfairly easy to whip up. Obviously this wouldn't be for 1.5.3. :-)\n\n \t \t \t \t \t- Ted\n"},{"id":"49357","messageId":"200708020018.18610.jnareb@gmail.com","threadId":"9334","inReplyTo":"7vr6mn5znm.fsf@assigned-by-dhcp.cox.net","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2007-08-01T22:18:18Z","receivedAt":"2007-08-01T22:18:18Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Junio C Hamano wrote:\n\n> Perhaps if the destination is local,\n> \n> - if -s is given, just set up alternates, do nothing else;\n> - by default, do \"always copy never hardlink\";\n> - with -l, do \"hardlink if possible\";\n> \n> Hmmmm...\n\nThat I think it is the best solution, together with support for\nfile:///path/to/repo.git scheme which would turn on old repacking\nbehavior. I'm all for it.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"49367","messageId":"46B10DE2.5060702@nrlssc.navy.mil","threadId":"9334","inReplyTo":"20070801220350.GD28106@thunk.org","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"Brandon Casey","fromEmail":"casey@nrlssc.navy.mil","sentAt":"2007-08-01T22:49:06Z","receivedAt":"2007-08-01T22:49:06Z","isPatch":false,"sender":{"key":"drafnel@gmail.com","avatar":"https://avatars.githubusercontent.com/u/921167?v=4"},"body":"Theodore Tso wrote:\n> On Wed, Aug 01, 2007 at 03:15:25AM -0700, Junio C Hamano wrote:\n\n>> Perhaps if the destination is local,\n>>\n>> - if -s is given, just set up alternates, do nothing else;\n> \n> As I understand it, the main objection with making -s the default is\n> surprising result that could happen if you do a git-prune in the base\n> repository which causes objects which are borrowed from the base\n> repository via .git/objects/info/alternates, right?\n\n-s would be a lot safer to use if repack -a -d (as used by git-gc) was smarter.\n-a -d has the nasty side effect of doing what it seems only prune is intended\nto do... that is to remove unreferenced objects.\n\n-s usage currently has to be very well thought out, unless you're just using it\nfor a short-lived temporary branch. If this unintended pruning could be avoided\nthen an average user could go about their merry business repacking and git-gc'ing\nwithout a care, and only when doing a git-gc --prune would they need to do\nsomething special.\n\n-brandon\n"},{"id":"49372","messageId":"200708020151.17301.jnareb@gmail.com","threadId":"9334","inReplyTo":"7vzm1b7i8v.fsf@assigned-by-dhcp.cox.net","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2007-08-01T23:51:16Z","receivedAt":"2007-08-01T23:51:16Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Junio C Hamano wrote:\n> Jakub Narebski writes:\n> \n> > About clone: there was \"pack loose, copy existing packs\" idea.\n> \n> Can you give more details --- I do not recall such an \"idea\"\n> discussed.\n\nThe idea was to avoid repacking, and just pack loose, unpacked objects \n(and save this pack if possible), then concatenate all packs and send \nthis concatenated pack as the result. This saves a bit (quite a bit) of \nCPU at the cost of additional bandwidth usage if packfiles are not \noptimized.\n\nThe only result of the discussion was that it would be fairly easy to \nsend multiple packs concatenated into one pack, without need to add \nsome multi-pack extension, as there would be required minor changes to \nsplit \"concatenated\" packfiles.\n\n-- \nJakub Narebski\nPoland\n"},{"id":"49383","messageId":"20070802040201.GI23484@lifeintegrity.com","threadId":"9334","inReplyTo":"20070801220350.GD28106@thunk.org","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"Allan Wind","fromEmail":"allan_wind@lifeintegrity.com","sentAt":"2007-08-02T04:02:01Z","receivedAt":"2007-08-02T04:02:01Z","isPatch":false,"sender":{"key":"allan_wind@lifeintegrity.com","avatar":null},"body":"On 2007-08-01T18:03:50-0400, Theodore Tso wrote:\n> Yeah, essentially, with a git-config option (and comand-line option)\n> to override the default for those people who are \"squeamish\" about git\n> clone -l. Linus's suggestion of using file:// as a way to indicate\n> non-local also makes a lot of sense to me.\n\nI would expect /something and file:///something to behave exactly the \nsame way (the latter just having bit extra syntax sugar).\n\n\n/Allan\n"},{"id":"49382","messageId":"alpine.LFD.0.999.0708012109040.3582@woody.linux-foundation.org","threadId":"9334","inReplyTo":"20070802040201.GI23484@lifeintegrity.com","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-08-02T04:13:16Z","receivedAt":"2007-08-02T04:13:16Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 2 Aug 2007, Allan Wind wrote:\n> \n> I would expect /something and file:///something to behave exactly the \n> same way (the latter just having bit extra syntax sugar).\n\nI do agree that they should be basically the same, but from an \nimplementation standpoint it actually makes a lot of sense to separate \nthem. Also, there's actually a small amount of \"logic\" in it: the \n/something is obviously a \"raw filename\", while the \"file:://something\" \nclearly is something a lot more abstract.\n\nI don't actually have a very strong opinion, but I do think that \"file://\" \nmakes sense regardless (ie the patch I sent out is probably a good idea).\n\nI also strongly dispute that \"file://something\" is _identical_ to just \n\"something\". There's a huge difference, as anybody who has ever tried to \ndo\n\n\tcp file://file-A file-B\n\nwill have hopefully found out. They may mean the same thing, but they have \ntotally different levels of abstraction, so it does actually make some \nsense that you end up *cloning* the same thing, but different ways.\n\n\t\tLinus\n"},{"id":"49390","messageId":"7vejim1n92.fsf@assigned-by-dhcp.cox.net","threadId":"9334","inReplyTo":"alpine.LFD.0.999.0708010937050.3582@woody.linux-foundation.org","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2007-08-02T06:09:13Z","receivedAt":"2007-08-02T06:09:13Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds writes:\n\n> We might make it something like: \"if you use an url, we don't default to \n> local\", so the difference would be that\n>\n> \tgit clone file:///directory/to/repo\n>\n> would work the way it does now, but\n>\n> \tgit clone /directory/to/repo\n>\n> would default to \"-l\" behaviour. That kind of would make sense (and should \n> be easy to implement: it would be a trivial fixup to \"connect.c\".\n\nThe attached does not default to \"-l\", but filesystem level copy\nbehaviour, which is what happens with \"clone -l\" across\nfilesystem boundaries with the current code.\n\nClone of linux-2.6 repository (the source is well packed)\n\n(hardlink -- obviously, almost no cost)\n$ /usr/bin/time git clone -l --bare linux-2.6 l-clone.git\nInitialized empty Git repository in /git/l-clone.git/\n0 blocks\n0.55user 1.00system 0:01.56elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+206724minor)pagefaults 0swaps\n\n(same-as-network)\n$ /usr/bin/time git clone --bare file://`pwd`/linux-2.6 n-clone.git\nInitialized empty Git repository in /git/n-clone.git/\nremote: Generating pack...\nremote: Counting objects: 1076746\nremote: Done counting 1169654 objects.\nremote: Deltifying 1169654 objects...\nremote: 100% (1169654/1169654) done\nIndexing 1169654 objects...\n 100% (1169654/1169654) done\nremote: Total 1169654 (delta 959223), reused 1160595 (delta 950164)\nResolving 959223 deltas...\n 100% (959223/959223) done\n172.85user 20.94system 4:25.88elapsed 72%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (6294major+2019874minor)pagefaults 0swaps\n\n(copy -- takes a lot more than hardlink but cheaper than net)\n$ /usr/bin/time git clone --bare linux-2.6 c-clone.git\nInitialized empty Git repository in /git/c-clone.git/\n1266644 blocks\n0.92user 10.81system 0:38.38elapsed 30%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (406major+204775minor)pagefaults 0swaps\n\nI am ambivalent between -l vs no -l.\n\n * Without -l (i.e. have all objects/ copied via cpio) would not\n catch the source repository corruption, and also risks\n corrupted recipient repository if an alpha-particle hits\n memory cell while indexing and resolving deltas. As long as\n the recipient is made uncorrupted, you have a good back-up.\n\n * same-as-network is expensive, but it would catch if the\n source is already corrupted. It still risks corrupted\n recipient repository. As long as the recipient is made\n uncorrupted, you have a good back-up.\n\n * With -l, as long as the source repository is healthy, it is\n very likely that the recipient would be, too. Also it is\n very cheap. You do not get any back-up benefit.\n\nNone of the method is resilient against the source repository\ncorruption, so let's discount that from the comparison. Then\nthe differences between -l and non -l matters primarily if you\nvalue the back-up benefit or not. If you want to use the cloned\nrepository as a back-up, then it is cheaper to do a non -l clone\nand two git-fsck (source before clone, recipient after clone)\nthan same-as-network clone, especially as you are likely to do a\ngit-fsck on the recipient if you are so paranoid anyway.\n\nWhich leads me to believe that being able to use file:/// is\nprobably a good idea, if only for testability, but probably of\nlittle practical value, and we can default to -l for everyday\nuse, and paranoids can use non -l as a way to make a back-up.\n\n---\n\n git-clone.sh | 61 +++++++++++++++++++++++---------------------\n t/t5500-fetch-pack.sh | 2 +-\n t/t5700-clone-reference.sh | 2 +-\n t/t5701-clone-local.sh | 17 ++++++++++++\n 4 files changed, 51 insertions(+), 31 deletions(-)\n\ndiff --git a/git-clone.sh b/git-clone.sh\nindex 0922554..0583f64 100755\n--- a/git-clone.sh\n+++ b/git-clone.sh\n@@ -87,7 +87,7 @@ Perhaps git-update-server-info needs to be run there?\"\n \n quiet=\n local=no\n-use_local=no\n+use_local_hardlink=no\n local_shared=no\n unset template\n no_checkout=\n@@ -108,9 +108,10 @@ while\n \t no_checkout=yes ;;\n \t*,--na|*,--nak|*,--nake|*,--naked|\\\n \t*,-b|*,--b|*,--ba|*,--bar|*,--bare) bare=yes ;;\n-\t*,-l|*,--l|*,--lo|*,--loc|*,--loca|*,--local) use_local=yes ;;\n+\t*,-l|*,--l|*,--lo|*,--loc|*,--loca|*,--local)\n+\t use_local_hardlink=yes ;;\n *,-s|*,--s|*,--sh|*,--sha|*,--shar|*,--share|*,--shared)\n- local_shared=yes; use_local=yes ;;\n+ local_shared=yes; ;;\n \t1,--template) usage ;;\n \t*,--template)\n \t\tshift; template=\"--template=$1\" ;;\n@@ -249,34 +250,36 @@ fi\n rm -f \"$GIT_DIR/CLONE_HEAD\"\n \n # We do local magic only when the user tells us to.\n-case \"$local,$use_local\" in\n-yes,yes)\n+case \"$local\" in\n+yes)\n \t( cd \"$repo/objects\" ) ||\n-\t\tdie \"-l flag seen but repository '$repo' is not local.\"\n+\t\tdie \"cannot chdir to local '$repo/objects'.\"\n \n-\tcase \"$local_shared\" in\n-\tno)\n-\t # See if we can hardlink and drop \"l\" if not.\n-\t sample_file=$(cd \"$repo\" && \\\n-\t\t\t find objects -type f -print | sed -e 1q)\n-\n-\t # objects directory should not be empty since we are cloning!\n-\t test -f \"$repo/$sample_file\" || exit\n-\n-\t l=\n-\t if ln \"$repo/$sample_file\" \"$GIT_DIR/objects/sample\" 2>/dev/null\n-\t then\n-\t\t l=l\n-\t fi &&\n-\t rm -f \"$GIT_DIR/objects/sample\" &&\n-\t cd \"$repo\" &&\n-\t find objects -depth -print | cpio -pumd$l \"$GIT_DIR/\" || exit 1\n-\t ;;\n-\tyes)\n-\t mkdir -p \"$GIT_DIR/objects/info\"\n-\t echo \"$repo/objects\" >> \"$GIT_DIR/objects/info/alternates\"\n-\t ;;\n-\tesac\n+\tif test \"$local_shared\" = yes\n+\tthen\n+\t\tmkdir -p \"$GIT_DIR/objects/info\"\n+\t\techo \"$repo/objects\" >>\"$GIT_DIR/objects/info/alternates\"\n+\telse\n+\t\tl= &&\n+\t\tif test \"$use_local_hardlink\" = yes\n+\t\tthen\n+\t\t\t# See if we can hardlink and drop \"l\" if not.\n+\t\t\tsample_file=$(cd \"$repo\" && \\\n+\t\t\t\t find objects -type f -print | sed -e 1q)\n+\t\t\t# objects directory should not be empty because\n+\t\t\t# we are cloning!\n+\t\t\ttest -f \"$repo/$sample_file\" || exit\n+\t\t\tif ln \"$repo/$sample_file\" \"$GIT_DIR/objects/sample\" 2>/dev/null\n+\t\t\tthen\n+\t\t\t\trm -f \"$GIT_DIR/objects/sample\"\n+\t\t\t\tl=l\n+\t\t\telse\n+\t\t\t\techo >&2 \"Warning: -l asked but cannot hardlink to $repo\"\n+\t\t\tfi\n+\t\tfi &&\n+\t\tcd \"$repo\" &&\n+\t\tfind objects -depth -print | cpio -pumd$l \"$GIT_DIR/\" || exit 1\n+\tfi\n \tgit-ls-remote \"$repo\" >\"$GIT_DIR/CLONE_HEAD\" || exit 1\n \t;;\n *)\ndiff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh\nindex 7da5153..7b6798d 100755\n--- a/t/t5500-fetch-pack.sh\n+++ b/t/t5500-fetch-pack.sh\n@@ -129,7 +129,7 @@ pull_to_client 2nd \"B\" $((64*3))\n \n pull_to_client 3rd \"A\" $((1*3)) # old fails\n \n-test_expect_success \"clone shallow\" \"git-clone --depth 2 . shallow\"\n+test_expect_success \"clone shallow\" \"git-clone --depth 2 file://`pwd`/. shallow\"\n \n (cd shallow; git count-objects -v) > count.shallow\n \ndiff --git a/t/t5700-clone-reference.sh b/t/t5700-clone-reference.sh\nindex 6d43252..4e93aaa 100755\n--- a/t/t5700-clone-reference.sh\n+++ b/t/t5700-clone-reference.sh\n@@ -51,7 +51,7 @@ diff expected current'\n cd \"$base_dir\"\n \n test_expect_success 'cloning with reference (no -l -s)' \\\n-'git clone --reference B A D'\n+'git clone --reference B file://`pwd`/A D'\n \n cd \"$base_dir\"\n \ndiff --git a/t/t5701-clone-local.sh b/t/t5701-clone-local.sh\nindex b093327..032c498 100755\n--- a/t/t5701-clone-local.sh\n+++ b/t/t5701-clone-local.sh\n@@ -43,4 +43,21 @@ test_expect_success 'local clone from x.git that does not exist' '\n \tfi\n '\n \n+test_expect_success 'Without -l, local will make a copy' '\n+\tcd \"$D\" &&\n+\tgit clone --bare x w &&\n+\tcd w &&\n+\tlinked=$(find objects -type f ! -links 1 | wc -l) &&\n+\ttest \"$linked\" = 0\n+'\n+\n+test_expect_success 'With -l, local will make a hardlink' '\n+\tcd \"$D\" &&\n+\trm -fr w &&\n+\tgit clone -l --bare x w &&\n+\tcd w &&\n+\tcopied=$(find objects -type f -links 1 | wc -l) &&\n+\ttest \"$copied\" = 0\n+'\n+\n test_done\n"},{"id":"49408","messageId":"86tzrikz5x.fsf@lola.quinscape.zz","threadId":"9334","inReplyTo":"7vejim1n92.fsf@assigned-by-dhcp.cox.net","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"David Kastrup","fromEmail":"dak@gnu.org","sentAt":"2007-08-02T10:29:14Z","receivedAt":"2007-08-02T10:29:14Z","isPatch":false,"sender":{"key":"dak@gnu.org","avatar":"https://avatars.githubusercontent.com/u/52141349?v=4"},"body":"Junio C Hamano writes:\n\n> * With -l, as long as the source repository is healthy, it is\n> very likely that the recipient would be, too. Also it is\n> very cheap. You do not get any back-up benefit.\n\nOh, but one does: an overzealous prune or rm -oopswrongoption in one\nrepo does not hurt the other.\n\n> Which leads me to believe that being able to use file:/// is\n> probably a good idea, if only for testability, but probably of\n> little practical value, and we can default to -l for everyday\n> use, and paranoids can use non -l as a way to make a back-up.\n\nSane enough, I guess.\n\n-- \nDavid Kastrup\n"},{"id":"49457","messageId":"200708021319.52565.jnareb@gmail.com","threadId":"9334","inReplyTo":"200708020018.18610.jnareb@gmail.com","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2007-08-02T11:19:51Z","receivedAt":"2007-08-02T11:19:51Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Jakub Narebski wrote:\n> Junio C Hamano wrote:\n> \n> > Perhaps if the destination is local,\n> > \n> > - if -s is given, just set up alternates, do nothing else;\n> > - by default, do \"always copy never hardlink\";\n> > - with -l, do \"hardlink if possible\";\n> > \n> > Hmmmm...\n> \n> That I think it is the best solution, together with support for\n> file:///path/to/repo.git scheme which would turn on old repacking\n> behavior. I'm all for it.\n\nBy the way, with \"-l\" you have hardlinks only till repack (\"git gc\"),\nisn't it?\n\n-- \nJakub Narebski\nPoland\n"},{"id":"49452","messageId":"46B21D82.5020802@ramsay1.demon.co.uk","threadId":"9334","inReplyTo":"7vr6mn5znm.fsf@assigned-by-dhcp.cox.net","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"Ramsay Jones","fromEmail":"ramsay@ramsay1.demon.co.uk","sentAt":"2007-08-02T18:08:02Z","receivedAt":"2007-08-02T18:08:02Z","isPatch":false,"sender":{"key":"ramsay@ramsayjones.plus.com","avatar":"https://avatars.githubusercontent.com/u/33702710?v=4"},"body":"Junio C Hamano wrote:\n> Are you suggesting to make -l the default for local, in other\n> words? I personally do not make local clone often enough that I\n> am not disturbed having to type extra \" -l\" on the command line.\n> \n> But giving a way to force \"copy not hardlink\" while still\n> avoiding \"the same as the networked case by doing pack transfer\"\n> overhead may be a good thing to do.\n> \n> Perhaps if the destination is local,\n> \n> - if -s is given, just set up alternates, do nothing else;\n> - by default, do \"always copy never hardlink\";\n> - with -l, do \"hardlink if possible\";\n> \n> Hmmmm...\n> \n\nAbout six weeks ago, I finally got around to installing Linux (ubuntu 7.04)\non my laptop. Naturally, I cloned my sparse and git repositories over from\nthe Windows XP partition. Unfortunately, that left me with a sparse repo that\nI could not modify; during the clone cpio copied the object directory, with\nperhaps a little too much fidelity, which resulted in a .git/objects tree\nwith 555 permissions (both files and directories). [It also set the file\ntimestamps with utime(), BTW]. A quick chmod fixed it up without problem,\nbut still ...\n\nWhen I cloned the git repo, however, I forgot the -l parameter and git-clone\neffectively did a \"git-fetch-pack --all -k $repo\", leaving me with a\nworking, and fully repacked, repository. Nice.\n\nSo, I was about to suggest that when invoked with -l, if the object database\ncannot be linked, due to EXDEV for example, it should fall back to the\n\"fetch-pack\" behaviour. Since I don't have access to a large repo, I can't\ncompare the filesystem-copy time versus the fetch-pack time for a \"realistic\"\nrepo, but I suppose the copy would always be faster. Oh Well.\n\nJust a data point.\n\nATB,\n\nRamsay Jones\n"},{"id":"49529","messageId":"7v3az1samx.fsf@assigned-by-dhcp.cox.net","threadId":"9334","inReplyTo":"86tzrikz5x.fsf@lola.quinscape.zz","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2007-08-03T00:51:50Z","receivedAt":"2007-08-03T00:51:50Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"David Kastrup writes:\n\n> Junio C Hamano writes:\n>\n>> * With -l, as long as the source repository is healthy, it is\n>> very likely that the recipient would be, too. Also it is\n>> very cheap. You do not get any back-up benefit.\n>\n> Oh, but one does: an overzealous prune or rm -oopswrongoption in one\n> repo does not hurt the other.\n\nThat's not \"back-up\" benefit I was thinking about. It is more\nabout protecting your data from hardware failure. You\nphysically have bits in two places, preferrably on separate disk\ndrives.\n\nAnd that is what you do not get from hardlinked clone.\n"},{"id":"49546","messageId":"858x8tw3em.fsf@lola.goethe.zz","threadId":"9334","inReplyTo":"7v3az1samx.fsf@assigned-by-dhcp.cox.net","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"David Kastrup","fromEmail":"dak@gnu.org","sentAt":"2007-08-03T06:14:25Z","receivedAt":"2007-08-03T06:14:25Z","isPatch":false,"sender":{"key":"dak@gnu.org","avatar":"https://avatars.githubusercontent.com/u/52141349?v=4"},"body":"Junio C Hamano writes:\n\n> David Kastrup writes:\n>\n>> Junio C Hamano writes:\n>>\n>>> * With -l, as long as the source repository is healthy, it is\n>>> very likely that the recipient would be, too. Also it is\n>>> very cheap. You do not get any back-up benefit.\n>>\n>> Oh, but one does: an overzealous prune or rm -oopswrongoption in one\n>> repo does not hurt the other.\n>\n> That's not \"back-up\" benefit I was thinking about. It is more\n> about protecting your data from hardware failure. You\n> physically have bits in two places, preferrably on separate disk\n> drives.\n>\n> And that is what you do not get from hardlinked clone.\n\nNot at the inode/blob level, but at least the directory manipulations\nof one are safe from the other.\n\n-- \nDavid Kastrup, Kriemhildstr. 15, 44793 Bochum\n"},{"id":"49560","messageId":"200708031020.43492.johan@herland.net","threadId":"9334","inReplyTo":"7v3az1samx.fsf@assigned-by-dhcp.cox.net","subject":"Re: Git benchmark - comparison with Bazaar, Darcs, Git and Mercurial","fromName":"Johan Herland","fromEmail":"johan@herland.net","sentAt":"2007-08-03T08:20:43Z","receivedAt":"2007-08-03T08:20:43Z","isPatch":false,"sender":{"key":"johan@herland.net","avatar":"https://avatars.githubusercontent.com/u/547031?v=4"},"body":"On Friday 03 August 2007, Junio C Hamano wrote:\n> David Kastrup writes:\n> \n> > Junio C Hamano writes:\n> >\n> >> * With -l, as long as the source repository is healthy, it is\n> >> very likely that the recipient would be, too. Also it is\n> >> very cheap. You do not get any back-up benefit.\n> >\n> > Oh, but one does: an overzealous prune or rm -oopswrongoption in one\n> > repo does not hurt the other.\n> \n> That's not \"back-up\" benefit I was thinking about. It is more\n> about protecting your data from hardware failure.\n\nIf one is serious about backing up ones repo to protect it from hardware \nfailure, there is not much use at all in cloning (by copy, hardlink, or \notherwise) to a different location on the _same_ filesystem. In order for a \nbackup to be at least marginally useful, it should be on a different disk \ndrive (which you hint at below), or even better; on a different \ncontinent...\n\nMy point is as follows: One has to clone a repo onto (at least) a different \nfilesystem if one is serious about backup. But if one is cloning to a \ndifferent filesystem, hardlinking is no longer an option; git _has_ to make \na copy of some sort. Therefore we might as well hardlink as long as we're \non a single filesystem (since the extra copy would not be worth much, \nbackup-wise).\n\n> You physically have bits in two places, preferrably on separate disk\n> drives.\n> And that is what you do not get from hardlinked clone.\n\nIf the two copies are on separate disk drives (i.e. separate filesystems), \nyou cannot make a hardlink in the first place. If the two copies are on the \nsame filesystem, they're not much more worth than a single copy \n(backup-wise).\n\nGiven the clone-to-same-filesystem(-with-hardlink-capability) scenario \n(which is the only scenario where we have the option of using hardlinks), \nwe have the following pros and cons when using hardlinks instead of \ncopying:\n\nPros:\n- Hardlink is _much_ faster (for big repos, we're talking orders of \nmagnitude faster)\n\nCons:\n- Hardlink will not leave two copies on the disk. But I'm arguing that the \nadditional copy will have pretty much _no_ value from a redundancy POV, \nsince the copy is still left on the _same_ filesystem. Some would even go \nas far as to say that the second copy provides a false sense of security as \nlong as it is located on the same filesystem.\n\n\nHave fun!\n\n...Johan\n\n-- \nJohan Herland, \nwww.herland.net\n"}]}