{"thread":{"id":"6771","subject":"Efficiency of initial clone from server","startedAt":"2007-02-11T19:53:47Z","lastAt":"2007-02-13T15:03:08Z","messageCount":32,"participants":["Jon Smirl","Shawn O. Pearce","Nicolas Pitre","Junio C Hamano","Johannes Schindelin","Theodore Tso","Jakub Narebski","Andreas Ericsson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"34186","messageId":"9e4733910702111153p1691ad99nda97325b34b7a13f@mail.gmail.com","threadId":"6771","inReplyTo":null,"subject":"Efficiency of initial clone from server","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2007-02-11T19:53:47Z","receivedAt":"2007-02-11T19:53:47Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"I'm doing a clone right now and I see this:\n\njonsmirl@jonsmirl:/extra$ cg clone\ngit://git2.kernel.org/pub/scm/linux/kernel/git/linville/wireless-dev.git\nInitialized empty Git repository in .git/\nFetching pack (head and objects)...\nremote: Generating pack...\nremote: Done counting 404120 objects.\nremote: Deltifying 404120 objects.\nremote: 100% (404120/404120) done\nIndexing 404120 objects.\nremote: Total 404120, written 404120 (delta 320324), reused 365290\n(delta 282572)\n 100% (404120/404120) done\nResolving 320324 deltas.\n....\n\nIs this happening because the repository on the server is not\ncompletely packed? It is basically building a pack of the whole thing\nand shipping it to me, right?\n\nIf that is the case, why not first pack the whole repository and then\ncopy it down the wire? Now the next clone that comes along doesn't\nhave to do so much work. Would this help to eliminate some of the load\nat kernel.org?\n\nSome this is wrong with this tree too, what are these errors about\nfatal: pack: not a valid SHA1\n\nfatal: pack: not a valid SHA1\nFetching tags... v2.6.12 v2.6.12-rc2 v2.6.12-rc3 v2.6.12-rc4\nv2.6.12-rc5 v2.6.12-rc6 v2.6.13 v2.6.13-rc1 v2.6.13-rc2 v2.6.13-rc3\nv2.6.13-rc4 v2.6.13-rc5 v2.6.13-rc6 v2.6.13-rc7 v2.6.14 v2.6.14-rc1\nv2.6.14-rc2 v2.6.14-rc3 v2.6.14-rc4 v2.6.14-rc5 v2.6.15 v2.6.15-rc1\nv2.6.15-rc2 v2.6.15-rc3 v2.6.15-rc4 v2.6.15-rc5 v2.6.15-rc6\nv2.6.15-rc7 v2.6.16 v2.6.16-rc1 v2.6.16-rc2 v2.6.16-rc3 v2.6.16-rc4\nv2.6.16-rc5 v2.6.16-rc6 v2.6.17 v2.6.17-rc1 v2.6.17-rc2 v2.6.17-rc3\nv2.6.17-rc4 v2.6.17-rc5 v2.6.17-rc6 v2.6.18 v2.6.18-rc1 v2.6.18-rc2\nv2.6.18-rc3 v2.6.18-rc4 v2.6.18-rc5 v2.6.18-rc6 v2.6.18-rc7 v2.6.19\nv2.6.19-rc1 v2.6.19-rc2 v2.6.19-rc3 v2.6.19-rc4 v2.6.19-rc5\nv2.6.19-rc6 v2.6.20-rc1 v2.6.20-rc2 v2.6.20-rc3 v2.6.20-rc4\nv2.6.20-rc5 v2.6.20-rc6\nremote: Generating pack...\nremote: Done counting 63 objects.\nremote: Deltifying 63 objects.\nremote: 100% (63/63) done\nIndexing 63 objects.\nremote: Total 63, written 63 (delta 0), reused 63 (delta 0)\n 100% (63/63) done\nfatal: pack: not a valid SHA1\nNew branch: 0953670fbcb75e26fb93340bddae934e85618f2e\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"34201","messageId":"20070211225326.GC31488@spearce.org","threadId":"6771","inReplyTo":"9e4733910702111153p1691ad99nda97325b34b7a13f@mail.gmail.com","subject":"Re: Efficiency of initial clone from server","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-02-11T22:53:26Z","receivedAt":"2007-02-11T22:53:26Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jon Smirl wrote:\n> Is this happening because the repository on the server is not\n> completely packed? It is basically building a pack of the whole thing\n> and shipping it to me, right?\n\nCorrect. The wire protocol only allows us to send one pack.\nSo we have to pack everything and transmit it as a single unit.\n \n> If that is the case, why not first pack the whole repository and then\n> copy it down the wire? Now the next clone that comes along doesn't\n> have to do so much work. Would this help to eliminate some of the load\n> at kernel.org?\n\nProbably, but then the daemon needs write access to the repository.\nThis isn't required right now; it can be strictly read-only and\nstill serve the contents.\n \n> remote: Total 63, written 63 (delta 0), reused 63 (delta 0)\n> 100% (63/63) done\n> fatal: pack: not a valid SHA1\n> New branch: 0953670fbcb75e26fb93340bddae934e85618f2e\n\nWhat version of git is this? That looks like we're assuming the word\npack was an object, but I'm not sure why we would do such a thing...\n\n-- \nShawn.\n"},{"id":"34209","messageId":"9e4733910702111525x176053d3y9fd6d809ac447c0a@mail.gmail.com","threadId":"6771","inReplyTo":"20070211225326.GC31488@spearce.org","subject":"Re: Efficiency of initial clone from server","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2007-02-11T23:25:59Z","receivedAt":"2007-02-11T23:25:59Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 2/11/07, Shawn O. Pearce wrote:\n> Jon Smirl wrote:\n> > Is this happening because the repository on the server is not\n> > completely packed? It is basically building a pack of the whole thing\n> > and shipping it to me, right?\n>\n> Correct. The wire protocol only allows us to send one pack.\n> So we have to pack everything and transmit it as a single unit.\n>\n> > If that is the case, why not first pack the whole repository and then\n> > copy it down the wire? Now the next clone that comes along doesn't\n> > have to do so much work. Would this help to eliminate some of the load\n> > at kernel.org?\n>\n> Probably, but then the daemon needs write access to the repository.\n> This isn't required right now; it can be strictly read-only and\n> still serve the contents.\n>\n> > remote: Total 63, written 63 (delta 0), reused 63 (delta 0)\n> > 100% (63/63) done\n> > fatal: pack: not a valid SHA1\n> > New branch: 0953670fbcb75e26fb93340bddae934e85618f2e\n>\n> What version of git is this? That looks like we're assuming the word\n> pack was an object, but I'm not sure why we would do such a thing...\n\njonsmirl@jonsmirl:/usr/local/bin$ git --version\ngit version 1.5.0.rc2.g53551-dirty\n\n\n>\n> --\n> Shawn.\n>\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"34210","messageId":"9e4733910702111529g715c7c0ch9246947fd70d40eb@mail.gmail.com","threadId":"6771","inReplyTo":"20070211225326.GC31488@spearce.org","subject":"Re: Efficiency of initial clone from server","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2007-02-11T23:29:00Z","receivedAt":"2007-02-11T23:29:00Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 2/11/07, Shawn O. Pearce wrote:\n> Jon Smirl wrote:\n> > Is this happening because the repository on the server is not\n> > completely packed? It is basically building a pack of the whole thing\n> > and shipping it to me, right?\n>\n> Correct. The wire protocol only allows us to send one pack.\n> So we have to pack everything and transmit it as a single unit.\n>\n> > If that is the case, why not first pack the whole repository and then\n> > copy it down the wire? Now the next clone that comes along doesn't\n> > have to do so much work. Would this help to eliminate some of the load\n> > at kernel.org?\n>\n> Probably, but then the daemon needs write access to the repository.\n> This isn't required right now; it can be strictly read-only and\n> still serve the contents.\n\nDoes it need write access for push to work, can the server check for\nwrite access and save the complete repack if it has access? This\nappears to be causing a lot of needless work for kernel.org.\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"34217","messageId":"9e4733910702111551t3935fb88yf19e3ea608094687@mail.gmail.com","threadId":"6771","inReplyTo":"9e4733910702111525x176053d3y9fd6d809ac447c0a@mail.gmail.com","subject":"Re: Efficiency of initial clone from server","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2007-02-11T23:51:05Z","receivedAt":"2007-02-11T23:51:05Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 2/11/07, Jon Smirl wrote:\n> > > remote: Total 63, written 63 (delta 0), reused 63 (delta 0)\n> > > 100% (63/63) done\n> > > fatal: pack: not a valid SHA1\n> > > New branch: 0953670fbcb75e26fb93340bddae934e85618f2e\n> >\n> > What version of git is this? That looks like we're assuming the word\n> > pack was an object, but I'm not sure why we would do such a thing...\n>\n> jonsmirl@jonsmirl:/usr/local/bin$ git --version\n> git version 1.5.0.rc2.g53551-dirty\n>\n\nI just whacked my git tree, cloned a new copy and installed it. Still\nget the same errors.\n\njonsmirl@jonsmirl:/extra$ rm -rf wireless-dev\ngjonsmirl@jonsmirl:/extra$ cg clone\ngit://git2.kernel.org/pub/scm/linux/kernel/gt/linville/wireless-dev.git\nInitialized empty Git repository in .git/\nFetching pack (head and objects)...\nremote: Generating pack...\nremote: Done counting 404120 objects.\nremote: Deltifying 404120 objects.\nremote: 100% (404120/404120) done\nIndexing 404120 objects.\nremote: Total 404120, written 404120 (delta 320324), reused 365290\n(delta 282572)\n 100% (404120/404120) done\nResolving 320324 deltas.\n 100% (320324/320324) done\nfatal: pack: not a valid SHA1\nFetching tags... v2.6.12 v2.6.12-rc2 v2.6.12-rc3 v2.6.12-rc4\nv2.6.12-rc5 v2.6.12-rc6 v2.6.13 v2.6.13-rc1 v2.6.13-rc2 v2.6.13-rc3\nv2.6.13-rc4 v2.6.13-rc5 v2.6.13-rc6 v2.6.13-rc7 v2.6.14 v2.6.14-rc1\nv2.6.14-rc2 v2.6.14-rc3 v2.6.14-rc4 v2.6.14-rc5 v2.6.15 v2.6.15-rc1\nv2.6.15-rc2 v2.6.15-rc3 v2.6.15-rc4 v2.6.15-rc5 v2.6.15-rc6\nv2.6.15-rc7 v2.6.16 v2.6.16-rc1 v2.6.16-rc2 v2.6.16-rc3 v2.6.16-rc4\nv2.6.16-rc5 v2.6.16-rc6 v2.6.17 v2.6.17-rc1 v2.6.17-rc2 v2.6.17-rc3\nv2.6.17-rc4 v2.6.17-rc5 v2.6.17-rc6 v2.6.18 v2.6.18-rc1 v2.6.18-rc2\nv2.6.18-rc3 v2.6.18-rc4 v2.6.18-rc5 v2.6.18-rc6 v2.6.18-rc7 v2.6.19\nv2.6.19-rc1 v2.6.19-rc2 v2.6.19-rc3 v2.6.19-rc4 v2.6.19-rc5\nv2.6.19-rc6 v2.6.20-rc1 v2.6.20-rc2 v2.6.20-rc3 v2.6.20-rc4\nv2.6.20-rc5 v2.6.20-rc6\nremote: Generating pack...\nremote: Done counting 63 objects.\nremote: Deltifying 63 objects.\nremote: 100% (63/63) done\nIndexing 63 objects.\nremote: Total 63, written 63 (delta 0), reused 63 (delta 0)\n 100% (63/63) done\nfatal: pack: not a valid SHA1\nNew branch: 0953670fbcb75e26fb93340bddae934e85618f2e\nCloned to wireless-dev/ (origin\ngit://git2.kernel.org/pub/scm/linux/kernel/git/linville/wireless-dev.git\navailable as branch \"origin\")\njonsmirl@jonsmirl:/extra$ git --version\ngit version 1.5.0.rc4.g1843e\njonsmirl@jonsmirl:/extra$ cg --version\ncogito-0.18.2 (cogito-0.18rc1-gb6a6e87)\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"34229","messageId":"Pine.LNX.4.64.0702112022430.1757@xanadu.home","threadId":"6771","inReplyTo":"9e4733910702111525x176053d3y9fd6d809ac447c0a@mail.gmail.com","subject":"Re: Efficiency of initial clone from server","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-02-12T01:38:09Z","receivedAt":"2007-02-12T01:38:09Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sun, 11 Feb 2007, Jon Smirl wrote:\n\n> On 2/11/07, Shawn O. Pearce wrote:\n> > Jon Smirl wrote:\n> > > remote: Total 63, written 63 (delta 0), reused 63 (delta 0)\n> > > 100% (63/63) done\n> > > fatal: pack: not a valid SHA1\n> > > New branch: 0953670fbcb75e26fb93340bddae934e85618f2e\n> >\n> > What version of git is this? That looks like we're assuming the word\n> > pack was an object, but I'm not sure why we would do such a thing...\n\nThis \"pack\" comes from pack-index when providing the name of the pack.\nIt is either \"pack\" or \"keep\" and specifies the name of the .keep file \nto remove in the later case.\nThis is caught by git-fetch.sh with some code identified with a comment \nthat reads: \"# special line coming from index-pack with the pack name\"\n\n> jonsmirl@jonsmirl:/usr/local/bin$ git --version\n> git version 1.5.0.rc2.g53551-dirty\n\nYou must have conflicting vintage of GIT installations on your machine \nwith missing support for the \"pack\" and \"keep\" stuff described above.\n\n\nNicolas\n"},{"id":"34231","messageId":"9e4733910702111815g5a42989fr8042e84b291ab576@mail.gmail.com","threadId":"6771","inReplyTo":"Pine.LNX.4.64.0702112022430.1757@xanadu.home","subject":"Re: Efficiency of initial clone from server","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2007-02-12T02:15:35Z","receivedAt":"2007-02-12T02:15:35Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 2/11/07, Nicolas Pitre wrote:\n> On Sun, 11 Feb 2007, Jon Smirl wrote:\n>\n> > On 2/11/07, Shawn O. Pearce wrote:\n> > > Jon Smirl wrote:\n> > > > remote: Total 63, written 63 (delta 0), reused 63 (delta 0)\n> > > > 100% (63/63) done\n> > > > fatal: pack: not a valid SHA1\n> > > > New branch: 0953670fbcb75e26fb93340bddae934e85618f2e\n> > >\n> > > What version of git is this? That looks like we're assuming the word\n> > > pack was an object, but I'm not sure why we would do such a thing...\n>\n> This \"pack\" comes from pack-index when providing the name of the pack.\n> It is either \"pack\" or \"keep\" and specifies the name of the .keep file\n> to remove in the later case.\n> This is caught by git-fetch.sh with some code identified with a comment\n> that reads: \"# special line coming from index-pack with the pack name\"\n>\n> > jonsmirl@jonsmirl:/usr/local/bin$ git --version\n> > git version 1.5.0.rc2.g53551-dirty\n>\n> You must have conflicting vintage of GIT installations on your machine\n> with missing support for the \"pack\" and \"keep\" stuff described above.\n\nI can clone Linus' git tree without getting errors.\n\nMaybe there is something wrong with the wireless-dev tree.\nIt looks like someone is working on it:\n\njonsmirl@jonsmirl:/extra$ git clone\ngit://git.kernel.org/pub/scm/linux/kernel/gt/linville/wireless-dev.git\nInitialized empty Git repository in /extra/wireless-dev/.git/\nfatal: The remote end hung up unexpectedly\nfetch-pack from\n'git://git.kernel.org/pub/scm/linux/kernel/gt/linville/wireless-dev.git'\nfailed.\njonsmirl@jonsmirl:/extra$\n\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"34236","messageId":"Pine.LNX.4.64.0702112251580.1757@xanadu.home","threadId":"6771","inReplyTo":"9e4733910702111815g5a42989fr8042e84b291ab576@mail.gmail.com","subject":"Re: Efficiency of initial clone from server","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-02-12T03:55:54Z","receivedAt":"2007-02-12T03:55:54Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sun, 11 Feb 2007, Jon Smirl wrote:\n\n> Maybe there is something wrong with the wireless-dev tree.\n> It looks like someone is working on it:\n> \n> jonsmirl@jonsmirl:/extra$ git clone\n> git://git.kernel.org/pub/scm/linux/kernel/gt/linville/wireless-dev.git\n> Initialized empty Git repository in /extra/wireless-dev/.git/\n> fatal: The remote end hung up unexpectedly\n> fetch-pack from\n> 'git://git.kernel.org/pub/scm/linux/kernel/gt/linville/wireless-dev.git'\n> failed.\n\nTry with git2.kernel.org then. It seems that git.kernel.org too often \nresolves to git1.kernel.org not sharing the load with git2.kernel.org \nappropriately.\n\n\nNicolas\n"},{"id":"34238","messageId":"7vd54gau3r.fsf@assigned-by-dhcp.cox.net","threadId":"6771","inReplyTo":"Pine.LNX.4.64.0702112022430.1757@xanadu.home","subject":"Re: Efficiency of initial clone from server","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-02-12T04:16:24Z","receivedAt":"2007-02-12T04:16:24Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Nicolas Pitre writes:\n\n> On Sun, 11 Feb 2007, Jon Smirl wrote:\n>\n>> On 2/11/07, Shawn O. Pearce wrote:\n>> > Jon Smirl wrote:\n>> > > remote: Total 63, written 63 (delta 0), reused 63 (delta 0)\n>> > > 100% (63/63) done\n>> > > fatal: pack: not a valid SHA1\n>> > > New branch: 0953670fbcb75e26fb93340bddae934e85618f2e\n>> >\n>> > What version of git is this? That looks like we're assuming the word\n>> > pack was an object, but I'm not sure why we would do such a thing...\n>\n> This \"pack\" comes from pack-index when providing the name of the pack.\n> It is either \"pack\" or \"keep\" and specifies the name of the .keep file \n> to remove in the later case.\n> This is caught by git-fetch.sh with some code identified with a comment \n> that reads: \"# special line coming from index-pack with the pack name\"\n\nThat is true only if Jon used git-fetch, git-pull or git-clone.\nUnfortunately I noticed that his commandline read \"cg clone\".\n"},{"id":"34241","messageId":"9e4733910702112029k6ef3fb7bl3bdb134a787512e8@mail.gmail.com","threadId":"6771","inReplyTo":"7vd54gau3r.fsf@assigned-by-dhcp.cox.net","subject":"Re: Efficiency of initial clone from server","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2007-02-12T04:29:19Z","receivedAt":"2007-02-12T04:29:19Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 2/11/07, Junio C Hamano wrote:\n> Nicolas Pitre writes:\n>\n> > On Sun, 11 Feb 2007, Jon Smirl wrote:\n> >\n> >> On 2/11/07, Shawn O. Pearce wrote:\n> >> > Jon Smirl wrote:\n> >> > > remote: Total 63, written 63 (delta 0), reused 63 (delta 0)\n> >> > > 100% (63/63) done\n> >> > > fatal: pack: not a valid SHA1\n> >> > > New branch: 0953670fbcb75e26fb93340bddae934e85618f2e\n> >> >\n> >> > What version of git is this? That looks like we're assuming the word\n> >> > pack was an object, but I'm not sure why we would do such a thing...\n> >\n> > This \"pack\" comes from pack-index when providing the name of the pack.\n> > It is either \"pack\" or \"keep\" and specifies the name of the .keep file\n> > to remove in the later case.\n> > This is caught by git-fetch.sh with some code identified with a comment\n> > that reads: \"# special line coming from index-pack with the pack name\"\n>\n> That is true only if Jon used git-fetch, git-pull or git-clone.\n> Unfortunately I noticed that his commandline read \"cg clone\".\n\nAll wireless-dev are all dead.\nI can still get to Linus' tree without problem. wireless-2.6 tree is ok too.\n\njonsmirl@jonsmirl:/extra$ git clone\ngit://git.kernel.org/pub/scm/linux/kernel/gt/linville/wireless-dev.git\nInitialized empty Git repository in /extra/wireless-dev/.git/\nfatal: The remote end hung up unexpectedly\nfetch-pack from\n'git://git.kernel.org/pub/scm/linux/kernel/gt/linville/wireless-dev.git'\nfailed.\njonsmirl@jonsmirl:/extra$ git clone\ngit://git2.kernel.org/pub/scm/linux/kernel/gt/linville/wireless-dev.git\nInitialized empty Git repository in /extra/wireless-dev/.git/\nfatal: The remote end hung up unexpectedly\nfetch-pack from\n'git://git2.kernel.org/pub/scm/linux/kernel/gt/linville/wireless-dev.git'\nfailed.\njonsmirl@jonsmirl:/extra$ git clone\ngit://git1.kernel.org/pub/scm/linux/kernel/gt/linville/wireless-dev.git\nInitialized empty Git repository in /extra/wireless-dev/.git/\nfatal: The remote end hung up unexpectedly\nfetch-pack from\n'git://git1.kernel.org/pub/scm/linux/kernel/gt/linville/wireless-dev.git'\nfailed.\njonsmirl@jonsmirl:/extra$\n\n\nClone of the git repo complete without errors\n\njonsmirl@jonsmirl:/extra$ git clone git://git2.kernel.org/pub/scm/git/git.git\nInitialized empty Git repository in /extra/git/.git/\nremote: Generating pack...\nremote: Done counting 39677 objects.\nremote: Deltifying 39677 objects.\nremote: 100% (39677/39677) done\nIndexing 39677 objects.\nremote: Total 39677, written 39677 (delta 27701), reused 39585 (delta 27642)\n 100% (39677/39677) done\nResolving 27701 deltas.\n 100% (27701/27701) done\nChecking files out...\n 100% (798/798) done\njonsmirl@jonsmirl:/extra$\n\nI think something is wrong with the wireless-dev tree.\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"34242","messageId":"7v4ppsatbr.fsf@assigned-by-dhcp.cox.net","threadId":"6771","inReplyTo":"9e4733910702112029k6ef3fb7bl3bdb134a787512e8@mail.gmail.com","subject":"Re: Efficiency of initial clone from server","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-02-12T04:33:12Z","receivedAt":"2007-02-12T04:33:12Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jon Smirl\" writes:\n\n> All wireless-dev are all dead.\n> I can still get to Linus' tree without problem. wireless-2.6 tree is ok too.\n>\n> jonsmirl@jonsmirl:/extra$ git clone\n> git://git.kernel.org/pub/scm/linux/kernel/gt/linville/wireless-dev.git\n> Initialized empty Git repository in /extra/wireless-dev/.git/\n> fatal: The remote end hung up unexpectedly\n\nAre you sure the above is \".../linux/kernel/gt/linville/...\"?\n"},{"id":"34243","messageId":"20070212044914.GA699@spearce.org","threadId":"6771","inReplyTo":"Pine.LNX.4.64.0702112251580.1757@xanadu.home","subject":"Re: Efficiency of initial clone from server","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-02-12T04:49:15Z","receivedAt":"2007-02-12T04:49:15Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Nicolas Pitre wrote:\n> Try with git2.kernel.org then. It seems that git.kernel.org too often \n> resolves to git1.kernel.org not sharing the load with git2.kernel.org \n> appropriately.\n\nHey! You are giving out my secret!\n\n(I use git2.kernel.org to fetch git.git.)\n\n-- \nShawn.\n"},{"id":"34244","messageId":"9e4733910702112053q2c196ddcr970a062281d332b7@mail.gmail.com","threadId":"6771","inReplyTo":"7v4ppsatbr.fsf@assigned-by-dhcp.cox.net","subject":"Re: Efficiency of initial clone from server","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2007-02-12T04:53:25Z","receivedAt":"2007-02-12T04:53:25Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 2/11/07, Junio C Hamano wrote:\n> \"Jon Smirl\" writes:\n>\n> > All wireless-dev are all dead.\n> > I can still get to Linus' tree without problem. wireless-2.6 tree is ok too.\n> >\n> > jonsmirl@jonsmirl:/extra$ git clone\n> > git://git.kernel.org/pub/scm/linux/kernel/gt/linville/wireless-dev.git\n> > Initialized empty Git repository in /extra/wireless-dev/.git/\n> > fatal: The remote end hung up unexpectedly\n>\n> Are you sure the above is \".../linux/kernel/gt/linville/...\"?\n\nYou're right it should be git instead of gt, somewhere in my copying\nstrings around I lost the 'i' and wasn't paying attention. That's not\na very good error message: \"fatal: The remote end hung up\nunexpectedly\" for a missing repository.\n\nHere's a git clone with the right string. It works:\n\njonsmirl@jonsmirl:/extra$ git clone\ngit://git2.kernel.org/pub/scm/linux/kernel/git/linville/wireless-dev.git\nInitialized empty Git repository in /extra/wireless-dev/.git/\nremote: Generating pack...\nremote: Done counting 411874 objects.\nremote: Deltifying 411874 objects.\nremote: 100% (411874/411874) done\nIndexing 411874 objects.\nremote: Total 411874, written 411874 (delta 327049), reused 374287\n(delta 289485)\n 100% (411874/411874) done\nResolving 327049 deltas.\n 100% (327049/327049) done\nChecking files out...\n 100% (21429/21429) done\njonsmirl@jonsmirl:/extra$\n\nSame thing with cg clone, it's what is broken.\ncg update is broken in the same way.\nI'm using the current git version of cogitio.\nI'll switch to the git commands, git clone is about 10x faster for the\nclone anyway.\n\njonsmirl@jonsmirl:/extra$ cg clone\ngit://git2.kernel.org/pub/scm/linux/kernel/git/linville/wireless-dev.git\nInitialized empty Git repository in .git/\nFetching pack (head and objects)...\nremote: Generating pack...\nremote: Done counting 404120 objects.\nremote: Deltifying 404120 objects.\nremote: 100% (404120/404120) done\nIndexing 404120 objects.\nremote: Total 404120, written 404120 (delta 320324), reused 365290\n(delta 282572)\n 100% (404120/404120) done\nResolving 320324 deltas.\n 100% (320324/320324) done\nfatal: pack: not a valid SHA1\nFetching tags... v2.6.12 v2.6.12-rc2 v2.6.12-rc3 v2.6.12-rc4\nv2.6.12-rc5 v2.6.12-rc6 v2.6.13 v2.6.13-rc1 v2.6.13-rc2 v2.6.13-rc3\nv2.6.13-rc4 v2.6.13-rc5 v2.6.13-rc6 v2.6.13-rc7 v2.6.14 v2.6.14-rc1\nv2.6.14-rc2 v2.6.14-rc3 v2.6.14-rc4 v2.6.14-rc5 v2.6.15 v2.6.15-rc1\nv2.6.15-rc2 v2.6.15-rc3 v2.6.15-rc4 v2.6.15-rc5 v2.6.15-rc6\nv2.6.15-rc7 v2.6.16 v2.6.16-rc1 v2.6.16-rc2 v2.6.16-rc3 v2.6.16-rc4\nv2.6.16-rc5 v2.6.16-rc6 v2.6.17 v2.6.17-rc1 v2.6.17-rc2 v2.6.17-rc3\nv2.6.17-rc4 v2.6.17-rc5 v2.6.17-rc6 v2.6.18 v2.6.18-rc1 v2.6.18-rc2\nv2.6.18-rc3 v2.6.18-rc4 v2.6.18-rc5 v2.6.18-rc6 v2.6.18-rc7 v2.6.19\nv2.6.19-rc1 v2.6.19-rc2 v2.6.19-rc3 v2.6.19-rc4 v2.6.19-rc5\nv2.6.19-rc6 v2.6.20-rc1 v2.6.20-rc2 v2.6.20-rc3 v2.6.20-rc4\nv2.6.20-rc5 v2.6.20-rc6\nremote: Generating pack...\nremote: Done counting 63 objects.\nremote: Deltifying 63 objects.\nremote: 100% (63/63) done\nIndexing 63 objects.\nremote: Total 63, written 63 (delta 0), reused 63 (delta 0)\n 100% (63/63) done\nfatal: pack: not a valid SHA1\nNew branch: 0953670fbcb75e26fb93340bddae934e85618f2e\nCloned to wireless-dev/ (origin\ngit://git2.kernel.org/pub/scm/linux/kernel/git/linville/wireless-dev.git\navailable as branch \"origin\")\njonsmirl@jonsmirl:/extra$\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"34245","messageId":"9e4733910702112101h256b1468j5de65c6e89109a22@mail.gmail.com","threadId":"6771","inReplyTo":"9e4733910702112053q2c196ddcr970a062281d332b7@mail.gmail.com","subject":"Re: Efficiency of initial clone from server","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2007-02-12T05:01:41Z","receivedAt":"2007-02-12T05:01:41Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 2/11/07, Jon Smirl wrote:\n> Same thing with cg clone, it's what is broken.\n> cg update is broken in the same way.\n> I'm using the current git version of cogitio.\n> I'll switch to the git commands, git clone is about 10x faster for the\n> clone anyway.\n\nDon't read anything into the 10x speed diff, my last git clone was\nreally slow. I'm probably fighting other people at kernel.org to keep\nthe tree in RAM.\n\nBut pack to the original point, can't the server check and see if it\nhas write access so that it can keep the fully packed tree? I've just\ncaused kernel.org to needlessly repack the wireless-dev tree a dozen\ntimes playing with this clone command. If it didn't have to keep\nrepacking for the clone, clone would be a lot faster.\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"34246","messageId":"20070212051108.GB699@spearce.org","threadId":"6771","inReplyTo":"9e4733910702112101h256b1468j5de65c6e89109a22@mail.gmail.com","subject":"Re: Efficiency of initial clone from server","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-02-12T05:11:08Z","receivedAt":"2007-02-12T05:11:08Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jon Smirl wrote:\n> But pack to the original point, can't the server check and see if it\n> has write access so that it can keep the fully packed tree? I've just\n> caused kernel.org to needlessly repack the wireless-dev tree a dozen\n> times playing with this clone command. If it didn't have to keep\n> repacking for the clone, clone would be a lot faster.\n\nWe probably could.\n\nI have actually been thinking about another problem that is\nsomewhat related. We cannot put more than 4 GiB of data into a\nsingle packfile, due to the current index size limitation, or more\nthan 2^32-1 objects into one packfile, due to the header nr_objects\nfield size.\n\nRight now we are sending a single packfile down to the client,\neven if the remote server end has the repository broken down into\na couple of packfiles (such as \"really old historical stuff\" and\n\"active stuff from this year\"). If we could send more than one\npackfile to the client in a single stream, we could still keep the\nfile size limitations.\n\nWe can also avoid this huge repack case on the server. Because it\ncould just send all of the packfiles that it already has, followed\nby whatever is loose which wasn't in a prior packfile. And no\nwrite access required.\n\nOf course, we still could do the optimization of caching the\npackfile, but I'm not sure how well that would work on kernel.org,\nas I understand the trees are owned by the devs which created them\nwhile the git daemon is probably not running as their UNIX user.\n\n-- \nShawn.\n"},{"id":"34247","messageId":"9e4733910702112117o53630946ja50572c7c7f2b2c1@mail.gmail.com","threadId":"6771","inReplyTo":"20070212051108.GB699@spearce.org","subject":"Re: Efficiency of initial clone from server","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2007-02-12T05:17:19Z","receivedAt":"2007-02-12T05:17:19Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 2/12/07, Shawn O. Pearce wrote:\n> Jon Smirl wrote:\n> > But pack to the original point, can't the server check and see if it\n> > has write access so that it can keep the fully packed tree? I've just\n> > caused kernel.org to needlessly repack the wireless-dev tree a dozen\n> > times playing with this clone command. If it didn't have to keep\n> > repacking for the clone, clone would be a lot faster.\n>\n> We probably could.\n>\n> I have actually been thinking about another problem that is\n> somewhat related. We cannot put more than 4 GiB of data into a\n> single packfile, due to the current index size limitation, or more\n> than 2^32-1 objects into one packfile, due to the header nr_objects\n> field size.\n>\n> Right now we are sending a single packfile down to the client,\n> even if the remote server end has the repository broken down into\n> a couple of packfiles (such as \"really old historical stuff\" and\n> \"active stuff from this year\"). If we could send more than one\n> packfile to the client in a single stream, we could still keep the\n> file size limitations.\n>\n> We can also avoid this huge repack case on the server. Because it\n> could just send all of the packfiles that it already has, followed\n> by whatever is loose which wasn't in a prior packfile. And no\n> write access required.\n>\n> Of course, we still could do the optimization of caching the\n> packfile, but I'm not sure how well that would work on kernel.org,\n> as I understand the trees are owned by the devs which created them\n> while the git daemon is probably not running as their UNIX user.\n\nI didn't want to cache the packfile, instead I wanted to repack the\nrepository and then copy the resulting pack file down the wire. A\nclone would just be a trigger to make sure everything in the repo was\npacked (maybe into multiple packs) before starting to send anything.\nDoing it this way means that everyone benefits from the packing.\n\n\n>\n> --\n> Shawn.\n>\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"34248","messageId":"7vzm7k9c3c.fsf@assigned-by-dhcp.cox.net","threadId":"6771","inReplyTo":"9e4733910702112101h256b1468j5de65c6e89109a22@mail.gmail.com","subject":"Re: Efficiency of initial clone from server","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-02-12T05:30:47Z","receivedAt":"2007-02-12T05:30:47Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jon Smirl\" writes:\n\n> On 2/11/07, Jon Smirl wrote:\n>> Same thing with cg clone, it's what is broken.\n>> cg update is broken in the same way.\n>> I'm using the current git version of cogitio.\n>> I'll switch to the git commands, git clone is about 10x faster for the\n>> clone anyway.\n>\n> Don't read anything into the 10x speed diff, my last git clone was\n> really slow. I'm probably fighting other people at kernel.org to keep\n> the tree in RAM.\n>\n> But pack to the original point, can't the server check and see if it\n> has write access so that it can keep the fully packed tree? I've just\n> caused kernel.org to needlessly repack the wireless-dev tree a dozen\n> times playing with this clone command. If it didn't have to keep\n> repacking for the clone, clone would be a lot faster.\n\nYou are assuming everybody does initial clone all the time. I\ndo not think that holds true in practice.\n\nFor something like tglx historical tree that will _never_\nchange, there is a specific hack the repository owner can take\nadvantage of to always feed a prepackaged pack, although its use\nis not advertised well enough (and I do not think it buys much\nin practice).\n"},{"id":"34251","messageId":"9e4733910702112155m3f5da9abgbeb2b6783005296f@mail.gmail.com","threadId":"6771","inReplyTo":"7vzm7k9c3c.fsf@assigned-by-dhcp.cox.net","subject":"Re: Efficiency of initial clone from server","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2007-02-12T05:55:36Z","receivedAt":"2007-02-12T05:55:36Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 2/12/07, Junio C Hamano wrote:\n> \"Jon Smirl\" writes:\n>\n> > On 2/11/07, Jon Smirl wrote:\n> >> Same thing with cg clone, it's what is broken.\n> >> cg update is broken in the same way.\n> >> I'm using the current git version of cogitio.\n> >> I'll switch to the git commands, git clone is about 10x faster for the\n> >> clone anyway.\n> >\n> > Don't read anything into the 10x speed diff, my last git clone was\n> > really slow. I'm probably fighting other people at kernel.org to keep\n> > the tree in RAM.\n> >\n> > But pack to the original point, can't the server check and see if it\n> > has write access so that it can keep the fully packed tree? I've just\n> > caused kernel.org to needlessly repack the wireless-dev tree a dozen\n> > times playing with this clone command. If it didn't have to keep\n> > repacking for the clone, clone would be a lot faster.\n>\n> You are assuming everybody does initial clone all the time. I\n\nWhy not use an initial clone as a trigger for a repack? Given the\nthousands of people playing with trees on kernel.org it must happen\nquite a bit. Add a log message to the server and we can find out for\nsure.\n\nI am guilty of doing initial clones for different kernel trees from\nkernel.org when I could be doing a local clone of linus' tree and then\npulling the deltas from kernel.org. But I'm lazy, I just kick the\nclone off in the background and it finishes in three or four minutes.\nI also do the clones when I have messed my local trees up so much that\nI don't know what is in them anymore.\n\n> do not think that holds true in practice.\n\ngit experts can avoid almost all of the clones, but most people don't\nlearn enough about git to avoid them.\n\nFor example, I've been using git for quite a while now and I still\nhaven't bother to figure out how to do this: start with a local clone\nof linus' tree, now I want to pull the wireless-dev tree into the same\nlocal tree as another branch. And maybe pull the wireless-2.6 into yet\nanother branch. Then can I pull updates from all of my remote\nrepositories with a single command?\n\n>\n> For something like tglx historical tree that will _never_\n> change, there is a specific hack the repository owner can take\n> advantage of to always feed a prepackaged pack, although its use\n> is not advertised well enough (and I do not think it buys much\n> in practice).\n>\n>\n>\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"34254","messageId":"7vr6svaowm.fsf@assigned-by-dhcp.cox.net","threadId":"6771","inReplyTo":"9e4733910702112155m3f5da9abgbeb2b6783005296f@mail.gmail.com","subject":"Re: Efficiency of initial clone from server","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-02-12T06:08:41Z","receivedAt":"2007-02-12T06:08:41Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jon Smirl\" writes:\n\n> I am guilty of doing initial clones for different kernel trees from\n> kernel.org when I could be doing a local clone of linus' tree and then\n> pulling the deltas from kernel.org. But I'm lazy, I just kick the\n> clone off in the background and it finishes in three or four minutes.\n> I also do the clones when I have messed my local trees up so much that\n> I don't know what is in them anymore.\n\nTime to learn to use --reference perhaps?\n\n\tgit clone --reference linux-2.6 git://.../linville/wireless-dev.git\n\nwhere \"linux-2.6\" is local repository which is my personal copy\nof Linus's repo.\n"},{"id":"34276","messageId":"Pine.LNX.4.63.0702121241130.22628@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"6771","inReplyTo":"7vzm7k9c3c.fsf@assigned-by-dhcp.cox.net","subject":"Re: Efficiency of initial clone from server","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-02-12T11:45:29Z","receivedAt":"2007-02-12T11:45:29Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sun, 11 Feb 2007, Junio C Hamano wrote:\n\n> You are assuming everybody does initial clone all the time. I do not \n> think that holds true in practice.\n\nIt depends how you interpret \"all the time\". What you (Junio) are \nsuggesting is that the count of initial clones is relatively small as \ncompared to the total number of fetches.\n\nHowever, you can interpret \"all the time\" in terms of \"time\". Most fetches \nare really small. They even often end up in no objects pulled at all. \nThese are cheap for the server. The initial clones take a long time. They \nare expensive.\n\nI'd be interested to learn how much of the CPU time is actually spent in \ninitial clones, rather than other types of fetches. It might make sense \nyet to optimize initial clones.\n\nCiao,\nDscho\n"},{"id":"34287","messageId":"9e4733910702120631o1be92037ue89b069b0354c8cb@mail.gmail.com","threadId":"6771","inReplyTo":"Pine.LNX.4.63.0702121241130.22628@wbgn013.biozentrum.uni-wuerzburg.de","subject":"Re: Efficiency of initial clone from server","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2007-02-12T14:31:48Z","receivedAt":"2007-02-12T14:31:48Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 2/12/07, Johannes Schindelin wrote:\n> Hi,\n>\n> On Sun, 11 Feb 2007, Junio C Hamano wrote:\n>\n> > You are assuming everybody does initial clone all the time. I do not\n> > think that holds true in practice.\n>\n> It depends how you interpret \"all the time\". What you (Junio) are\n> suggesting is that the count of initial clones is relatively small as\n> compared to the total number of fetches.\n>\n> However, you can interpret \"all the time\" in terms of \"time\". Most fetches\n> are really small. They even often end up in no objects pulled at all.\n> These are cheap for the server. The initial clones take a long time. They\n> are expensive.\n>\n> I'd be interested to learn how much of the CPU time is actually spent in\n> initial clones, rather than other types of fetches. It might make sense\n> yet to optimize initial clones.\n\nI don't think CPU is a problem at kernel.org, but disk IO defnitely\nis. The initial clones cause several minutes (sometimes 10 min or more\nwhen there kernel.org is loaded) worth of disk IO. They also totally\nthrash the kernel.org cache. The alternative of using a clone to\ntrigger a repack would go through this once, and then use sendfile (is\ngitd that smart?) to send the packs. Sendfile uses the smallest cache\nrequired.\n\nWhy doesn't clone copy the existing packs down first with sendfile,\nthen build a small pack for what is left and avoid the initial step of\nmaking a giant pack. Isn't clone going to break when the repo exceeds\n2GB?\n\n\n>\n> Ciao,\n> Dscho\n>\n>\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"34289","messageId":"Pine.LNX.4.64.0702121013150.1757@xanadu.home","threadId":"6771","inReplyTo":"9e4733910702112117o53630946ja50572c7c7f2b2c1@mail.gmail.com","subject":"Re: Efficiency of initial clone from server","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-02-12T15:20:31Z","receivedAt":"2007-02-12T15:20:31Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 12 Feb 2007, Jon Smirl wrote:\n\n> I didn't want to cache the packfile, instead I wanted to repack the\n> repository and then copy the resulting pack file down the wire. A\n> clone would just be a trigger to make sure everything in the repo was\n> packed (maybe into multiple packs) before starting to send anything.\n> Doing it this way means that everyone benefits from the packing.\n\nRepacking on clone is not the solution at all.\n\nThis problem is going to largely be resolved when GIT 1.5.0 gets \ninstalled on kernel.org. With latest GIT, pushes are kept as packs on \nthe remote end (when they're big enough which is over 100 objects by \ndefault). Then repacking multiple packs into one is almost free as most \nof the data is simply copied from one pack and sent over the wire as a \nsingle pack.\n\nAs for the cache problem on kernel.org, that would be largely resolved \nif all kernel related projects were repacked with reference to Linus' \nrepository to avoid copying the same set of data all over the place.\n\n\nNicolas\n"},{"id":"34290","messageId":"9e4733910702120724y2a0ba6b0h3f571aa2306872a5@mail.gmail.com","threadId":"6771","inReplyTo":"7vr6svaowm.fsf@assigned-by-dhcp.cox.net","subject":"Re: Efficiency of initial clone from server","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2007-02-12T15:24:43Z","receivedAt":"2007-02-12T15:24:43Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 2/12/07, Junio C Hamano wrote:\n> \"Jon Smirl\" writes:\n>\n> > I am guilty of doing initial clones for different kernel trees from\n> > kernel.org when I could be doing a local clone of linus' tree and then\n> > pulling the deltas from kernel.org. But I'm lazy, I just kick the\n> > clone off in the background and it finishes in three or four minutes.\n> > I also do the clones when I have messed my local trees up so much that\n> > I don't know what is in them anymore.\n>\n> Time to learn to use --reference perhaps?\n>\n> git clone --reference linux-2.6 git://.../linville/wireless-dev.git\n>\n> where \"linux-2.6\" is local repository which is my personal copy\n> of Linus's repo.\n\nI knew you smart guys would have a command to do this. This is in the\ncategory of a command that I use infrequently enough that I forget\nabout it.\n\nSomething like Cogito could be smart so that when you did a clone\ncommand it could prompt you if you wanted a new repo or to share an\nexisting one.\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"34294","messageId":"9e4733910702120840p3fd2a18ar2e3b91564556dc43@mail.gmail.com","threadId":"6771","inReplyTo":"9e4733910702120724y2a0ba6b0h3f571aa2306872a5@mail.gmail.com","subject":"Re: Efficiency of initial clone from server","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2007-02-12T16:40:13Z","receivedAt":"2007-02-12T16:40:13Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 2/12/07, Jon Smirl wrote:\n> On 2/12/07, Junio C Hamano wrote:\n> > \"Jon Smirl\" writes:\n> >\n> > > I am guilty of doing initial clones for different kernel trees from\n> > > kernel.org when I could be doing a local clone of linus' tree and then\n> > > pulling the deltas from kernel.org. But I'm lazy, I just kick the\n> > > clone off in the background and it finishes in three or four minutes.\n> > > I also do the clones when I have messed my local trees up so much that\n> > > I don't know what is in them anymore.\n> >\n> > Time to learn to use --reference perhaps?\n> >\n> > git clone --reference linux-2.6 git://.../linville/wireless-dev.git\n> >\n> > where \"linux-2.6\" is local repository which is my personal copy\n> > of Linus's repo.\n\nDoes this use hard links so that if I whack my linux-2.6 it won't also\ndestroy my wireless-dev repo?\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"34295","messageId":"Pine.LNX.4.64.0702121131550.1757@xanadu.home","threadId":"6771","inReplyTo":"20070212044914.GA699@spearce.org","subject":"Re: Efficiency of initial clone from server","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-02-12T16:42:42Z","receivedAt":"2007-02-12T16:42:42Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sun, 11 Feb 2007, Shawn O. Pearce wrote:\n\n> Nicolas Pitre wrote:\n> > Try with git2.kernel.org then. It seems that git.kernel.org too often \n> > resolves to git1.kernel.org not sharing the load with git2.kernel.org \n> > appropriately.\n> \n> Hey! You are giving out my secret!\n\nWell... this issue (and work-around) is mentioned on the k.o front page \nat http://www.kernel.org/.\n\nI didn't meant to spoil a secret... sorry. ;-)\n\n\nNicolas\n"},{"id":"34296","messageId":"20070212170403.GA29621@spearce.org","threadId":"6771","inReplyTo":"9e4733910702120840p3fd2a18ar2e3b91564556dc43@mail.gmail.com","subject":"Re: Efficiency of initial clone from server","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-02-12T17:04:03Z","receivedAt":"2007-02-12T17:04:03Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jon Smirl wrote:\n> On 2/12/07, Jon Smirl wrote:\n> >On 2/12/07, Junio C Hamano wrote:\n> >> Time to learn to use --reference perhaps?\n> >>\n> >> git clone --reference linux-2.6 \n> >git://.../linville/wireless-dev.git\n> >>\n> >> where \"linux-2.6\" is local repository which is my personal copy\n> >> of Linus's repo.\n> \n> Does this use hard links so that if I whack my linux-2.6 it won't also\n> destroy my wireless-dev repo?\n\nNo. It uses a Git specific 'symlink workalike'. We record\nthe path to the reference repository in the plain text file\n\n .git/objects/info/alternates\n\nThis file can list multiple repositories that this repository borrows\nobjects from. If you delete any of those, then this repository\nwill likely lose access to objects it thinks should be here,\nthereby totally screwing this repository.\n\nThere's also some risk with repacking or pruneing the reference\nsource (your linux-2.6 repository) as that repository doesn't know\nwhat objects it has which it itself doesn't need, but which are in\nuse by the other repositories.\n\n\nThe easiest way to prevent this from destroying a repository later\non is to make your shiny new wireless-dev clone standalone by\ncompletely repacking it after creation (*without* -l !) :\n\n git repack -a -d\n rm .git/objects/info/alternates\n\nAnother solution is don't clone from the remote, but instead clone\nlocally then update the origin and refetch:\n\n git clone -l -n linux-2.6 wireless-dev\n cd wireless-dev\n git config remote.origin.url git://.../linville/wireless-dev.git\n git fetch\n\nThe initial clone will setup hardlinks, but also uses the wrong\norigin URL. Hence we have to change it just before we attempt to\nfetch from wireless-dev.\n\n-- \nShawn.\n"},{"id":"34297","messageId":"20070212170647.GB29621@spearce.org","threadId":"6771","inReplyTo":"9e4733910702120631o1be92037ue89b069b0354c8cb@mail.gmail.com","subject":"Re: Efficiency of initial clone from server","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-02-12T17:06:47Z","receivedAt":"2007-02-12T17:06:47Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jon Smirl wrote:\n> Why doesn't clone copy the existing packs down first with sendfile,\n> then build a small pack for what is left and avoid the initial step of\n> making a giant pack. Isn't clone going to break when the repo exceeds\n> 2GB?\n\nBecause the network format allows an unlimited packfile size,\nat least until you reach the 2^32-1 object count barrier anyway.\nIt also allows only one packfile to be sent.\n\nThe problem is the local system. We are unable to build an index\nfile locally on a packfile that is larger than 4 GiB, as the offsets\nto positions within the packfile are 32 bit unsigned big-endian\nintegers. The index gets built on the client as it receives the\npackfile from the server.\n\n-- \nShawn.\n"},{"id":"34307","messageId":"20070212193507.GA18730@thunk.org","threadId":"6771","inReplyTo":"Pine.LNX.4.64.0702121013150.1757@xanadu.home","subject":"Re: Efficiency of initial clone from server","fromName":"Theodore Tso","fromEmail":"tytso@mit.edu","sentAt":"2007-02-12T19:35:07Z","receivedAt":"2007-02-12T19:35:07Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Mon, Feb 12, 2007 at 10:20:31AM -0500, Nicolas Pitre wrote:\n> Repacking on clone is not the solution at all.\n> \n> This problem is going to largely be resolved when GIT 1.5.0 gets \n> installed on kernel.org. With latest GIT, pushes are kept as packs on \n> the remote end (when they're big enough which is over 100 objects by \n> default). Then repacking multiple packs into one is almost free as most \n> of the data is simply copied from one pack and sent over the wire as a \n> single pack.\n\nEven before we get Git 1.5.0 installed on master.kernel.org (and we\nshould really ask hpa to do that), is there a reason we haven't done\nsomething like this across all of the kernel repo's on\nmaster.kernel.org?\n\nfor i in \ndo\n pushd $i\n if [ ! -f objects/info/alternates ]; then\n\techo /pub/scm/linux/kernel/git/torvalds/linux-2.6.git/objects \\\n\t > objects/info/alternates\n\tgit repack -a -d -l\n fi\n popd\ndone\n\n\n\t\t\t\t\t- Ted\n"},{"id":"34318","messageId":"7vd54f85du.fsf@assigned-by-dhcp.cox.net","threadId":"6771","inReplyTo":"20070212193507.GA18730@thunk.org","subject":"Re: Efficiency of initial clone from server","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2007-02-12T20:53:17Z","receivedAt":"2007-02-12T20:53:17Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Theodore Tso writes:\n\n> Even before we get Git 1.5.0 installed on master.kernel.org (and we\n> should really ask hpa to do that), is there a reason we haven't done\n> something like this across all of the kernel repo's on\n> master.kernel.org?\n>\n> for i in \n> do\n> pushd $i\n> if [ ! -f objects/info/alternates ]; then\n> \techo /pub/scm/linux/kernel/git/torvalds/linux-2.6.git/objects \\\n> \t > objects/info/alternates\n> \tgit repack -a -d -l\n> fi\n> popd\n> done\n\nPerhaps s//& minus a few/?\n\n\"Minus a few\" are (obviously) Linus's repository and bkcvs\nhistorical ones (I think there are two of them).\n\nOther than that I do not think of a major problem. Repacking\ninto one would inconvenience http clients but that is not a new\nissue and would have happened when the owner of the individual\nrepository chose to do so anyway.\n\nOlder clients do not understand the more efficient packfile\nformat (delta-base-offset encoding) that can be used with recent\ngit. The feature is not turned on by default and is controlled\nby configuration repack.usedeltabaseoffset. Whoever does the\n\"git repack\" above should make sure he does not enable it with\nhis $HOME/.gitconfig.\n"},{"id":"34322","messageId":"Pine.LNX.4.64.0702121629270.1757@xanadu.home","threadId":"6771","inReplyTo":"7vd54f85du.fsf@assigned-by-dhcp.cox.net","subject":"Re: Efficiency of initial clone from server","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-02-12T21:33:06Z","receivedAt":"2007-02-12T21:33:06Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Mon, 12 Feb 2007, Junio C Hamano wrote:\n\n> Older clients do not understand the more efficient packfile format \n> (delta-base-offset encoding) that can be used with recent git. The \n> feature is not turned on by default and is controlled by configuration \n> repack.usedeltabaseoffset. Whoever does the \"git repack\" above should \n> make sure he does not enable it with his $HOME/.gitconfig.\n\nOf course, if we don't care about old GIT client using HTTP transport, \nthen enabling this option won't affect GIT client using the native \nprotocol as the delta-base-offset encoding can be translated into the \nlegacy encoding on the fly depending on the client's advertized \ncapabilities.\n\n\nNicolas\n"},{"id":"34350","messageId":"eqr1up$uvv$1@sea.gmane.org","threadId":"6771","inReplyTo":"7vd54f85du.fsf@assigned-by-dhcp.cox.net","subject":"Re: Efficiency of initial clone from server","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2007-02-13T00:51:16Z","receivedAt":"2007-02-13T00:51:16Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Junio C Hamano wrote:\n\n> Theodore Tso writes:\n> \n>> Even before we get Git 1.5.0 installed on master.kernel.org (and we\n>> should really ask hpa to do that), is there a reason we haven't done\n>> something like this across all of the kernel repo's on\n>> master.kernel.org?\n>>\n>> for i in \n>> do\n>> pushd $i\n>> if [ ! -f objects/info/alternates ]; then\n>> echo /pub/scm/linux/kernel/git/torvalds/linux-2.6.git/objects \\\n>> > objects/info/alternates\n>> git repack -a -d -l\n>> fi\n>> popd\n>> done\n> \n> Perhaps s//& minus a few/?\n> \n> \"Minus a few\" are (obviously) Linus's repository and bkcvs\n> historical ones (I think there are two of them).\n> \n> Other than that I do not think of a major problem. Repacking\n> into one would inconvenience http clients but that is not a new\n> issue and would have happened when the owner of the individual\n> repository chose to do so anyway.\n\nSo objects/info/http-alternates should be also set, I guess...\n\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"},{"id":"34411","messageId":"45D1D32C.1070900@op5.se","threadId":"6771","inReplyTo":"9e4733910702112053q2c196ddcr970a062281d332b7@mail.gmail.com","subject":"Re: Efficiency of initial clone from server","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2007-02-13T15:03:08Z","receivedAt":"2007-02-13T15:03:08Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Jon Smirl wrote:\n> On 2/11/07, Junio C Hamano wrote:\n>> \"Jon Smirl\" writes:\n>>\n>> > jonsmirl@jonsmirl:/extra$ git clone\n>> > git://git.kernel.org/pub/scm/linux/kernel/gt/linville/wireless-dev.git\n>> > Initialized empty Git repository in /extra/wireless-dev/.git/\n>> > fatal: The remote end hung up unexpectedly\n>>\n>> Are you sure the above is \".../linux/kernel/gt/linville/...\"?\n> \n> You're right it should be git instead of gt, somewhere in my copying\n> strings around I lost the 'i' and wasn't paying attention. That's not\n> a very good error message: \"fatal: The remote end hung up\n> unexpectedly\" for a missing repository.\n> \n\nIt's necessary for security reasons that the git daemon doesn't tell you\n*why* it failed though, otherwise attackers could use the git daemon to\nbrowse the existance of files and directories on the remote end.\n\nIt could be nice to add \"Are you sure $path_to_repo hosts a repository?\"\nto the message though, which would toss any spelling errors in the users'\nface.\n\n-- \nAndreas Ericsson andreas.ericsson@op5.se\nOP5 AB www.op5.se\nTel: +46 8-230225 Fax: +46 8-230231\n"}]}