{"thread":{"id":"7173","subject":"Re: Errors cloning large repo","startedAt":"2007-03-10T02:37:55Z","lastAt":"2007-03-17T13:23:28Z","messageCount":14,"participants":["Anton Tropashko","Shawn O. Pearce","Linus Torvalds","Jakub Narebski","Martin Waitz"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"36721","messageId":"645002.46177.qm@web52608.mail.yahoo.com","threadId":"7173","inReplyTo":null,"subject":"Re: Errors cloning large repo","fromName":"Anton Tropashko","fromEmail":"atropashko@yahoo.com","sentAt":"2007-03-10T02:37:55Z","receivedAt":"2007-03-10T02:37:55Z","isPatch":false,"sender":{"key":"atropashko@yahoo.com","avatar":null},"body":"> I suspect we shouldn't bother with the diffstat for the initial commit. \n> Just removing \"--root\" migth be sufficient.\n\nMy problem is git-clone though since for commit it's no big deal\nto git commit [a-c]* , or use xargs as a workaround\n\nFor git clone I got this\n\nDeltifying 144511 objects.\n 100% (144511/144511) done\n1625.375MB  (1713 kB/s)       \n1729.057MB  (499 kB/s)       \n/usr/bin/git-clone: line 321: 24360 File size limit exceededgit-fetch-pack --all -k $quiet \"$repo\"\n\nagain after git repack and don't see how to work around that aside from artifically\nsplitting the tree at the top or resorting to a tarball on an ftp site.\nThat 64 bit indexing code you previously mentioned would force me to upgrade git on both ends?\nAnywhere I can pull it out from?\n\n\n\n\n\n\n \n____________________________________________________________________________________\nFood fight? Enjoy some healthy debate \nin the Yahoo! Answers Food & Drink Q&A.\nhttp://answers.yahoo.com/dir/?link=list&sid=396545367\n"},{"id":"36722","messageId":"20070310030718.GA2927@spearce.org","threadId":"7173","inReplyTo":"645002.46177.qm@web52608.mail.yahoo.com","subject":"Re: Errors cloning large repo","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-03-10T03:07:18Z","receivedAt":"2007-03-10T03:07:18Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Anton Tropashko <atropashko@yahoo.com> wrote:\n> again after git repack and don't see how to work around that aside from artifically\n> splitting the tree at the top or resorting to a tarball on an ftp site.\n> That 64 bit indexing code you previously mentioned would force me to upgrade git on both ends?\n> Anywhere I can pull it out from?\n\nI'm shocked you were able to repack an 8.5 GiB repository.\nThe default git-repack script that we ship assumes you want to\ncombine everything into one giant packfile; this is what is also\nhappening during git-clone.  Clearly your system is rejecting this\npackfile; and even if the OS allowed us to make that file the\nindex offsets would all be wrong as they are only 32 bits wide.\nThe repository becomes corrupt when those overflow.\n\n\nTroy Telford (with the help of Eric Biederman) recently posted a\npatch that attempts to push the index to 64 bits:\n\n  http://thread.gmane.org/gmane.comp.version-control.git/40680/focus=40999\n\nYou can try Troy's patch.  Nico and my's 64 bit index work is *not*\nready for anyone to use.  It doesn't exist as a compileable chunk\nof code.  ;-)\n\nJust to warn you, I have (re)done some of Troy's changes and Junio\nhas applied them to the current 'master' branch.  So Troy's patch\nwould need to be applied to something that is futher back, like\naround 2007-02-28 (when Troy sent the patch).  But my changes alone\nare not enough to get \"64 bit packfiles\" working.\n\n\nAs Linus said earlier in this thread; Nico and I are working on\npushing out the packfile limits, just not fast enough for some users\nneeds apparently (sorry about that!).  Troy's patch was rejected\nmainly because it is a file format change that is not backwards\ncompatible (once you use the 64 bit index, anything accessing that\nrepository *must* also support that).\n\nNico and I are working on other file format changes that are\nmore extensive than just expanding the index out to 64 bits, and\nlikewise are also not backwards compatible.  To help users manage\nthe upgrades, we want to do a single file format change in 2007,\nnot two.  So we are trying to be very sure that what we give Junio\nfor final application really is the best we can do this year.\n\nOtherwise we would have worked with Troy to help test his patch and\nget that into shape for application to main the git.git repository.\n\n\nOne thing that you could do is segment the repository into multiple\npackfiles yourself, and then clone using rsync or http, rather than\nusing the native Git protocol.\n\nFor segmenting the repository, you would do something like:\n\n\tgit rev-list --objects HEAD >S\n\t# segment S up into several files, e.g. T1, T2, T3\n\tforeach s in T*\n\tdo\n\t\tname=$(git pack-objects tmp <$s)\n\t\ttouch .git/objects/pack/pack-$name.keep\n\t\tmv tmp-$name.pack .git/objects/pack/pack-$name.pack\n\t\tmv tmp-$name.idx .git/objects/pack/pack-$name.idx\n\tdone\n\tgit prune-packed\n\nThe trick here is to segment S up into enough T1, T2, ... files such\nthat when packed they each are less than 2 GiB.  You can then clone\nthis repository by copying the .git directory using more standard\nfilesystem tools, which is what a clone with rsync or http is\n(more or less) doing.\n\nYes, the above process is horribly tedious and has a some trial\nand error involved in terms of selecting the packfile segmenting.\nWe don't have anything that can automate this right now.\n\n-- \nShawn.\n"},{"id":"36726","messageId":"Pine.LNX.4.64.0703092057570.10832@woody.linux-foundation.org","threadId":"7173","inReplyTo":"645002.46177.qm@web52608.mail.yahoo.com","subject":"Re: Errors cloning large repo","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-03-10T05:10:16Z","receivedAt":"2007-03-10T05:10:16Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Fri, 9 Mar 2007, Anton Tropashko wrote:\n> \n> My problem is git-clone though since for commit it's no big deal\n> to git commit [a-c]* , or use xargs as a workaround\n\nSure, but there were two problems.\n\nThe \"git commit\" problem is trivial, and in no way fundamental. The thing \nthat uses tons of memory is literally just eyecandy, to show you *what* \nyou're committing.\n\nIn fact, by the time it starts using tons of memory, the commit has \nliterally already happened. It's just doing statistics afterwards that \nbloats it up.\n\n> For git clone I got this\n\nThe \"git clone\" problem is different, in that it's due to the 2GB \npack-file limit. It's not \"fundmentally hard\" either, but it's at least \nnot just a small tiny silly detail.\n\nIn fact, you can just do\n\n\tgit add .\n\tgit commit -q\n\nand the \"-q\" flag (or \"--quiet\") will mean that the diffstat is never \ndone, and the commit should be almost instantaneous (all the real work is \ndone by the \"git add .\")\n\nSo \"git commit\" issue really is just a small beauty wart.\n\n> Deltifying 144511 objects.\n>  100% (144511/144511) done\n> 1625.375MB  (1713 kB/s)       \n> 1729.057MB  (499 kB/s)       \n> /usr/bin/git-clone: line 321: 24360 File size limit exceededgit-fetch-pack --all -k $quiet \"$repo\"\n> \n> again after git repack and don't see how to work around that aside from artifically\n> splitting the tree at the top or resorting to a tarball on an ftp site.\n\nSo the \"git repack\" actually worked for you? It really shouldn't have \nworked.\n\nIs the server side perhaps 64-bit? If so, the limit ends up being 4GB \ninstead of 2GB, and your 8.5GB project may actually fit.\n\nIf so, we can trivially fix it with the current index file even for a \n32-bit machine. The reason we limit pack-files to 2GB on 32-bit machines \nis purely that we don't use O_LARGEFILE. If we enable O_LARGEFILE, that \nmoves the limit up from 31 bits to 32 bits, and it might be enough for \nyou. No new data structures for the index necessary at all.\n\n\t\tLinus\n"},{"id":"36729","messageId":"Pine.LNX.4.64.0703092145550.10832@woody.linux-foundation.org","threadId":"7173","inReplyTo":"20070310030718.GA2927@spearce.org","subject":"Re: Errors cloning large repo","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-03-10T05:54:17Z","receivedAt":"2007-03-10T05:54:17Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Fri, 9 Mar 2007, Shawn O. Pearce wrote:\n> \n> I'm shocked you were able to repack an 8.5 GiB repository.\n\nSide note - it would be nice to hear just how big the repository *really* \nis.\n\nFor example, if \"du -sh\" says 8.5GB, it doesn't necessarily mean that \nthere really is 8.5GB of data there.\n\nWith a normal 4kB blocksize filesystem, and ~150.000 filesystem objects, \nyou'd have an average of 300MB of just padding (roughly 2kB per file). \nDepending on the file statistics, it could be even more.\n\nAnd if it's compressible, it's entirely possible that even without \nmuch delta compression, it could fit in a pack-file smaller than 4GB. At \nwhich point a 32-bit index file should work fine, just not with a 32-bit \noff_t.\n\nSo this really could be a situation where just small tweaks makes it work \nout for now. We'll need the full 64-bit index eventually for sure, but..\n\n\t\tLinus\n"},{"id":"36730","messageId":"20070310060144.GA3223@spearce.org","threadId":"7173","inReplyTo":"Pine.LNX.4.64.0703092145550.10832@woody.linux-foundation.org","subject":"Re: Errors cloning large repo","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-03-10T06:01:44Z","receivedAt":"2007-03-10T06:01:44Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Linus Torvalds <torvalds@linux-foundation.org> wrote:\n> On Fri, 9 Mar 2007, Shawn O. Pearce wrote:\n> > \n> > I'm shocked you were able to repack an 8.5 GiB repository.\n> \n> Side note - it would be nice to hear just how big the repository *really* \n> is.\n> \n> For example, if \"du -sh\" says 8.5GB, it doesn't necessarily mean that \n> there really is 8.5GB of data there.\n\nOh, good point.  Thanks for reminding me of reality.\n\nI'm just so used to not looking at repository size unless the\nrepository has been fully repacked first.  So I somehow just read\nthis thread has Anton having 8.5 GiB worth of *packed* data (where\nfilesystem wastage in the tail block is minimal) and not 8.5 GiB\nof loose objects.\n\nIts very likely this did fit in just under 4 GiB of packed data,\nbut as you said, without O_LARGEFILE we can't work with it.\n \n> So this really could be a situation where just small tweaks makes it work \n> out for now. We'll need the full 64-bit index eventually for sure, but..\n\nYes.  ;-)\n\n-- \nShawn.\n"},{"id":"36742","messageId":"esu11i$m54$1@sea.gmane.org","threadId":"7173","inReplyTo":"20070310030718.GA2927@spearce.org","subject":"Re: Errors cloning large repo","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2007-03-10T10:27:25Z","receivedAt":"2007-03-10T10:27:25Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Shawn O. Pearce wrote:\n\n> One thing that you could do is segment the repository into multiple\n> packfiles yourself, and then clone using rsync or http, rather than\n> using the native Git protocol.\n\nBy the way, it would be nice to have talked about fetch / clone\nsupport for sending (and creating) _multiple_ pack files. Beside\nthe situation where we must use more than one packfile because\nof size limits, it would also help clone as it could send existing\npacks and pack only loose objects (trading perhaps some bandwidth\nwith CPU load on the server; think kernel.org).\n\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"},{"id":"36764","messageId":"Pine.LNX.4.64.0703101414380.10330@woody.linux-foundation.org","threadId":"7173","inReplyTo":"82B0999F-73E8-494E-8D66-FEEEDA25FB91@adacore.com","subject":"Re: Errors cloning large repo","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-03-10T22:21:27Z","receivedAt":"2007-03-10T22:21:27Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 10 Mar 2007, Geert Bosch wrote:\n> \n> Larger packs might still be sent over the network, but they\n> wouldn't have an index and could be constructed on the fly,\n> without ever writing any multi-gigabyte files to disk.\n\nI have to say, I think that's a good idea. Rather than supporting a 64-bit \nindex, not generating big pack-files in general is probably a great idea.\n\nFor the streaming formats, we'd obviously generate arbitrarily large \npack-files, but as you say, they never have an index at all, and the \nreceiver always re-writes them *anyway* (ie we now always run \n\"git-index-pack --fix-thin\" on them), so we could just modify that \n\"--fix-thin\" logic to also split the pack when it reaches some arbitrary \nlimit).\n\nSome similar logic in git-pack-objects would mean that we'd never generate \nbigger packs in the first place..\n\nIt's not that 64-bit index file support is \"wrong\", but it does seem like \nit's not really necessary.\n\n\t\tLinus\n"},{"id":"36765","messageId":"20070310223251.GA20726@admingilde.org","threadId":"7173","inReplyTo":"20070310060144.GA3223@spearce.org","subject":"Re: Errors cloning large repo","fromName":"Martin Waitz","fromEmail":"tali@admingilde.org","sentAt":"2007-03-10T22:32:51Z","receivedAt":"2007-03-10T22:32:51Z","isPatch":false,"sender":{"key":"tali@admingilde.org","avatar":"https://gravatar.com/avatar/3f89b03eee362187effabe257898735b475673a12265c398ea9161259ae91553?d=mp&s=160"},"body":"hoi :)\n\nOn Sat, Mar 10, 2007 at 01:01:44AM -0500, Shawn O. Pearce wrote:\n> Its very likely this did fit in just under 4 GiB of packed data,\n> but as you said, without O_LARGEFILE we can't work with it.\n\nbut newer git version can cope with it:\n\n-r--r--r-- 1 martin martin 3847536413 18. Feb 10:36 pack-ffe867679d673ea5fbfa598b28aca1e58528b8cd.pack\n\n-- \nMartin Waitz\n"},{"id":"36766","messageId":"Pine.LNX.4.64.0703101443411.10330@woody.linux-foundation.org","threadId":"7173","inReplyTo":"20070310223251.GA20726@admingilde.org","subject":"Re: Errors cloning large repo","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-03-10T22:46:35Z","receivedAt":"2007-03-10T22:46:35Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 10 Mar 2007, Martin Waitz wrote:\n> \n> On Sat, Mar 10, 2007 at 01:01:44AM -0500, Shawn O. Pearce wrote:\n> > Its very likely this did fit in just under 4 GiB of packed data,\n> > but as you said, without O_LARGEFILE we can't work with it.\n> \n> but newer git version can cope with it:\n> \n> -r--r--r-- 1 martin martin 3847536413 18. Feb 10:36 pack-ffe867679d673ea5fbfa598b28aca1e58528b8cd.pack\n\nAre you sure you're not just running a 64-bit process?\n\n64-bit processes don't need O_LARGEFILE to process files larger than 2GB, \nsince for them, off_t is already 64-bit.\n\nGrepping for O_LARGEFILE shows nothing.\n\nOh, except we have that \n\n\t#define _FILE_OFFSET_BITS 64\n\nwhich is just a horrible hack. That's nasty. We should just use \nO_LARGEFILE rather than depend on some internal glibc thing that works \nnowhere else.\n\n\t\tLinus\n"},{"id":"36774","messageId":"20070311020013.GD10343@spearce.org","threadId":"7173","inReplyTo":"esu11i$m54$1@sea.gmane.org","subject":"Re: Errors cloning large repo","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-03-11T02:00:13Z","receivedAt":"2007-03-11T02:00:13Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jakub Narebski <jnareb@gmail.com> wrote:\n> Shawn O. Pearce wrote:\n> \n> > One thing that you could do is segment the repository into multiple\n> > packfiles yourself, and then clone using rsync or http, rather than\n> > using the native Git protocol.\n> \n> By the way, it would be nice to have talked about fetch / clone\n> support for sending (and creating) _multiple_ pack files. Beside\n> the situation where we must use more than one packfile because\n> of size limits, it would also help clone as it could send existing\n> packs and pack only loose objects (trading perhaps some bandwidth\n> with CPU load on the server; think kernel.org).\n\nI've thought about adding that type of protocol extension on\nmore than one occasion, but have now convinced myself that it is\ncompletely unnecessary.  Well at least until a project has more\nthan 2^32-1 objects anyway.\n\nThe reason is we can send any size packfile over the network; there\nis no index sent so there is no limit on how much data we transfer.\nWe could easily just dump all existing packfiles as-is (just clip\nthe header/footers and generate our own for the entire stream)\nand then send the loose objects on the end.\n\nThe client could easily segment that into multiple packfiles\nlocally using two rules:\n\n  - if the last object was not a OBJ_COMMIT and this object is\n  an OBJ_COMMIT, start a new packfile with this object.\n\n  - if adding this object to the current packfile exceeds my local\n  filesize threshold, start a new packfile.\n\nThe first rule works because we sort objects by type, and commits\nappear at the front of a packfile.  So if you see a non-commit\nfollowed by a commit, that's the packfile boundary that the\nserver had.\n\nThe second rule is just common sense.  But I'm not sure the first\nrule is even worthwhile; the server's packfile boundaries have no\nreal interest for the client.\n\n\nBut Linus has already pointed all of this out (more or less) in a\ndifferent fork of this thread.  ;-)\n\n-- \nShawn.\n"},{"id":"36866","messageId":"20070311213523.GB20726@admingilde.org","threadId":"7173","inReplyTo":"Pine.LNX.4.64.0703101443411.10330@woody.linux-foundation.org","subject":"Re: Errors cloning large repo","fromName":"Martin Waitz","fromEmail":"tali@admingilde.org","sentAt":"2007-03-11T21:35:24Z","receivedAt":"2007-03-11T21:35:24Z","isPatch":false,"sender":{"key":"tali@admingilde.org","avatar":"https://gravatar.com/avatar/3f89b03eee362187effabe257898735b475673a12265c398ea9161259ae91553?d=mp&s=160"},"body":"hoi :)\n\nOn Sat, Mar 10, 2007 at 02:46:35PM -0800, Linus Torvalds wrote:\n> Are you sure you're not just running a 64-bit process?\n\npretty sure, yes :-)\n\n> 64-bit processes don't need O_LARGEFILE to process files larger than 2GB, \n> since for them, off_t is already 64-bit.\n\nbut O_LARGEFILE is a Linux-only thing, right?\n\n> Oh, except we have that \n> \n> \t#define _FILE_OFFSET_BITS 64\n> \n> which is just a horrible hack. That's nasty. We should just use \n> O_LARGEFILE rather than depend on some internal glibc thing that works \n> nowhere else.\n\nWell, if I remember correctly the *BSD systems always use 64bit now,\nits sad that glibc does not do the same out of the box for Linux.\n_FILE_OFFSET_BITS is the documented way to get 64bit file sizes on\nglibc, so I think it is the right thing for us (even when that define\nis really ugly).\n\n-- \nMartin Waitz\n"},{"id":"36896","messageId":"200703121209.35052.jnareb@gmail.com","threadId":"7173","inReplyTo":"20070311020013.GD10343@spearce.org","subject":"Re: Errors cloning large repo","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2007-03-12T11:09:33Z","receivedAt":"2007-03-12T11:09:33Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Shawn O. Pearce wrote:\n> Jakub Narebski <jnareb@gmail.com> wrote:\n>> Shawn O. Pearce wrote:\n>> \n>>> One thing that you could do is segment the repository into multiple\n>>> packfiles yourself, and then clone using rsync or http, rather than\n>>> using the native Git protocol.\n>> \n>> By the way, it would be nice to have talked about fetch / clone\n>> support for sending (and creating) _multiple_ pack files. Beside\n>> the situation where we must use more than one packfile because\n>> of size limits, it would also help clone as it could send existing\n>> packs and pack only loose objects (trading perhaps some bandwidth\n>> with CPU load on the server; think kernel.org).\n> \n> I've thought about adding that type of protocol extension on\n> more than one occasion, but have now convinced myself that it is\n> completely unnecessary.  Well at least until a project has more\n> than 2^32-1 objects anyway.\n> \n> The reason is we can send any size packfile over the network; there\n> is no index sent so there is no limit on how much data we transfer.\n> We could easily just dump all existing packfiles as-is (just clip\n> the header/footers and generate our own for the entire stream)\n> and then send the loose objects on the end.\n\nBut what would happen if server supporting concatenated packfiles\nsends such stream to the old client? So I think some kind of protocol\nextension, or at least new request / new feature is needed for that.\n\nWouldn't it be better to pack loose objects into separate pack\n(and perhaps save it, if some threshold is crossed, and we have\nwriting rights to repo), by the way?\n\n> The client could easily segment that into multiple packfiles\n> locally using two rules:\n> \n>   - if the last object was not a OBJ_COMMIT and this object is\n>   an OBJ_COMMIT, start a new packfile with this object.\n> \n>   - if adding this object to the current packfile exceeds my local\n>   filesize threshold, start a new packfile.\n> \n> The first rule works because we sort objects by type, and commits\n> appear at the front of a packfile.  So if you see a non-commit\n> followed by a commit, that's the packfile boundary that the\n> server had.\n> \n> The second rule is just common sense.  But I'm not sure the first\n> rule is even worthwhile; the server's packfile boundaries have no\n> real interest for the client.\n\nWithout first rule, wouldn't client end with strange packfile?\nOr would it have to rewrite a pack?\n\n-- \nJakub Narebski\nPoland\n"},{"id":"36910","messageId":"20070312142451.GC15150@spearce.org","threadId":"7173","inReplyTo":"200703121209.35052.jnareb@gmail.com","subject":"Re: Errors cloning large repo","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-03-12T14:24:51Z","receivedAt":"2007-03-12T14:24:51Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jakub Narebski <jnareb@gmail.com> wrote:\n> But what would happen if server supporting concatenated packfiles\n> sends such stream to the old client? So I think some kind of protocol\n> extension, or at least new request / new feature is needed for that.\n\nNo, a protocol extension is not required.  The packfile format\nis: 12 byte header, objects, 20 byte SHA-1 footer.  When sending\nconcatenated packfiles to a client the server just needs to:\n\n  - figure out how many objects total will be sent;\n  - send its own (new) header with that count;\n  - initialize a SHA-1 context and update it with the header;\n  - for each packfile to be sent:\n    - strip the first 12 bytes of the packfile;\n    - send the remaining bytes, except the last 20;\n    - update the SHA-1 context with the packfile data;\n  - send its own footer with the SHA-1 context.\n\nVery simple.  Even the oldest Git clients (pre multi-ack extension)\nwould understand that.  That's what's great about the way the\npackfile protocol and disk format is organized.  ;-)\n \n> Wouldn't it be better to pack loose objects into separate pack\n> (and perhaps save it, if some threshold is crossed, and we have\n> writing rights to repo), by the way?\n\nPerhaps.  Interesting food for thought, something nobody has tried\nto experiment with.  Currently servers pack to update the fetching\nclient.  That means they may be sending a mixture of already-packed\n(older) objects and loose (newer) objects.  But with the new kept\npack thing in receive-pack its more likely that things are already\npacked on the server, and not loose.  (I suspect most public open\nsource users are pushing >100 objects when they do push to their\nserver.)\n \n> > The client could easily segment that into multiple packfiles\n> > locally using two rules:\n> > \n> >   - if the last object was not a OBJ_COMMIT and this object is\n> >   an OBJ_COMMIT, start a new packfile with this object.\n...\n> \n> Without first rule, wouldn't client end with strange packfile?\n> Or would it have to rewrite a pack?\n\nNope.  We don't care about the order of the objects in a packfile.\nNever have.  Never will.  Even in pack v4 where we have special\nobject types that should only appear once in a packfile, they can\nappear at any position within the packfile.  MUCH simpler code.\n\n-- \nShawn.\n"},{"id":"37311","messageId":"200703171423.29265.jnareb@gmail.com","threadId":"7173","inReplyTo":"20070312142451.GC15150@spearce.org","subject":"Re: Errors cloning large repo","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2007-03-17T13:23:28Z","receivedAt":"2007-03-17T13:23:28Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"On Mon, 12 March 2007, Shawn O. Pearce wrote:\n> Jakub Narebski <jnareb@gmail.com> wrote:\n\n>> But what would happen if server supporting concatenated packfiles\n>> sends such stream to the old client? So I think some kind of protocol\n>> extension, or at least new request / new feature is needed for that.\n> \n> No, a protocol extension is not required.  The packfile format\n> is: 12 byte header, objects, 20 byte SHA-1 footer.  When sending\n> concatenated packfiles to a client the server just needs to:\n> \n>   - figure out how many objects total will be sent;\n>   - send its own (new) header with that count;\n>   - initialize a SHA-1 context and update it with the header;\n>   - for each packfile to be sent:\n>     - strip the first 12 bytes of the packfile;\n>     - send the remaining bytes, except the last 20;\n>     - update the SHA-1 context with the packfile data;\n>   - send its own footer with the SHA-1 context.\n> \n> Very simple.  Even the oldest Git clients (pre multi-ack extension)\n> would understand that.  That's what's great about the way the\n> packfile protocol and disk format is organized.  ;-)\n\nIt would be a very nice thing to have, if it is backwards compatibile.\nIt would ease load to server on clone, even if packs are divided into\nlarge tight archive pack and perhaps a few more current packs to make\ndumb transport do not neeed to download everything on [incremental]\nfetch.\n\nOn fetch... perhaps there should be some configuration variable which\nwould change balance between load and bandwidth used...\n\nAnd automatic splitting large pack on client side would help if for\nexample we have huge repository (non-compressable binaries) and client\nhas smaller filesystem limit on maximum file size than server.\n\n-- \nJakub Narebski\nPoland\n"}]}