{"thread":{"id":"11075","subject":"git pack-objects input list","startedAt":"2007-12-01T10:45:02Z","lastAt":"2007-12-02T02:23:19Z","messageCount":4,"participants":["Mike Hommey","Linus Torvalds","Nicolas Pitre"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"61572","messageId":"20071201104502.GA1457@glandium.org","threadId":"11075","inReplyTo":null,"subject":"git pack-objects input list","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2007-12-01T10:45:02Z","receivedAt":"2007-12-01T10:45:02Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"Hi,\n\nWhile playing around with git-pack-objects, it seemed to me that the\ninput it can take is not a simple list of object SHA1s. Unfortunately,\nthe man page is not very verbose about that. While I'd happily send a\npatch for that, I'd prefer to actually know what kind of input it can\ntake, and what it uses it for.\n\nAFAICT, it can take the output of git-rev-list --all --objects (so,\nSHA1s followed by file names for blobs), which seems to be the same as\nwhat git-pack-objects --revs does internally, but it seems to have a\nstring impact on how deltas are calculated (not giving file names makes\nit create a smaller pack in some cases, a bigger one in other cases).\n\nCould someone knowing the delta calculation internals enlighten me ?\n\nThanks\n\nMike\n"},{"id":"61590","messageId":"alpine.LFD.0.9999.0712010933370.8458@woody.linux-foundation.org","threadId":"11075","inReplyTo":"20071201104502.GA1457@glandium.org","subject":"Re: git pack-objects input list","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-12-01T17:49:00Z","receivedAt":"2007-12-01T17:49:00Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 1 Dec 2007, Mike Hommey wrote:\n> \n> While playing around with git-pack-objects, it seemed to me that the\n> input it can take is not a simple list of object SHA1s.\n\nWell, it *can* take a simple list of object SHA1's. But yes, the preferred \nformat is a list of \"SHA1 <basename>\", where the basename is used as part \nof the heuristics on what other objects to try doing a delta against.\n\nBut if you give no basename, that heuristic just won't have the name hint, \nand things will still *work*, it's just more likely (but not certain) that \nthe resulting packfile will be larger.\n\n> Could someone knowing the delta calculation internals enlighten me ?\n\nThe delta calculations simply create a small hash based on the basename, \nand use that to clump blobs/trees with the same basename together. That's \n*usually* a huge win in terms of finding good deltas, since the most \nlikely delta is for a previous version of the same file (or tree!) and \nsince we don't try to find deltas against *all* other blobs, but just use \na sliding window, having good delta candidates close to each other is \ngoing to help a lot.\n\nWithout the basename information, the delta list will just be sorted by \ntype and size, which works fine, but generally finds fewer deltas.\n\nBut it's all a heuristic, and if can go both ways. If you have lots of \nrenames (which aren't just cross-directory ones, but actually change the \nbasename), then the basename information may actually hurt.\n\n(Btw: the hash we generate is on purpose not a very good one. It actually \nthinks that the last characters are \"more important\", so it tends to hash \nfiles that end in the same few characters together. So *.c files clump \ntogether etc. At least that's the intent).\n\nSee builtin-pack-objects.c:\n - type_size_sort(): this is the rule for sortign objects for deltaing. \n\n   Type is most important (ie we always sort commits, trees, blobs \n   separately and clump them together and effectively delta them only \n   against objects of the same type)\n\n   Then comes the basename hash (so that we sort objects with the same \n   name together, and *.c files closer to each other than to *.h files, \n   for example).\n\n   Then comes the preferred_base (so that we sort things that already have \n   specific delta bases together), and then the size (so that we sort\n   files that are similar in size).\n\n   And finally, if everything else is equal (the size will generally be \n   identical for tree objects of the same directory with no new files but \n   just SHA1 changes, for example) we sort by the order they were found in \n   the history (\"recency\") by just comparing the pointer itself, since \n   the original thing will be just one big array filled in by order of \n   objects.\n\n - find_deltas() - this is the actual thing that does the \"look through \n   the object window and try to find good deltas\", which operates on the \n   array that was created by the type_size_sort.\n\nHope that clarified something.\n\n\t\tLinus\n"},{"id":"61603","messageId":"20071201223849.GA15110@glandium.org","threadId":"11075","inReplyTo":"alpine.LFD.0.9999.0712010933370.8458@woody.linux-foundation.org","subject":"Re: git pack-objects input list","fromName":"Mike Hommey","fromEmail":"mh@glandium.org","sentAt":"2007-12-01T22:38:49Z","receivedAt":"2007-12-01T22:38:49Z","isPatch":false,"sender":{"key":"mh@glandium.org","avatar":"https://avatars.githubusercontent.com/u/1038527?v=4"},"body":"On Sat, Dec 01, 2007 at 09:49:00AM -0800, Linus Torvalds wrote:\n> Hope that clarified something.\n\nThanks, that helped me understand my observations when trying to pack\nwith and without file names in pack-objects input on different kind of\ndatasets, where some would be best packed with and others would be without.\n\nI'll try to add some words about the pack-objects input format in the\ndocumentation. I don't know if it's worth adding information about the\npacking process itself in the manual page. Or maybe that should be added\nto a more technical document about git (a bit like \"git for computer\nscientists\")\n\nMike\n"},{"id":"61609","messageId":"alpine.LFD.0.99999.0712012120060.9605@xanadu.home","threadId":"11075","inReplyTo":"20071201223849.GA15110@glandium.org","subject":"Re: git pack-objects input list","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2007-12-02T02:23:19Z","receivedAt":"2007-12-02T02:23:19Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Sat, 1 Dec 2007, Mike Hommey wrote:\n\n> On Sat, Dec 01, 2007 at 09:49:00AM -0800, Linus Torvalds wrote:\n> > Hope that clarified something.\n> \n> Thanks, that helped me understand my observations when trying to pack\n> with and without file names in pack-objects input on different kind of\n> datasets, where some would be best packed with and others would be without.\n> \n> I'll try to add some words about the pack-objects input format in the\n> documentation. I don't know if it's worth adding information about the\n> packing process itself in the manual page. Or maybe that should be added\n> to a more technical document about git (a bit like \"git for computer\n> scientists\")\n\nLook at Documentation/technical/ for existing technically oriented \ndocuments.  The pack format and packing heuristics have documents of \ntheir own already.  If you feel like adding more documentation there \nplease just go ahead.\n\n\nNicolas\n"}]}