{"thread":{"id":"1095","subject":"\"Incremental packs\"","startedAt":"2005-07-03T23:00:52Z","lastAt":"2005-07-03T23:00:52Z","messageCount":1,"participants":["Linus Torvalds"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"5612","messageId":"Pine.LNX.4.58.0507031543170.3570@g5.osdl.org","threadId":"1095","inReplyTo":null,"subject":"\"Incremental packs\"","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-07-03T23:00:52Z","receivedAt":"2005-07-03T23:00:52Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\nOk,\n I've changed things around a bit to make it much easier to create various \nincremental packs. It's still not the \"just append everything\" thing that \nPeter envisions, but it should hopefully be fairly usable in practice, and \nmight even be worth trying on kernel.org now.\n\nOne big user-visible change: in order to allow multiple packs, where the \npack-name itself does not cause clashes when you rsync them between \nrepositories, I changed git-pack-objects to add the SHA1 of all the \nobjects added to the pack to the name.\n\nSo when you do\n\n\tgit-pack-objects pack\n\nit will create two files: \"pack-xyz.idx\" and \"pack-xyz.pack\", where the \n\"xyz\" is the SHA1 of all the object names that were added to the pack. The \ncommand also writes that SHA1 to its standard output, so you can do\n\n\tpackname=$(git-rev-list ... | git-pack-objects pack)\n\nand \"packname\" will contain the part you're missing (ie then you can look\nat \"pack-$packname.idx\" to see the index).\n\nI've also added flags to make it possible to easily do \"incremental \npacks\", which just pack the stuff that wasn't already packed. So \n\"git-rev-list --unpacked\" will consider any packed commit to be \nun-interesting, and \"git-pack-objects --incremental\" will similarly just \nignore any objects that already exist in a pack.\n\nThis, together with \"git-rev-parse --all\", which shows all the current \nrefs, makes it possible to do \"git repack\", which basically boils down to\n\n\tpackname=$(git-rev-list --unpacked --objects $(git-rev-parse --all) |\n\t\tgit-pack-objects --non-empty --incremental .tmp-pack)\n\nwhere \"non-empty\" means that if the list of objects is empty, we won't\ncreate a pack at all (and git-pack-objects will not write any SHA1 to its \nstdout, so \"packname\" will be empty).\n\nLook at \"git repack\" for all the small details if you care, but basically \nit's designed so that you can run \"git repack\" on a git archive every \nweek, and it will create pack-files of any new data.\n\nNow, once you actually trust the pack-files, you can then do\n\n\tgit-prune-packed\n\nwhich will remove all unpacked objects that are available in a pack. \nCAREFUL! Only do this once you trust the pack! \"git repack\" will not do \nthis on its own.\n\nThe theory behind all this is that since the pack-files are now named \naccording to their contents, they are also distributable exactly the same \nway the normal object files are, so you can still use \"rsync\" if you \nreally want to distribute the resulting mess. A git-aware distribution is \nobviously preferable, but at least rsync won't corrupt any archive, and \nthis allows you to get rid of the \"millions of small files\" effect.\n\nIt might be worth trying out on kernel.org. Peter, what do you think? \nDoing the initial \"git repack\" is pretty expensive, but then the weekly \nincremental packs should be pretty cheap. And it really should not be done \nvery often, because then the pack-files end up just being many and small, \nwhich is what we're trying to avoid in the first place.\n\nOh, one (obvious) detail: in order for webgit to continue working,\nwww.kernel.org needs to have a recent enough git, of course.\n\n\t\tLinus\n"}]}