{"thread":{"id":"5035","subject":"git prune pig slow","startedAt":"2006-07-29T09:02:50Z","lastAt":"2006-07-29T20:03:33Z","messageCount":4,"participants":["Russell King","Johannes Schindelin","Linus Torvalds"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"24334","messageId":"20060729090250.GC26956@flint.arm.linux.org.uk","threadId":"5035","inReplyTo":null,"subject":"git prune pig slow","fromName":"Russell King","fromEmail":"rmk@arm.linux.org.uk","sentAt":"2006-07-29T09:02:50Z","receivedAt":"2006-07-29T09:02:50Z","isPatch":false,"sender":{"key":"rmk@arm.linux.org.uk","avatar":null},"body":"Hi,\n\ngit 1.4.0, P4 2.6GHz, 1GB.\n\nI'm trying to use \"git prune\" to remove some unreachable objects from\nmy git tree.  However, it appears to be _extremely_ expensive:\n\nrmk      13376 91.3 15.7 165980 161556 pts/0   R+   09:50   5:14 git-fsck-object\n\nstracing it shows that it's doing lots and lots of brk() calls.\n\nI killed it after 10 minutes and decided to do the job manually -\ngit-fsck-objects --unreachable and deleting the objects one by one is\n_much_ quicker than git-fsck-objects --full --cache --unreachable.\n\n-- \nRussell King\n"},{"id":"24338","messageId":"Pine.LNX.4.63.0607291339200.29667@wbgn013.biozentrum.uni-wuerzburg.de","threadId":"5035","inReplyTo":"20060729090250.GC26956@flint.arm.linux.org.uk","subject":"Re: git prune pig slow","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2006-07-29T11:40:01Z","receivedAt":"2006-07-29T11:40:01Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Sat, 29 Jul 2006, Russell King wrote:\n\n> Hi,\n> \n> git 1.4.0, P4 2.6GHz, 1GB.\n> \n> I'm trying to use \"git prune\" to remove some unreachable objects from\n> my git tree.  However, it appears to be _extremely_ expensive:\n> \n> rmk      13376 91.3 15.7 165980 161556 pts/0   R+   09:50   5:14 git-fsck-object\n> \n> stracing it shows that it's doing lots and lots of brk() calls.\n\nDoes git-count-objects show a high amount of unpacked objects? You should \ntry \"git-repack -a -d\" _before_ git-prune, then.\n\nHth,\nDscho\n"},{"id":"24345","messageId":"Pine.LNX.4.64.0607291110180.4168@g5.osdl.org","threadId":"5035","inReplyTo":"20060729090250.GC26956@flint.arm.linux.org.uk","subject":"Re: git prune pig slow","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-07-29T18:14:02Z","receivedAt":"2006-07-29T18:14:02Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 29 Jul 2006, Russell King wrote:\n> \n> I killed it after 10 minutes and decided to do the job manually -\n> git-fsck-objects --unreachable and deleting the objects one by one is\n> _much_ quicker than git-fsck-objects --full --cache --unreachable.\n\nIt's also very dangerous.\n\nIf you have partial packing (which you can get if you fetch data using \nrsync or http, for example), not havign the \"--full\" means that \ngit-fsck-objects will report on objects being \"unreachable\" if they are \nonly reachable from another object that is packed.\n\nNow, in practice, if you only use the git native protocol, this should \nnever happen, and you're fine. But there's a _very_ real reason why \"git \nprune\" passes the \"--full\" flag to git-fsck-cache. \"git prune\" is simply \ntoo dangerous without it.\n\nThat said, the current \"git prune\" in 1.4.2-rc is much faster, because it \ndoes the reachability analysis on its own, and doesn't do all the other \nthings that git-fsck-cache does.\n\nBtw, another alternative to \"git prune\" is actually to do\n\n\tgit repack -a -d\n\nand then just delete all unpacked objects.\n\n\t\t\tLinus\n"},{"id":"24346","messageId":"Pine.LNX.4.64.0607291245230.4168@g5.osdl.org","threadId":"5035","inReplyTo":"Pine.LNX.4.64.0607291110180.4168@g5.osdl.org","subject":"Re: git prune pig slow","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2006-07-29T20:03:33Z","receivedAt":"2006-07-29T20:03:33Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 29 Jul 2006, Linus Torvalds wrote:\n> \n> It's also very dangerous.\n> \n> If you have partial packing (which you can get if you fetch data using \n> rsync or http, for example), not havign the \"--full\" means that \n> git-fsck-objects will report on objects being \"unreachable\" if they are \n> only reachable from another object that is packed.\n> \n> Now, in practice, if you only use the git native protocol, this should \n> never happen, and you're fine.\n\nSide note: in _practice_, it probably doesn't happen even with rsync and \nhttp, so in that sense, it's true that \"--full\" is almost always likely to \njust be a waste of time, and I can't come up with a schenario where you \nreally need \"--full\" for pruning unless you did something strange. All the \nnormal workflows means that if you have an object that is in a pack, \neverything it points to will _also_ be in a pack, and as such, \"git prune\" \nwould never remove anything that wasn't safe to remove, even without the \n\"--full\".\n\nBut just to get an example of how a _strange_ schenario could happen, \nlet's say that\n\n - you're tracking a upstreams repository using rsync or http (ie you will \n   get the objects in the same format that upstream tracks them, either as \n   individual objects, or as \"packs\")\n\n - that upstreams repository does _incremental_ repacks every once in a \n   while. \n\n - the last time you fetched was _just_ before upstream did an incremental \n   pack, we call this \"State A\".\n\n\tAs a result, you now have his old state A all as individual \n\tobjects in your object database.\n\n - you fetch again, now after upstream has done _two_ incremntal packs \n   (one to pack all the loose objects that you already had, and one to \n   pack the new state). Upstream is now at \"State B\"\n\n\tAs a result, you get all of his _new_ objects as one nice pack: \n\tyou do not get his other pack, because you already have all \n\t_those_ objects (which are \"state A\") as individual objects.\n\n - so now, since you're only tracking the other ends state, and have no \n   objects of your own (in particular, the last fetch/pull did _not_ \n   generate a merge object of your own to connect the new pack with the \n   old objects), what has happened is that all your heads point into the \n   new incremental pack you just fetched, and that pack itself will have \n   pointers to the individual objects that you fetched last time, because \n   it was an incremental pack to \"state A\".\n\n - what happens now is that if you run \"git-fsck-objects\" without the \n   \"--full\", it will claim that _all_ of your unpacked objects are \n   unreachable, because they really are reachable only though that new \n   pack.\n\nSo in this (very very unusual) circumstance, \"git prune\" without the \n\"--full\" would literally prune away objects that you very much need.\n\nI hope this explains why that \"unnecessary\" (and admittedly much more \nexpensive) --full is there. It really is unnecessary in practice: partly \nbecause Junio has made \"git repack -a -d\" so efficient that doing \nincremental packs isn't even worth it for most people, and partly because \nyou probably use the native git protocol and repack yourself, and thus \nnever use another persons pack directly (which also avoids this problem).\n\nBut yeah, the olf \"git prune\" was really very expensive. It's much better \nin the current git branch, although it's still not _cheap_ (because it \ndoes do the whole reachability analysis, though all pack-files, because it \nwants to get the above special case right).\n\nIf we really wanted to, we could add a \"core.fullpacks\" flag that you \ncould set, and that would cause the non-native protocols to not work (or \nalternatively force a re-pack after they have fetched a pack), and that \nwould disallow incremental repacking locally, and then we could optimize \nthe hell out of \"git prune\" and say that it never needs to look at any \nreachability for an object that is already packed.\n\nThat would make \"git prune\" basically instantaneous, the way \"git \nfsck-objects\" is by default. But to be safe, it really needs to have some \nper-repository flag that is honored by the other git commands.\n\n\t\t\tLinus\n"}]}