{"thread":{"id":"1255","subject":"Kernel Hacker's guide to git (updated)","startedAt":"2005-07-15T17:36:21Z","lastAt":"2005-07-26T16:44:07Z","messageCount":5,"participants":["Jeff Garzik","Linus Torvalds","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"6175","messageId":"42D7F415.30609@pobox.com","threadId":"1255","inReplyTo":null,"subject":"Kernel Hacker's guide to git (updated)","fromName":"Jeff Garzik","fromEmail":"jgarzik@pobox.com","sentAt":"2005-07-15T17:36:21Z","receivedAt":"2005-07-15T17:36:21Z","isPatch":false,"sender":{"key":"jgarzik@pobox.com","avatar":null},"body":"\nI've updated my git quickstart guide at\n\n\thttp://linux.yyz.us/git-howto.html\n\nIt now points to DaveJ's daily snapshots for the initial bootstrap \ntarball, is reorganized for better navigation, and other things.\n\n\nAlso, a bonus recipe:  how to import Linus's pack files (it's easy).\n\nThis recipe presumes that you have a vanilla-Linus repo \n(/repo/linux-2.6) and your own repo (/repo/myrepo-2.6).\n\n$ cd /repo/myrepo-2.6\n$ git-fsck-cache\t\t# fsck, make sure we're OK\n$ git pull /repo/linux-2.6/.git\t# make sure we're up-to-date\n$ cp -al ../linux-2.6/.git/objects/pack .git/objects\n$ cp ../linux-2.6/.git/refs/tags/* .git/refs/tags\n$ git-prune-packed\n$ git-fsck-cache\t\t# fsck #2, make sure we're OK\n\nThis recipe reduced my kernel.org sync from ~50,000 files to ~5,000 files.\n\n\tJeff\n"},{"id":"6456","messageId":"Pine.LNX.4.58.0507252145470.6074@g5.osdl.org","threadId":"1255","inReplyTo":"42D7F415.30609@pobox.com","subject":"Re: Why pack+unpack?","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-07-26T04:53:45Z","receivedAt":"2005-07-26T04:53:45Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 26 Jul, Jeff Garzik wrote:\n>\n>\t\t\t\tAFAICT this is\n> just a complete waste of time.  Why does this occur?\n>\n> Packing 1394 objects\n> Unpacking 1394 objects\n>   100% (1394/1394) done\n> \n> It doesn't seem to make any sense to perform work, then immediately undo\n> that work, just for a local pull.\n\nFirst, make sure you have a recent git, it does better at optimizing the \nobjects, so there are fewer of them. Of course, the above could be a real \npull of a a fair amount of work, but check that your git has this commit:\n\n\tcommit 4311d328fee11fbd80862e3c5de06a26a0e80046\n\n\t    Be more aggressive about marking trees uninteresting\n\nbecause otherwise you sometimes get a fair number of objects just because\ngit-rev-list wasn't always being very careful, and took more objects than\nit strictly needed.\n\nSecondly, what's the problem? Sure, I could special-case the local case, \nbut do you really want to have two _totally_ different code-paths? In \nother words, it's absolutely NOT a complete waste of time: it's very much \na case of trying to have a unified architecture, and the fact that it \nspends a few seconds doing things in a way that is network-transparent is \ntime well spent.\n\nPut another way: do you argue that X network transparency is a total waste\nof time? You could certainly optimize X if you always made it be\nlocal-machine only. Or you could make tons of special cases, and have X \nhave separate code-paths for local clients and for remote clients, rather \nthan just always opening a socket connection.\n\nSee? Trying to have one really solid code-path is not a waste of time. \n\nWe do end up having a special code-path for \"clone\" (the \"-l\" flag), which\ndoes need it, but I seriously doubt you need it for a local pull. The most \nexpensive operation in a local pull tends to be (if the repositories are \nunpacked and cold-cache) just figuring out the objects to pull, not the \npacking/unpacking per se.\n\n\t\t\tLinus\n"},{"id":"6457","messageId":"42E5C677.2020403@pobox.com","threadId":"1255","inReplyTo":"Pine.LNX.4.58.0507252145470.6074@g5.osdl.org","subject":"Re: Why pack+unpack?","fromName":"Jeff Garzik","fromEmail":"jgarzik@pobox.com","sentAt":"2005-07-26T05:13:27Z","receivedAt":"2005-07-26T05:13:27Z","isPatch":false,"sender":{"key":"jgarzik@pobox.com","avatar":null},"body":"Linus Torvalds wrote:\n> First, make sure you have a recent git, it does better at optimizing the \n\nI was using vanilla git, as of 10 minutes before I sent the email.  Top \nof tree is 154d3d2dd2656c23ea04e9d1c6dd4e576a7af6de.\n\n\n> Secondly, what's the problem? Sure, I could special-case the local case, \n> but do you really want to have two _totally_ different code-paths? In \n> other words, it's absolutely NOT a complete waste of time: it's very much \n> a case of trying to have a unified architecture, and the fact that it \n> spends a few seconds doing things in a way that is network-transparent is \n> time well spent.\n> \n> Put another way: do you argue that X network transparency is a total waste\n> of time? You could certainly optimize X if you always made it be\n> local-machine only. Or you could make tons of special cases, and have X \n> have separate code-paths for local clients and for remote clients, rather \n> than just always opening a socket connection.\n\nPoor example...   sure it opens a socket, but X certainly does have a \nspecial case local path (mit shm), and they're adding more for 3D due \nthe massive amount of data involved in 3D.\n\n\n> We do end up having a special code-path for \"clone\" (the \"-l\" flag), which\n> does need it, but I seriously doubt you need it for a local pull. The most \n> expensive operation in a local pull tends to be (if the repositories are \n> unpacked and cold-cache) just figuring out the objects to pull, not the \n> packing/unpacking per se.\n\nWell, I'm not overly concerned, mostly curious.  The pack+unpack step \n(a) appears completely redundant and (b) is the step that takes the most \ntime here, for local pulls, after the diffstat.\n\n\tJeff\n"},{"id":"6458","messageId":"7vek9mayar.fsf@assigned-by-dhcp.cox.net","threadId":"1255","inReplyTo":"Pine.LNX.4.58.0507252145470.6074@g5.osdl.org","subject":"Re: Why pack+unpack?","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-07-26T06:14:52Z","receivedAt":"2005-07-26T06:14:52Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds <torvalds@osdl.org> writes:\n\n> See? Trying to have one really solid code-path is not a waste of time. \n\nAn alternative code path specialized for local case would not be\ntoo bad.\n\nFirst, finding the list of objects to copy.  You can use\nalternate object pool to cover the upstream repository to pull\nfrom, and the downstream repository to pull into (both local),\nrun rev-list --objects, giving it prefix '^' for all refs in the\ndownstream repository, and the upstream head SHA1 you are\npulling.  If the upstream head you are pulling is a tag, then\nyou may need to dereference it as well.\n\nAmong those objects, ones unpacked in the upstream can be\ncopied/linked to the downstream repository.\n\nHandling packs involves a little bit of policy decision.  The\ncurrent pack/unpack way always unpacks, and to emulate it, we\ncan cat-file in the upstream object database, pipe that to\n\"hash-object -w\" (after giving hash-object an option to read\nfrom the standard input) to write in the downstream repository\nunpacked.  Easier alternative is to just hardlink all the packs\nfrom the upstream object database into the downstream object\ndatabase, and keep packed things packed.\n\nWell, it starts to sound somewhat bad...\n"},{"id":"6464","messageId":"Pine.LNX.4.58.0507260937460.19309@g5.osdl.org","threadId":"1255","inReplyTo":"42E5C677.2020403@pobox.com","subject":"Re: Why pack+unpack?","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-07-26T16:44:07Z","receivedAt":"2005-07-26T16:44:07Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Tue, 26 Jul 2005, Jeff Garzik wrote:\n> > \n> > Put another way: do you argue that X network transparency is a total waste\n> > of time? You could certainly optimize X if you always made it be\n> > local-machine only. Or you could make tons of special cases, and have X \n> > have separate code-paths for local clients and for remote clients, rather \n> > than just always opening a socket connection.\n> \n> Poor example...   sure it opens a socket, but X certainly does have a \n> special case local path (mit shm), and they're adding more for 3D due \n> the massive amount of data involved in 3D.\n\n.. and that's still a special case. Exactly like git does the \"clone -l\" \nspecial case.\n\n> Well, I'm not overly concerned, mostly curious.  The pack+unpack step \n> (a) appears completely redundant and (b) is the step that takes the most \n> time here, for local pulls, after the diffstat.\n\nIt's not actually redundant. Some of the _compression_ may be, and you \ncould see if you prefer a smaller delta window (use \"--window=0\" to \ngit-pack-objects to totally disable delta compression), but in general you \ncan't actually just link the files over like with \"git clone\", because \nthat would create total chaos and a real mess if the other end was packed.\n\nSo \"git pull\" actually needs to copy one object at a time in order to have \nsensible semantics together with \"git repack\". Now, you could make that \n\"one object at a time\" thing have its own special cases (\"if it's packed, \nextract it as a unpacked object in the destination, if it's unpacked, just \nlink it if you can\"), but it would just be pretty ugly.\n\nIf it ever gets to be a real performance problem, we can certainly fix it,\nbut in the meantime I _much_ prefer having one single path. I dislike the\nrsync (and the http) paths immensely already, but at least I don't have to\nuse them..\n\n\t\t\tLinus\n"}]}