{"thread":{"id":"2552","subject":"[QUESTION] Access to a huge GIT repository.","startedAt":"2005-11-16T12:24:25Z","lastAt":"2005-11-22T19:10:26Z","messageCount":22,"participants":["Franck","Linus Torvalds","Junio C Hamano","Martin Langhoff","Lukas Sandström"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"12012","messageId":"cda58cb80511160424j1acac7c6j@mail.gmail.com","threadId":"2552","inReplyTo":null,"subject":"[QUESTION] Access to a huge GIT repository.","fromName":"Franck","fromEmail":"vagabon.xyz@gmail.com","sentAt":"2005-11-16T12:24:25Z","receivedAt":"2005-11-16T12:24:25Z","isPatch":false,"sender":{"key":"vagabon.xyz@gmail.com","avatar":null},"body":"Hi,\n\nI'm trying to clone a small part of a big repository. This repository\ncontains several branchs that are useless for me. Actually this\nrepository is the linux-mips one, and branchs are used to track each\nkernel minor version. That is to say it contains 4 branchs which are\nlinux-2.0, linux-2.2, linux-2.4 and master (linux-2.6).\n\nI'd like to clone this repository without grabbing linux-2.0,\nlinux-2.2, linux-2.4 branchs. I tried several things like:\n\n        $ git init-db\n        $ git fetch rsync://ftp.linux-mips.org/git/linux.git master\n\nBut all tries download every objects of each branchs. I believe that's\nbecause of (a) the protocol used to access of the remote repo (b) the\nmaster branch has been created from linux-2.4 branch so its first\ncommit object contains a branch 2.4 commit obj as parent object (let's\ncall it the \"father\" object). Is that correct ?\n\nSo I downloaded the whole thing, and try to remove \"father\" object and\nright after run a 'git prune'. But unfortunately I can't find it\nanywhere in .git directory. I did:\n\n        $ git-verify-pack < .git/objects/pack/*.idx\n        $ git-unpack-objects < .git/objects/pack/*.pack\n\nBut I can't find \"father object\" anywhere in .git/objects directory.\nStill it's referenced by .git/objects/pack/pack-....idx file.\n\nCan anybody give me some advices ?\n\nThanks\n--\n               Franck\n"},{"id":"12018","messageId":"Pine.LNX.4.64.0511160837000.13959@g5.osdl.org","threadId":"2552","inReplyTo":"cda58cb80511160424j1acac7c6j@mail.gmail.com","subject":"Re: [QUESTION] Access to a huge GIT repository.","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-11-16T16:46:01Z","receivedAt":"2005-11-16T16:46:01Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Wed, 16 Nov 2005, Franck wrote:\n> \n> I'm trying to clone a small part of a big repository. This repository\n> contains several branchs that are useless for me. Actually this\n> repository is the linux-mips one, and branchs are used to track each\n> kernel minor version. That is to say it contains 4 branchs which are\n> linux-2.0, linux-2.2, linux-2.4 and master (linux-2.6).\n> \n> I'd like to clone this repository without grabbing linux-2.0,\n> linux-2.2, linux-2.4 branchs. I tried several things like:\n> \n>         $ git init-db\n>         $ git fetch rsync://ftp.linux-mips.org/git/linux.git master\n\nFirst off, \"rsync://\" will never do what you want. It uses rsync \n(surprised surprise) to grab the objects, so since it has no clue what the \nobjects are, it has no choice but to just grab them all.\n\n> But all tries download every objects of each branchs. I believe that's\n> because of (a) the protocol used to access of the remote repo\n\nYes. Note that ftp.linux-mips.org does run the git daemon, so you can use \n\"git://\" instead of \"rsync://\" (that's how I pull from them when I sync).\n\n>\t\t\t\t\t\t\t\t (b) the\n> master branch has been created from linux-2.4 branch so its first\n> commit object contains a branch 2.4 commit obj as parent object (let's\n> call it the \"father\" object). Is that correct ?\n\nNot having looked at that particular repo, I don't know. git can do it \neither way - either one long common history, or branches with totally \nunrelated histories. \n\nThe fact that _some_ of the linux-mips repositories are based on mine \nmakes me suspect that all their 2.6-based ones are rooted like mine is, \nbut that may or may not be true.\n\nJust try your above command line with \"git://\" instead.\n\n(NOTE! Doing a full clone like the above is pretty expensive with git for \nthe server side, so it might take a while before it starts feeding you \nstuff if it is under heavy load)\n\n> So I downloaded the whole thing, and try to remove \"father\" object and\n> right after run a 'git prune'. But unfortunately I can't find it\n> anywhere in .git directory. I did:\n> \n>         $ git-verify-pack < .git/objects/pack/*.idx\n>         $ git-unpack-objects < .git/objects/pack/*.pack\n> \n> But I can't find \"father object\" anywhere in .git/objects directory.\n> Still it's referenced by .git/objects/pack/pack-....idx file.\n> \n> Can anybody give me some advices ?\n\nIf you want to get rid of other branches, do:\n\n\t# Remove the branches\n\tgit branch -D linux-2.0\n\tgit branch -D linux-2.2\n\tgit branch -D linux-2.4\n\t.. whatever other branches you don't want ..\n\n\t# Repack the repo\n\tgit repack -a -d\n\n\t# Prune it all down\n\tgit prune-packed\n\tgit prune\n\nand you should have a nice single pack that only contains the branch(es) \nthat you're interested in.\n\n\t\tLinus\n"},{"id":"12026","messageId":"7vveys8n8q.fsf@assigned-by-dhcp.cox.net","threadId":"2552","inReplyTo":"cda58cb80511160424j1acac7c6j@mail.gmail.com","subject":"Re: [QUESTION] Access to a huge GIT repository.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-11-16T18:24:21Z","receivedAt":"2005-11-16T18:24:21Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Franck <vagabon.xyz@gmail.com> writes:\n\n> I'm trying to clone a small part of a big repository.\n\nWhat Linus has already said...\n\nThis is the second time this week the issue came up, so maybe\n\"large bundled repository, whose users typically are interested\nin only one branch\" may not be so uncommon as I first expected.\nAn optional form of 'git-clone' that clones only from a limited\nsubset of branches might be useful.\n\nThe underlying network transfer program, 'git-clone-pack', can\nbe told to clone only specified branches.  If somebody is\ninterested, updating the 'git-clone' wrapper to use it should\nnot be too hard -- obviously this needs to be done for other\ntransports as well, though.\n"},{"id":"12037","messageId":"46a038f90511161201h54c5fa73l20b5dcf3b5c19399@mail.gmail.com","threadId":"2552","inReplyTo":"7vveys8n8q.fsf@assigned-by-dhcp.cox.net","subject":"Re: [QUESTION] Access to a huge GIT repository.","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2005-11-16T20:01:04Z","receivedAt":"2005-11-16T20:01:04Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On 11/17/05, Junio C Hamano <junkio@cox.net> wrote:\n> The underlying network transfer program, 'git-clone-pack', can\n> be told to clone only specified branches.  If somebody is\n> interested, updating the 'git-clone' wrapper to use it should\n> not be too hard -- obviously this needs to be done for other\n> transports as well, though.\n\ncg-clone already does this. One tricky thing with the selective\ncloning is that you want to pull the named head plus all its related\nobjects, and then pull only the _relevant_ tags. There's been\ndiscussion of pulling tagsrefs+related tag objects and then pruning\nany tagrefs+tagobjects where the commit is unreachable with the\nobjects you already have. You surely remember my 'git-rev-parse  is\ncrashing' thread.\n\nIf you just pull tagrefs and all the objects needed for them, chances\nare you'll get the whole repo anyway ;-)\n\ncheers,\n\n\nmartin\n"},{"id":"12039","messageId":"Pine.LNX.4.64.0511161208140.13959@g5.osdl.org","threadId":"2552","inReplyTo":"46a038f90511161201h54c5fa73l20b5dcf3b5c19399@mail.gmail.com","subject":"Re: [QUESTION] Access to a huge GIT repository.","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-11-16T20:10:56Z","receivedAt":"2005-11-16T20:10:56Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 17 Nov 2005, Martin Langhoff wrote:\n>\n> cg-clone already does this. One tricky thing with the selective\n> cloning is that you want to pull the named head plus all its related\n> objects, and then pull only the _relevant_ tags.\n\nWell, if you keep to native git protocols, you can trivially do that by \njust fetching the required heads, and then fetching only the tags for \nwhich you have the pointed-to object (ie look for the ^{} thing in \ngit-ls-remote, and check if you have that object, then get those tags).\n\nFor rsync, since you get all objects anyway, there's no point to limiting \nthe branches. Might as well just delete them and prune them.\n\n\t\tLinus\n"},{"id":"12045","messageId":"7vwtj85o1y.fsf@assigned-by-dhcp.cox.net","threadId":"2552","inReplyTo":"46a038f90511161201h54c5fa73l20b5dcf3b5c19399@mail.gmail.com","subject":"Re: [QUESTION] Access to a huge GIT repository.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-11-16T20:35:05Z","receivedAt":"2005-11-16T20:35:05Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Martin Langhoff <martin.langhoff@gmail.com> writes:\n\n> On 11/17/05, Junio C Hamano <junkio@cox.net> wrote:\n>> The underlying network transfer program, 'git-clone-pack', can\n>> be told to clone only specified branches.  If somebody is\n>> interested, updating the 'git-clone' wrapper to use it should\n>> not be too hard -- obviously this needs to be done for other\n>> transports as well, though.\n>\n> cg-clone already does this.\n\nYes, but it can be improved.  It does fetch-pack for git native\ntransports, which means the receiving end expands the pack.  I\nthink we would either need to teach cg-clone to tell cg-fetch\nthat it is cloning and invoke git-clone-pack, or add an optional\n\"do not expand\" option to git-fetch-pack (git-clone-pack always\nkeeps the downloaded pack unexpanded).  Huge pack transfer\nspends long time on the downloader side unpacking the pack.\n\n> If you just pull tagrefs and all the objects needed for them, chances\n> are you'll get the whole repo anyway ;-)\n\nYes, that's why we changed ls-remote to report the magic ^{}\nentries and it is used in cg-fetch::fetch_tags.\n"},{"id":"12085","messageId":"cda58cb80511170236p4a7e2baay@mail.gmail.com","threadId":"2552","inReplyTo":"Pine.LNX.4.64.0511160837000.13959@g5.osdl.org","subject":"Re: [QUESTION] Access to a huge GIT repository.","fromName":"Franck","fromEmail":"vagabon.xyz@gmail.com","sentAt":"2005-11-17T10:36:47Z","receivedAt":"2005-11-17T10:36:47Z","isPatch":false,"sender":{"key":"vagabon.xyz@gmail.com","avatar":null},"body":"Hi Linus\n\n2005/11/16, Linus Torvalds <torvalds@osdl.org>:\n>\n>\n> First off, \"rsync://\" will never do what you want. It uses rsync\n> (surprised surprise) to grab the objects, so since it has no clue what the\n> objects are, it has no choice but to just grab them all.\n>\n\nyes that was a bad example, actually I've tried several protocol\nincluding git protocol.\n\n> The fact that _some_ of the linux-mips repositories are based on mine\n> makes me suspect that all their 2.6-based ones are rooted like mine is,\n> but that may or may not be true.\n>\n\nhmm, don't think \"linux mips\" repository is rooted like yours. It\nseems that it has been imported from mips cvs repository which\ncontains linux-mips history since 1995.\n\n> Just try your above command line with \"git://\" instead.\n>\n> (NOTE! Doing a full clone like the above is pretty expensive with git for\n> the server side, so it might take a while before it starts feeding you\n> stuff if it is under heavy load)\n>\n\nwell I've already tried that but have renounced after waiting more\nthan 4 hours ! I don't know if the server was under heavy load or if\ngit protocol needs a lots of resources but it seems useless to setup a\ngit dameon on it...\n\n> If you want to get rid of other branches, do:\n\nSorry, I forget to tell that I have already tried what you suggested\nat first (except that I did not do a 'git repack -a -d') but it didn't\nwork out (and that the reason why I tried the \"kill father object\"\nthing). Since I forgot the repack thing, I retried again, and it did\nlast more than 4 hours to cpmplete all git commands. After that I run\ngitk --all to check that all old branch's objects have been removed\nbut I can see all of them.\n\nI plan to take look at git code to understand how object removal is\ndone, but until this do you have any ideas why this fails ?\n\nThanks\n--\n               Franck\n"},{"id":"12119","messageId":"Pine.LNX.4.64.0511170817480.13959@g5.osdl.org","threadId":"2552","inReplyTo":"cda58cb80511170236p4a7e2baay@mail.gmail.com","subject":"Re: [QUESTION] Access to a huge GIT repository.","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-11-17T16:23:07Z","receivedAt":"2005-11-17T16:23:07Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 17 Nov 2005, Franck wrote:\n> \n> Sorry, I forget to tell that I have already tried what you suggested\n> at first (except that I did not do a 'git repack -a -d') but it didn't\n> work out (and that the reason why I tried the \"kill father object\"\n> thing).\n\nIf that didn't work, then you can't kill the father object: it will not \nonly just get rid of one single object, but it will make fsck unhappy that \nyour tree is incomplete.\n\nWhat you probably _can_ do is to find whatever top-most commit you want \n(say, the v2.6.0 commit), and use grafting to make that have no parents. \nThen you can do git-prune to get rid of everything under it.\n\n> Since I forgot the repack thing, I retried again, and it did\n> last more than 4 hours to cpmplete all git commands.\n\nSounds like the original archive was fully unpacked. That really does get \nvery very slow after a while.\n\n>\t\t After that I run\n> gitk --all to check that all old branch's objects have been removed\n> but I can see all of them.\n\nNote that if the branches are still there, git prune won't do anything to \nthem (and git repack will pack them all). You need to remove the branches \n_first_, so that git repack and prune sees that their objects aren't \nneeded.\n\nBut yes, if the tree is actually based on some full CVS history and really \ngoes back all the way to 1995, then you can't prune it (since all branches \nare actually part of the history from the very top), and you'd need to \ngraft away part of it like above.\n\n\t\tLinus\n"},{"id":"12148","messageId":"cda58cb80511171347yef4f090g@mail.gmail.com","threadId":"2552","inReplyTo":"Pine.LNX.4.64.0511170817480.13959@g5.osdl.org","subject":"Re: [QUESTION] Access to a huge GIT repository.","fromName":"Franck","fromEmail":"vagabon.xyz@gmail.com","sentAt":"2005-11-17T21:47:20Z","receivedAt":"2005-11-17T21:47:20Z","isPatch":false,"sender":{"key":"vagabon.xyz@gmail.com","avatar":null},"body":"2005/11/17, Linus Torvalds <torvalds@osdl.org>:\n>\n> What you probably _can_ do is to find whatever top-most commit you want\n> (say, the v2.6.0 commit), and use grafting to make that have no parents.\n> Then you can do git-prune to get rid of everything under it.\n>\n\nok that's what I was trying to do by killing the parent object.  Now\nwhen looking a the graph using gitk all old objects have been removed.\nBut I'm suprised because the git repository is the same size as it was\nbefore pruning all old objects. Can you explain why ?\n\nThanks\n--\n               Franck\n"},{"id":"12154","messageId":"Pine.LNX.4.64.0511171443000.13959@g5.osdl.org","threadId":"2552","inReplyTo":"cda58cb80511171347yef4f090g@mail.gmail.com","subject":"Re: [QUESTION] Access to a huge GIT repository.","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-11-17T22:44:09Z","receivedAt":"2005-11-17T22:44:09Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 17 Nov 2005, Franck wrote:\n\n> 2005/11/17, Linus Torvalds <torvalds@osdl.org>:\n> >\n> > What you probably _can_ do is to find whatever top-most commit you want\n> > (say, the v2.6.0 commit), and use grafting to make that have no parents.\n> > Then you can do git-prune to get rid of everything under it.\n> >\n> \n> ok that's what I was trying to do by killing the parent object.  Now\n> when looking a the graph using gitk all old objects have been removed.\n> But I'm suprised because the git repository is the same size as it was\n> before pruning all old objects. Can you explain why ?\n\nmake sure you re-pack if it was packed. \"git prune\" will not remove packs \nat all, so..\n\n\t\tLinus\n"},{"id":"12324","messageId":"cda58cb80511190423w1e46bf5bu@mail.gmail.com","threadId":"2552","inReplyTo":"Pine.LNX.4.64.0511171443000.13959@g5.osdl.org","subject":"Re: [QUESTION] Access to a huge GIT repository.","fromName":"Franck","fromEmail":"vagabon.xyz@gmail.com","sentAt":"2005-11-19T12:23:40Z","receivedAt":"2005-11-19T12:23:40Z","isPatch":false,"sender":{"key":"vagabon.xyz@gmail.com","avatar":null},"body":"2005/11/17, Linus Torvalds <torvalds@osdl.org>:\n>\n> > On Thu, 17 Nov 2005, Franck wrote:\n> >\n> > ok that's what I was trying to do by killing the parent object.  Now\n> > when looking a the graph using gitk all old objects have been removed.\n> > But I'm suprised because the git repository is the same size as it was\n> > before pruning all old objects. Can you explain why ?\n>\n> make sure you re-pack if it was packed. \"git prune\" will not remove packs\n> at all, so..\n>\n\nI just looked at git-prune script and it seems to remove unreachable\nobjects only in .git/objects/[0-9a-f][0-9a-f] directories, not in pack\nfiles.\n\nThen by running git-repack -a -d, I build a new small pack that\ncontains only latest objects, but then the script runs\ngit-redundant-pack script which erases the new small one since all its\nobjects are included in the old big one. Is that correct ? If so\ngit-redundant-pack script could return the oldest redundant ?\n\nThanks\n--\n               Franck\n"},{"id":"12325","messageId":"437F1E67.8020302@etek.chalmers.se","threadId":"2552","inReplyTo":"cda58cb80511190423w1e46bf5bu@mail.gmail.com","subject":"Re: [QUESTION] Access to a huge GIT repository.","fromName":"Lukas Sandström","fromEmail":"lukass@etek.chalmers.se","sentAt":"2005-11-19T12:45:27Z","receivedAt":"2005-11-19T12:45:27Z","isPatch":false,"sender":{"key":"luksan@gmail.com","avatar":"https://avatars.githubusercontent.com/u/152281?v=4"},"body":"Franck wrote:\n> 2005/11/17, Linus Torvalds <torvalds@osdl.org>:\n> \n>>>On Thu, 17 Nov 2005, Franck wrote:\n>>>\n>>>ok that's what I was trying to do by killing the parent object.  Now\n>>>when looking a the graph using gitk all old objects have been removed.\n>>>But I'm suprised because the git repository is the same size as it was\n>>>before pruning all old objects. Can you explain why ?\n>>\n>>make sure you re-pack if it was packed. \"git prune\" will not remove packs\n>>at all, so..\n>>\n> \n> \n> I just looked at git-prune script and it seems to remove unreachable\n> objects only in .git/objects/[0-9a-f][0-9a-f] directories, not in pack\n> files.\n> \n> Then by running git-repack -a -d, I build a new small pack that\n> contains only latest objects, but then the script runs\n> git-redundant-pack script which erases the new small one since all its\n> objects are included in the old big one. Is that correct ? If so\n> git-redundant-pack script could return the oldest redundant ?\n> \n> Thanks\n> --\n>                Franck\n\nThe reason the old pack is kept instead of the new one is that it\nis a proper superset of the new one. \n\nThe \"git-repack -a -d\" case is fixed in Junios master, and I have sent out\na patch which lets you pipe git-fsck-objects --full --unreachable to\ngit-pack-redundant in the other cases, so the problem should hopefully\nbe resolved shortly.\n\n/Lukas\n"},{"id":"12336","messageId":"Pine.LNX.4.64.0511190953520.13959@g5.osdl.org","threadId":"2552","inReplyTo":"cda58cb80511190423w1e46bf5bu@mail.gmail.com","subject":"Re: [QUESTION] Access to a huge GIT repository.","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-11-19T17:56:59Z","receivedAt":"2005-11-19T17:56:59Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Sat, 19 Nov 2005, Franck wrote:\n> \n> Then by running git-repack -a -d, I build a new small pack that\n> contains only latest objects, but then the script runs\n> git-redundant-pack script which erases the new small one since all its\n> objects are included in the old big one. Is that correct ?\n\nGaah.\n\nThe git-redundant-pack logic has totally broken \"git repack -a -d\"\n\nJunio, can we undo that, please? With \"-a\", redundant pack removal is \ntrivial (and we used to do it right, exactly because it is trivial), and \nwithout \"-a\", redundant pack removal is pointless (since it will never \npack objects that are already packed).\n\nSo \"git repack\" should _never_ call git-redundant-pack. It's always either \nwrong or pointless.\n\ngit-redundant-pack is great for \"git prune\", not for repack.\n\n\t\tLinus\n"},{"id":"12338","messageId":"7vzmo04dpl.fsf@assigned-by-dhcp.cox.net","threadId":"2552","inReplyTo":"Pine.LNX.4.64.0511190953520.13959@g5.osdl.org","subject":"Re: [QUESTION] Access to a huge GIT repository.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-11-19T19:52:54Z","receivedAt":"2005-11-19T19:52:54Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Linus Torvalds <torvalds@osdl.org> writes:\n\n> So \"git repack\" should _never_ call git-redundant-pack. It's always either \n> wrong or pointless.\n\nRight-o.\n"},{"id":"12342","messageId":"7vu0e84bfq.fsf@assigned-by-dhcp.cox.net","threadId":"2552","inReplyTo":"437F1E67.8020302@etek.chalmers.se","subject":"Re: [QUESTION] Access to a huge GIT repository.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-11-19T20:42:01Z","receivedAt":"2005-11-19T20:42:01Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Lukas Sandström <lukass@etek.chalmers.se> writes:\n\n> The \"git-repack -a -d\" case is fixed in Junios master, and I have sent out\n> a patch which lets you pipe git-fsck-objects --full --unreachable to\n> git-pack-redundant in the other cases, so the problem should hopefully\n> be resolved shortly.\n\nAfter thinking a bit about it, I tend to agree with what Linus\nsaid, and would vote for the simplicity of the older way.  From\nusage point of view, git-repack is primarily about repacking,\nand the -d option is a funny exception applicable to -a case,\nonly because it is so obvious (i.e. after repacking everything\ninto one, it is obvious any other packs are unneeded).  We might\neven want to make the \"-a\" flag to imply \"-a -d\" there.\n"},{"id":"12466","messageId":"cda58cb80511211211l2ff6ff12j@mail.gmail.com","threadId":"2552","inReplyTo":"7vzmo04dpl.fsf@assigned-by-dhcp.cox.net","subject":"Re: [QUESTION] Access to a huge GIT repository.","fromName":"Franck","fromEmail":"vagabon.xyz@gmail.com","sentAt":"2005-11-21T20:11:27Z","receivedAt":"2005-11-21T20:11:27Z","isPatch":false,"sender":{"key":"vagabon.xyz@gmail.com","avatar":null},"body":"2005/11/19, Junio C Hamano <junkio@cox.net>:\n> Linus Torvalds <torvalds@osdl.org> writes:\n>\n> > So \"git repack\" should _never_ call git-redundant-pack. It's always either\n> > wrong or pointless.\n>\n> Right-o.\n>\n\nOk, it works now. My new git repository is only 60Mo. But :), I would\nlike to make up my public repository based on this \"light\" repository.\nAnd if someone has the big repository, I would like him to be able to\npull my public repo into his one. But since I used grafting to \"cut\"\nmy light repo and .git/info/grafts file is not copied during\npush/pull/clone operations it's not going to work. Is it a scheme that\ncould work ?\n\nMoreover, I'm wondering if my public repository really needs to store\nbig repo's pack files as it is described in git tutorial ?\n\nThanks\n--\n               Franck\n"},{"id":"12469","messageId":"7vhda5pw6l.fsf@assigned-by-dhcp.cox.net","threadId":"2552","inReplyTo":"cda58cb80511211211l2ff6ff12j@mail.gmail.com","subject":"Re: [QUESTION] Access to a huge GIT repository.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-11-21T20:45:06Z","receivedAt":"2005-11-21T20:45:06Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Franck <vagabon.xyz@gmail.com> writes:\n\n> ... But since I used grafting to \"cut\"\n> my light repo and .git/info/grafts file is not copied during\n> push/pull/clone operations it's not going to work. Is it a scheme that\n> could work ?\n\nIf you tell your downloaders that your repository is incomplete\nand they need to have at least up to such and such commits from\nanother repository, they should be able to slurp from you.\n\nIt might be possible to teach upload-pack (that is run when your\ndownloaders run git-fetch or git-clone against your repository)\nto somehow send a customized error message to the client when it\nfinds the other end needs certain objects that you yourself do\nnot even have. In that message you could say something like \"due\nto space constraints this repository is an incomplete one, and\nyou can only use it on top of a clone of such and such\nrepository, found at this URL: ...\".\n\n> Moreover, I'm wondering if my public repository really needs to store\n> big repo's pack files as it is described in git tutorial ?\n\nCopying is one way, but if the repository you are borrowing from\nlives on the same machine, you could use objects/info/alternates\nto point at it from your public repository.  In either case, the\npoint in that section is to reduce the need to transfer and\nexpand much stuff in your public repository while keeping the\nrepository complete.  The recommended procedure in the tutorial\nalways assumes that the public repository is kept fsck-objects\nclean and complete.\n\nWhat you are trying to do is to keep your public repository\nfsck-objects *un*clean and still let downloaders work with it;\nso I suspect following that section of the tutorial procedure\ndefeats the purpose your experiments.\n"},{"id":"12530","messageId":"cda58cb80511220122r76ca69a2y@mail.gmail.com","threadId":"2552","inReplyTo":"7vhda5pw6l.fsf@assigned-by-dhcp.cox.net","subject":"Re: [QUESTION] Access to a huge GIT repository.","fromName":"Franck","fromEmail":"vagabon.xyz@gmail.com","sentAt":"2005-11-22T09:22:46Z","receivedAt":"2005-11-22T09:22:46Z","isPatch":false,"sender":{"key":"vagabon.xyz@gmail.com","avatar":null},"body":"2005/11/21, Junio C Hamano <junkio@cox.net>:\n> Franck <vagabon.xyz@gmail.com> writes:\n>\n> > ... But since I used grafting to \"cut\"\n> > my light repo and .git/info/grafts file is not copied during\n> > push/pull/clone operations it's not going to work. Is it a scheme that\n> > could work ?\n>\n> If you tell your downloaders that your repository is incomplete\n> and they need to have at least up to such and such commits from\n> another repository, they should be able to slurp from you.\n>\n\nWhat do you mean by \"have at least up to such and such commits\" ? I\ncan see only one commit that they need: the one I used to create my\npublic repository...\n\n> It might be possible to teach upload-pack (that is run when your\n> downloaders run git-fetch or git-clone against your repository)\n> to somehow send a customized error message to the client when it\n> finds the other end needs certain objects that you yourself do\n> not even have. In that message you could say something like \"due\n> to space constraints this repository is an incomplete one, and\n> you can only use it on top of a clone of such and such\n> repository, found at this URL: ...\".\n>\n\nThat's a good idea. We get the same thing when cloning linux\nrepository. BTW how is it done in that case ?\n\n> > Moreover, I'm wondering if my public repository really needs to store\n> > big repo's pack files as it is described in git tutorial ?\n>\n> What you are trying to do is to keep your public repository\n> fsck-objects *un*clean and still let downloaders work with it;\n> so I suspect following that section of the tutorial procedure\n> defeats the purpose your experiments.\n>\n\nAbsolutely.  My question was not accurate sorry. It should have been\n\"can I have a public repository wiith fsck-objects unclean and with a\ngrafts file that should be downloaded when cloning it.\n\nThanks\n--\n               Franck\n"},{"id":"12531","messageId":"7vfyppf1va.fsf@assigned-by-dhcp.cox.net","threadId":"2552","inReplyTo":"cda58cb80511220122r76ca69a2y@mail.gmail.com","subject":"Re: [QUESTION] Access to a huge GIT repository.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-11-22T09:50:01Z","receivedAt":"2005-11-22T09:50:01Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Franck <vagabon.xyz@gmail.com> writes:\n\n> 2005/11/21, Junio C Hamano <junkio@cox.net>:\n>> Franck <vagabon.xyz@gmail.com> writes:\n>>\n>> > ... But since I used grafting to \"cut\"\n>> > my light repo and .git/info/grafts file is not copied during\n>> > push/pull/clone operations it's not going to work. Is it a scheme that\n>> > could work ?\n>>\n>> If you tell your downloaders that your repository is incomplete\n>> and they need to have at least up to such and such commits from\n>> another repository, they should be able to slurp from you.\n\nI was not talking about _your_ case specifically.  If you happen\nto have based your partial history on top of a single commit\nthen the set of \"such and such commits\" might be only one, but\nyou could for example clone from Linus tip, merge in a couple of\njgarzik branch heads, put your own commits on top of them and\nthen cauterize your history, stopping at those foreign commits.\nIn such a case you obviously need to tell others where you\nchopped your history off.\n"},{"id":"12533","messageId":"cda58cb80511220240u45267b18o@mail.gmail.com","threadId":"2552","inReplyTo":"7vfyppf1va.fsf@assigned-by-dhcp.cox.net","subject":"Re: [QUESTION] Access to a huge GIT repository.","fromName":"Franck","fromEmail":"vagabon.xyz@gmail.com","sentAt":"2005-11-22T10:40:36Z","receivedAt":"2005-11-22T10:40:36Z","isPatch":false,"sender":{"key":"vagabon.xyz@gmail.com","avatar":null},"body":"2005/11/22, Junio C Hamano <junkio@cox.net>:\n> I was not talking about _your_ case specifically.  If you happen\n> to have based your partial history on top of a single commit\n> then the set of \"such and such commits\" might be only one, but\n> you could for example clone from Linus tip, merge in a couple of\n> jgarzik branch heads, put your own commits on top of them and\n> then cauterize your history, stopping at those foreign commits.\n> In such a case you obviously need to tell others where you\n> chopped your history off.\n>\n\nI built the lite repository and got an error depending on which\noriginal repo I used to push the lite one. Here is the history:\n\n--------------------------------------------------------------\n#\n# building my pub repo from the \"lite\" repository\n#\n\n$ git checkout -b lite\n\n$ git-show-branch\n* [lite] Merge with Linux 2.6.14.\n ! [master] Merge with Linux 2.6.14.\n  ! [origin] Merge with db93a82fa9d8b4d6e31c227922eaae829253bb88.\n---\n\n# push the initial commit object\n\n$ git push /home/franck/pub/linux/git/linux-lite.git lite\nupdating 'refs/heads/lite'\n  from 0000000000000000000000000000000000000000\n  to   8643db584b46a61c968ae230897869f789bae020\nPacking 19558 objects\nUnpacking 19558 objects\n 100% (19558/19558) done\nrefs/heads/lite: 0000000000000000000000000000000000000000 ->\n8643db584b46a61c968ae230897869f789bae020\n\n#\n# push first changes from lite repositoy -> KO\n#\n$ making some hard work\n\n$ git commit -m \"Did some hard work\"\n$ git push /home/franck/pub/linux/git/linux-lite.git lite\nupdating 'refs/heads/lite'\n  from 8643db584b46a61c968ae230897869f789bae020\n  to   730518eea7523afd5b7891bb7849973cab52d963\nPacking 0 objects\nUnpacking 0 objects\n\nerror: unpack should have generated\n730518eea7523afd5b7891bb7849973cab52d963, but I can't find it!\n$\n\n#\n# push first changes from \"full\" repository -> OK\n#\n\n# go to full repository and checkout linux.2.6.14\n\n$ git commit -m \"Did some hard work\"\n$ git push /home/franck/pub/linux/git/linux-lite.git lite\nupdating 'refs/heads/lite'\n  from 8643db584b46a61c968ae230897869f789bae020\n  to   067d05600fe7251b8c923fbeb9ba0068ee272110\nPacking 108 objects\nUnpacking 108 objects\n 100% (108/108) done\nrefs/heads/lite: 8643db584b46a61c968ae230897869f789bae020 ->\n067d05600fe7251b8c923fbeb9ba0068ee272110\n--------------------------------------------------------------\n\nIt seems that the \"lite\" repository can't be used as a working\nrepository. And If I use the last method to push some work, I can only\npull that changes from a full repository. From a lite one (without any\nchanges of course) I get this error:\n\n$ git pull /home/franck/pub/linux/git/linux-lite.git lite\nPacking 108 objects\nUnpacking 108 objects\n 100% (108/108) done\nMerging HEAD with f42aaff3bf8041c2d43f1ff6fdfe5df6e8a5b00b\nMerging:\n8643db584b46a61c968ae230897869f789bae020 Merge with Linux 2.6.14.\nf42aaff3bf8041c2d43f1ff6fdfe5df6e8a5b00b Did some hard work\nfound 0 common ancestor(s):\nTraceback (most recent call last):\n  File \"/home/fbuihuu/bin/git-merge-recursive\", line 870, in ?\n    firstBranch, secondBranch, graph)\n  File \"/home/fbuihuu/bin/git-merge-recursive\", line 67, in merge\n    mergedCA = ca[0]\nIndexError: list index out of range\nNo merge strategy handled the merge.\n\nDo you have any clues ?\n\nThanks\n--\n               Franck\n"},{"id":"12545","messageId":"7v1x18fw8j.fsf@assigned-by-dhcp.cox.net","threadId":"2552","inReplyTo":"cda58cb80511220240u45267b18o@mail.gmail.com","subject":"Re: [QUESTION] Access to a huge GIT repository.","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-11-22T17:06:20Z","receivedAt":"2005-11-22T17:06:20Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Franck <vagabon.xyz@gmail.com> writes:\n\n> It seems that the \"lite\" repository can't be used as a working\n> repository.\n>...\n> Do you have any clues ?\n\nWere you around on this list, around the beginning of this\nmonth?  The thread that starts here may be of interest:\n\n\thttp://marc.theaimsgroup.com/?l=git&m=113089701819420\n\nI think (although I do not exactly know how the \"lite\"\nrepository was constructed) what you are doing is similar to\nwhat I called \"shallow clone\" there in that thread.  At the end\nof the thread I think I listed what you can and cannot do in\nsuch an incomplete repository.\n"},{"id":"12556","messageId":"cda58cb80511221110i3d7ea4dj@mail.gmail.com","threadId":"2552","inReplyTo":"7v1x18fw8j.fsf@assigned-by-dhcp.cox.net","subject":"Re: [QUESTION] Access to a huge GIT repository.","fromName":"Franck","fromEmail":"vagabon.xyz@gmail.com","sentAt":"2005-11-22T19:10:26Z","receivedAt":"2005-11-22T19:10:26Z","isPatch":false,"sender":{"key":"vagabon.xyz@gmail.com","avatar":null},"body":"2005/11/22, Junio C Hamano <junkio@cox.net>:\n> Were you around on this list, around the beginning of this\n> month?\n\nnope, I subscribed to the list one week later.\n\n> The thread that starts here may be of interest:\n>\n>         http://marc.theaimsgroup.com/?l=git&m=113089701819420\n>\n> I think (although I do not exactly know how the \"lite\"\n> repository was constructed) what you are doing is similar to\n> what I called \"shallow clone\" there in that thread.\n\nIndeed. What I'm doing to get my \"shallow clone\" is basicaly remove\nevery ref/branchs I don't want, then run git-prune, git-repack -a -d.\nBut the result should be the same as your \"shallow clone\" method.\n\n>  At the end\n> of the thread I think I listed what you can and cannot do in\n> such an incomplete repository.\n>\n\nYes. But the big difference is that, in my case, the shallow copy is\nused as a public repository, whereas in your case the shallow copy is\nused as a working repository. Anyway, Ralf (the mips arch maintainer)\nis going to create a new pruned repository that should resolve my\nproblem. I'm going to wait for it instead of trying to make my own\n\"broken\" repository.\n\nThanks\n--\n               Franck\n"}]}