{"thread":{"id":"2307","subject":"Clone a repository with only the objects needed for a single tag","startedAt":"2005-11-02T02:02:47Z","lastAt":"2005-11-02T09:20:42Z","messageCount":6,"participants":["Ben Lau","Junio C Hamano","Andreas Ericsson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"10981","messageId":"43681E47.4010203@ust.hk","threadId":"2307","inReplyTo":null,"subject":"Clone a repository with only the objects needed for a single tag","fromName":"Ben Lau","fromEmail":"benlau@ust.hk","sentAt":"2005-11-02T02:02:47Z","receivedAt":"2005-11-02T02:02:47Z","isPatch":false,"sender":{"key":"benlau@ust.hk","avatar":null},"body":"Hi,\n\n   Is there any method to clone/copy a repository with only the objects \nneeded\nfor a single tag in order to save disk space?\n\n   For example, if I want to start a new project based on a specific \nversion of kernel\nlike v2.6.14. I would run `git-clone` and then checkout a new branch \nbased on the tag.\nHowever, one of the development host is a notebook computer which has \nonly 2GB space\nleave. Therefore I hope the space occupied by the respository could be \nmininiumized\nwhile it keeps to be able to fetch/update from my master repository.\n\n  Thanks for any advise.\n"},{"id":"10993","messageId":"7vy847y1m2.fsf@assigned-by-dhcp.cox.net","threadId":"2307","inReplyTo":"43681E47.4010203@ust.hk","subject":"Re: Clone a repository with only the objects needed for a single tag","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-11-02T07:01:41Z","receivedAt":"2005-11-02T07:01:41Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Lau <benlau@ust.hk> writes:\n\n>   Is there any method to clone/copy a repository with only the\n> objects needed for a single tag in order to save disk space?\n>   For example, if I want to start a new project based on a\n> specific version of kernel like v2.6.14. I would run\n> `git-clone` and then checkout a new branch based on the tag.\n\nDepends on what you want to do in that shallow copy.\n\nIf the only thing you would want to do is to build it, then you\ncould 'git-tar-tree v2.6.14' and extract that on your notebook.\nThe output is just a tar so there will no history, though.\n\nIf you want to develop while on the road, but do not\nparticularly need to be able to inspect the history beyond the\npoint you started, you could create a deliberately broken\nrepository, using the git-shallow-clone script (attached), like\nthis:\n\n    $ git clone -n $mothership satellite\n    $ cd satellite\n    $ git-shallow-pack --all\n    $ rm -f .git/objects/pack/pack-*\n    $ mv pack-* .git/objects/pack/.\n\nIf the original repository you are cloning from is local, you\ncould instead do:\n\n    $ git clone -l -s -n $mothership satellite\n    $ cd satellite\n    $ git-shallow-pack --all\n    $ rm .git/objects/info/alternates\n    $ mv pack-* .git/objects/pack/.\n\nYou could develop in this repository, even build up your own\ncommit chains, and when you come back you could push from this\nrepository back to your 'mothership' repository.  In essense,\nany operation that does not require you to have full history\nshould work.\n\nOne important thing that would not always work would be to pull\ninto this repository over git-aware protocols, although pulling\nfrom your 'mothership' repository would probably work most of\nthe time.\n\nOne case that would probably break is if the mothership side\nreverted a commit beyond this shallow-pack boundary and then you\ntry to pull from there.  After the revert, the trees and blobs\nin that new commit you will be pulling from the mothership are\nlikely to be the same as the ones contained in commits before\nthe shallow clone is made.  Because your satellite repo would\nclaim to have everything that is reachable from the tip (as of\nthe time the clone was made) of the branch, you cannot complain\nif the mothership side assumes you must have those blobs and\ntrees and did not send them to you when you pull.\n\n---\n#!/bin/sh\n# git-shallow-pack\n\ngit-rev-parse --revs-only --no-flags --default HEAD \"$@\" |\nwhile read sha1\ndo\n\techo \"$sha1\"\n\twhile type=`git-cat-file -t \"$sha1\"` &&\n\t      case \"$type\" in tag) ;; *) break ;; esac\n\tdo\n\t\tnext=`git-cat-file tag \"$sha1\" |\n\t\t\tsed -ne 's/^object //p' -e q`\n\t\techo \"$next\"\n\t\tsha1=\"$next\"\n\tdone\n\tgit-rev-parse --verify \"$sha1^{tree}\" 2>/dev/null &&\n\tgit-ls-tree -r \"$sha1\" | sed -e 's/^[0-7]* [^ ]* //'\ndone |\nsort -k 1,1 -u |\ngit-pack-objects pack\n"},{"id":"10999","messageId":"43687869.7060104@ust.hk","threadId":"2307","inReplyTo":"7vy847y1m2.fsf@assigned-by-dhcp.cox.net","subject":"Re: Clone a repository with only the objects needed for a single tag","fromName":"Ben Lau","fromEmail":"benlau@ust.hk","sentAt":"2005-11-02T08:27:21Z","receivedAt":"2005-11-02T08:27:21Z","isPatch":false,"sender":{"key":"benlau@ust.hk","avatar":null},"body":"Hi Junio,\n\n  It works!  Thanks a lot.\n\n  However, it has a problem when involves the gitk/git-log. Because the \nparent commit is missed in the shallow repository, gitk would complain \nthe object is missed and exit immediately. To solve the problem, i added \na pair of\nID of the new root object into .git/info/grafts.\n\nExample:\n\n  $ cat .git/info/grafts\n  741b2252a5e14d6c60a913c77a6099abe73a854a \n741b2252a5e14d6c60a913c77a6099abe73a854a\n\ngit-log/gitk do not complains afterward, but it also make gitk shows \nnothing during run. Any solution?\n\n By the way, although I am not sure do any other people also require \nthis feature, I wish the process could be more smooth.  As the \ngit-shallow-pack script do not destroy and modify any thing in the \ncloned repository beside the newly created pack file, I would suggest it \ncan run the script in the 'monthership' repository, take out\nthe pack file to another directory or remote machine. And then build a \nnew git repository based on the pack file.\n\n  To achieve the process, it need another script that could create a git \nrepository from pack file. Do any similar script existed?\n\nJunio C Hamano wrote:\n\n>Ben Lau <benlau@ust.hk> writes:\n>\n>  \n>\n>>  Is there any method to clone/copy a repository with only the\n>>objects needed for a single tag in order to save disk space?\n>>  For example, if I want to start a new project based on a\n>>specific version of kernel like v2.6.14. I would run\n>>`git-clone` and then checkout a new branch based on the tag.\n>>    \n>>\n>\n>Depends on what you want to do in that shallow copy.\n>\n>If the only thing you would want to do is to build it, then you\n>could 'git-tar-tree v2.6.14' and extract that on your notebook.\n>The output is just a tar so there will no history, though.\n>\n>If you want to develop while on the road, but do not\n>particularly need to be able to inspect the history beyond the\n>point you started, you could create a deliberately broken\n>repository, using the git-shallow-clone script (attached), like\n>this:\n>\n>    $ git clone -n $mothership satellite\n>    $ cd satellite\n>    $ git-shallow-pack --all\n>    $ rm -f .git/objects/pack/pack-*\n>    $ mv pack-* .git/objects/pack/.\n>\n>If the original repository you are cloning from is local, you\n>could instead do:\n>\n>    $ git clone -l -s -n $mothership satellite\n>    $ cd satellite\n>    $ git-shallow-pack --all\n>    $ rm .git/objects/info/alternates\n>    $ mv pack-* .git/objects/pack/.\n>\n>You could develop in this repository, even build up your own\n>commit chains, and when you come back you could push from this\n>repository back to your 'mothership' repository.  In essense,\n>any operation that does not require you to have full history\n>should work.\n>\n>One important thing that would not always work would be to pull\n>into this repository over git-aware protocols, although pulling\n>from your 'mothership' repository would probably work most of\n>the time.\n>\n>One case that would probably break is if the mothership side\n>reverted a commit beyond this shallow-pack boundary and then you\n>try to pull from there.  After the revert, the trees and blobs\n>in that new commit you will be pulling from the mothership are\n>likely to be the same as the ones contained in commits before\n>the shallow clone is made.  Because your satellite repo would\n>claim to have everything that is reachable from the tip (as of\n>the time the clone was made) of the branch, you cannot complain\n>if the mothership side assumes you must have those blobs and\n>trees and did not send them to you when you pull.\n>\n>---\n>#!/bin/sh\n># git-shallow-pack\n>\n>git-rev-parse --revs-only --no-flags --default HEAD \"$@\" |\n>while read sha1\n>do\n>\techo \"$sha1\"\n>\twhile type=`git-cat-file -t \"$sha1\"` &&\n>\t      case \"$type\" in tag) ;; *) break ;; esac\n>\tdo\n>\t\tnext=`git-cat-file tag \"$sha1\" |\n>\t\t\tsed -ne 's/^object //p' -e q`\n>\t\techo \"$next\"\n>\t\tsha1=\"$next\"\n>\tdone\n>\tgit-rev-parse --verify \"$sha1^{tree}\" 2>/dev/null &&\n>\tgit-ls-tree -r \"$sha1\" | sed -e 's/^[0-7]* [^ ]* //'\n>done |\n>sort -k 1,1 -u |\n>git-pack-objects pack\n>\n>\n>\n>\n>\n>  \n>\n"},{"id":"11002","messageId":"43687DAE.2010604@op5.se","threadId":"2307","inReplyTo":"43687869.7060104@ust.hk","subject":"Re: Clone a repository with only the objects needed for a single tag","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2005-11-02T08:49:50Z","receivedAt":"2005-11-02T08:49:50Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Ben Lau wrote:\n> Hi Junio,\n> \n>  It works!  Thanks a lot.\n> \n>  However, it has a problem when involves the gitk/git-log.\n\n\nBoth those programs use the history, which you don't have. With a \nshallow repository like this some commands just won't work. git-*log and \ngitk are among those.\n\n> \n> git-log/gitk do not complains afterward, but it also make gitk shows \n> nothing during run. Any solution?\n> \n\n* Get a second machine with more disk-space and run the history tools \nthere. If you use it as mothership and push your commits to it you'll be \nable to track your changes from the laptop as well.\n\n* Get a larger disk. They're not terribly expensive now adays.\n\n* Check out one tag (i.e. release) more than you need. Then you'll get \nhistory back to that tag.\n\n-- \nAndreas Ericsson                   andreas.ericsson@op5.se\nOP5 AB                             www.op5.se\nTel: +46 8-230225                  Fax: +46 8-230231\n"},{"id":"11004","messageId":"436882A1.1020207@ust.hk","threadId":"2307","inReplyTo":"43687DAE.2010604@op5.se","subject":"Re: Clone a repository with only the objects needed for a single tag","fromName":"Ben Lau","fromEmail":"benlau@ust.hk","sentAt":"2005-11-02T09:10:57Z","receivedAt":"2005-11-02T09:10:57Z","isPatch":false,"sender":{"key":"benlau@ust.hk","avatar":null},"body":"Andreas Ericsson wrote:\n\n> Ben Lau wrote:\n>\n>> Hi Junio,\n>>\n>>  It works!  Thanks a lot.\n>>\n>>  However, it has a problem when involves the gitk/git-log.\n>\n>\n>\n> Both those programs use the history, which you don't have. With a \n> shallow repository like this some commands just won't work. git-*log \n> and gitk are among those.\n\nyes, it is expected. However, I just think with a fake parent like what \n.git/info/grafts could provide may solve the issue. At least it make \ngit-log be happy to show the only log message in the shallow repository \nwithout any error.\nThe rest of problem is gitk could not show any item like what git-log shown.\n\nI am not sure is it right to have a pair of same ID into \n.git/info/grafts or it should be solved by a little patch to gitk.\n\n>\n>>\n>> git-log/gitk do not complains afterward, but it also make gitk shows \n>> nothing during run. Any solution?\n>>\n>\n> * Get a second machine with more disk-space and run the history tools \n> there. If you use it as mothership and push your commits to it you'll \n> be able to track your changes from the laptop as well.\n>\n> * Get a larger disk. They're not terribly expensive now adays.\n>\n> * Check out one tag (i.e. release) more than you need. Then you'll get \n> history back to that tag.\n>\n"},{"id":"11006","messageId":"7voe53v21h.fsf@assigned-by-dhcp.cox.net","threadId":"2307","inReplyTo":"43687869.7060104@ust.hk","subject":"Re: Clone a repository with only the objects needed for a single tag","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-11-02T09:20:42Z","receivedAt":"2005-11-02T09:20:42Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ben Lau <benlau@ust.hk> writes:\n\n>  However, it has a problem when involves the gitk/git-log.\n\nThat's why I said anything that requires you to have a complete\nhistory would not work.\n\nThe shallow repository is by definition a *broken* repository;\nthe refs are supposed to mean the repository has everything\nreachable from them, and many tools rely on that assumption, but\nthe shallow setup deliberately breaks that assumption, so you\nneed to be aware of what operations you can and cannot do\nwithout having the full history.  Although using grafts to\ncauterize somewhat helps as you discovered, you are operating in\n\"do it at your own risk\" territory.\n\nHaving said that, here are the things that *should* work without\nhaving full history (not many):\n\n        . git diff between your index, working tree, and commits\n          your shallow setup happens to have.\n\n        . git commit on top of any of the tip of branches your\n          shallow copy started out with, including git am and git\n          applymbox.\n\n        . git fetch/pull over commit walker from a remote\n          repository.\n\n        . git push to send the work done in the shallow repository\n          back into your mothership repository (running git pull on\n          the mothership to fetch from the shallow copy probably\n          would not work, unless you use commit walkers).\n\n\t. git whatchanged, git log, and gitk to view the work\n\t  you did in your shallow repository.\n\nCreating packs in the mothership repository is certainly\npossible.  You could instead do this:\n\n\t$ git-shallow-pack --all ;# in mothership\n        $ mkdir -p /var/tmp/shallow\n        $ tar cf - .git/HEAD .git/refs/ |\n          (cd /var/tmp/shallow; git-init-db; tar xf -)\n        $ mv pack-* /var/tmp/shallow/.git/objects/pack\n\nIf you want a bit deeper history, instead of giving '--all' to\ngit-shallow-pack, you could probably say something silly like\nthis:\n\n\t$ (git-rev-parse --all; git-rev-list --max-count=20 HEAD) |\n          xargs git-shallow-pack\n\nThe shallow-pack script I sent earlier is probably not very\nuseful in practice.  To polish it to be somewhat more useful, it\nwould probably need the following enhancements, at least:\n\n\t. Instead of taking 'git-rev-parse' arguments, take the\n\t  names of refs;\n\n\t. Instead of packing the objects contained in the\n          commits and trees the named refs reference, include\n          all blobs/trees/commits between the given refs and\n          their common ancestor to create a pack;\n\n\t. Create a tarball that contains:\n\n\t  (1) the pack file created by the above procedure,\n              stored in .git/objects/pack/.;\n\n          (2) .git/refs/ to be used in the shallow copy; this\n              should include only the refs given to the command\n              to create the above pack.\n\n\t  (3) .git/info/grafts to cauterize the common ancestor\n              commit and side branches merged into the lines you\n              are taking (computing the latter may be somewhat\n              expensive).\n\nThen the command can be run in the mothership repository like\nthis:\n\n\t$ cd linux-2.6\n        $ git-shallow-pack v2.6.14 v2.6.14-rc2 master\n\nto produce a tarball, which can be taken to another location.\nWhen extracted, it would contain all commits between the three\nnamed refs, and you could view the history across them.\n"}]}