{"thread":{"id":"14015","subject":"sharing object packs","startedAt":"2008-06-18T19:57:26Z","lastAt":"2008-06-19T17:01:39Z","messageCount":3,"participants":["marc.zonzon+git@gmail.com","Shawn O. Pearce","marc zonzon"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"80247","messageId":"20080618195726.GA20440@kernoel.dyndns.org","threadId":"14015","inReplyTo":null,"subject":"sharing object packs","fromName":"","fromEmail":"marc.zonzon+git@gmail.com","sentAt":"2008-06-18T19:57:26Z","receivedAt":"2008-06-18T19:57:26Z","isPatch":false,"sender":{"key":"marc.zonzon+git@gmail.com","avatar":null},"body":"Hello\n\nI have a big bare repository 'main.git' and many small git repositories sub1, sub2, ... subn.\nAll repositories lie in the same file file system, and each subx repository track and fetch main.git in a remote branch.\n\nI would like to avoid duplicating main.git objects, I have made some tries:\n\n- Putting a hard link to the pack in the object repository of main.git into subx object repository before fetching the main.git remote. \nIt works well... until the first repack on either side. \nNote that the problem is the same for any clone of a local repository, the hard link of packs vanish on the first repack. \nYou end up with a pack with the same objects, and so the same name, but organized in a different way so with an associated idx file and often a different file size.\n\n- Using an objects/info/alternates with the path of main.git object repository.\nIt work well too, but I import objects from main.git inside subx, and they don't have the same life time than those in main.git. So they can, disapear during a git-prune-packed or gc. (The same problem we have with: git clone --share)\n\n- I tried also to use git-relink to synchronize diverging repositories. But git-relink see different packs with the same name, (because repacked in a different way) and refuse to hard-link the packs but hard-links the .idx which has also an identical name and had the same size in my experiments.\nSo the .idx does no longer agree with the .pack and git fsck fail. Of course we can recover the repository by generating a new idx, but git-relink is a dangerous tool to use here.\n\nIf we unpack all objects we have not all these problems, but the main.git is in this case quite big. Of course it may be better to have a big repository than 20 copies of the packed repository. \nBut I suppose that that there is some way to share an object repository in a safe way, even if it includes some packs.\n\nI am quite new to git core and internals, so I may have missed the point.\n\nMarc\n"},{"id":"80301","messageId":"20080619090129.GJ11793@spearce.org","threadId":"14015","inReplyTo":"20080618195726.GA20440@kernoel.dyndns.org","subject":"Re: sharing object packs","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2008-06-19T09:01:30Z","receivedAt":"2008-06-19T09:01:30Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"marc.zonzon+git@gmail.com wrote:\n> I have a big bare repository 'main.git' and many small git repositories sub1, sub2, ... subn.\n>\n> All repositories lie in the same file file system, and each subx\n> repository track and fetch main.git in a remote branch.\n>\n> I would like to avoid duplicating main.git objects\n...\n> - Using an objects/info/alternates with the path of main.git object repository.\n> It work well too, but I import objects from main.git inside subx,\n> and they don't have the same life time than those in main.git. So\n> they can, disapear during a git-prune-packed or gc. (The same\n> problem we have with: git clone --share)\n\nThis is the approach you want to use.  The risk is that you do\nnot allow objects to be added to main.git to later be deleted from\nmain.git.  This means main.git cannot rewind/reset/delete a branch.\n\nIf that is not acceptable perhaps you could instead create 3 tiers:\n\n\tmain.git ---\n\t            \\\n\t            shared.git\n\t            /\n\tsubx.git ---  \n\nHave main.git and subx.git both use shared.git as an alternate\n(place path of shared.git/objects in their objects/info/alternates).\nYou can still allow subx.git to fetch main.git.\n\nPush only stable commits to shared.git that will never be\nrewind/reset/deleted.  Once something enters shared.git it should\nnever be deleted.  This way shared objects will not be removed by\ngit-prune or git-gc.  Every so often push newer stable branches from\nmain.git to shared.git, once they cannot be rewind/reset/deleted.\n\nRepack main.git and subx.git using `git gc` as that includes the\n-l flag to `git repack`.  Any objects which are now available from\nshared.git will not be included in main.git or subx.git, so their\nusage will shrink after shared.git is updated.\n\n\nIf you also configure gc.packrefs to never in shared.git and\nsymlink shared.git/refs into main.git/refs/shared and also into\nsubx.git/refs/shared and do this configuration on both server and\nclient systems you can have everyone transfer only the minimal\nobjects necessary.\n\nI use basically that arrangement at day-job to avoid 7 copies of\nroughly 150 MB of shared history across 8 repositories.  This\nreduces the amount of data the OS needs to store in buffer cache\nby nearly 1050 MB and thus makes things run rather quickly.  (Yes,\nmany operations hit all 8 repositories in rather rapid succession,\nits a submodule sort of arrangement.)\n\n-- \nShawn.\n"},{"id":"80332","messageId":"71295b5a0806191001ra1d4df1wb6fc709dac35f617@mail.gmail.com","threadId":"14015","inReplyTo":"20080619090129.GJ11793@spearce.org","subject":"Re: sharing object packs","fromName":"marc zonzon","fromEmail":"marc.zonzon+git@gmail.com","sentAt":"2008-06-19T17:01:39Z","receivedAt":"2008-06-19T17:01:39Z","isPatch":false,"sender":{"key":"marc.zonzon+git@gmail.com","avatar":null},"body":"Thank You Shawn\n\nOn Thu, Jun 19, 2008 at 11:01 AM, Shawn O. Pearce <spearce@spearce.org> wrote:\n> marc.zonzon+git@gmail.com wrote:\n>> I have a big bare repository 'main.git' and many small git repositories sub1, sub2, ... subn.\n>>\n>> All repositories lie in the same file file system, and each subx\n>> repository track and fetch main.git in a remote branch.\n>>\n>> I would like to avoid duplicating main.git objects\n> ...\n>> - Using an objects/info/alternates with the path of main.git object repository.\n>> It work well too, but I import objects from main.git inside subx,\n>> and they don't have the same life time than those in main.git. So\n>> they can, disapear during a git-prune-packed or gc. (The same\n>> problem we have with: git clone --share)\n>\n> This is the approach you want to use.  The risk is that you do\n> not allow objects to be added to main.git to later be deleted from\n> main.git.  This means main.git cannot rewind/reset/delete a branch.\n>\n> If that is not acceptable perhaps you could instead create 3 tiers:\n>\n>        main.git ---\n>                    \\\n>                    shared.git\n>                    /\n>        subx.git ---\n>\n> Have main.git and subx.git both use shared.git as an alternate\n> (place path of shared.git/objects in their objects/info/alternates).\n> You can still allow subx.git to fetch main.git.\n>\n\nYour solution of 3 tiers seems to solve the problems I met when trying\nto take main.git as alternates.\n\nBut I feel we can even make it more secure than what you explain:\n\n> Push only stable commits to shared.git that will never be\n> rewind/reset/deleted.  Once something enters shared.git it should\n> never be deleted.  This way shared objects will not be removed by\n> git-prune or git-gc.  Every so often push newer stable branches from\n> main.git to shared.git, once they cannot be rewind/reset/deleted.\n>\n\nMy option is to fetch from shared not only the branches of main, but\nall the branches of all the subx.\n\nSo shared.git host all the objects of the sum of main and all subx.\n\nThen there is any problem to reset, delete, or rewind a\nbranch in main, even if you fetch the resetted branch from shared (a\nnon fast-forward fetch), The objects of the deleted branch are either\nnot in any sub directory, and nothing is lost when they are\npruned, or they have been imported in some branch and they will be\nkept, since there is a reference to them.\n\n\n> Repack main.git and subx.git using `git gc` as that includes the\n> -l flag to `git repack`.  Any objects which are now available from\n> shared.git will not be included in main.git or subx.git, so their\n> usage will shrink after shared.git is updated.\n>\n\nYes I tested that, with git gc, I had no immediate shrinking, I suppose\nwe have to wait for gc.pruneExpire to see the result.\n\nBut:\n * fetching the remotes of shared.git,\n * packing shared.git,\n * packing and pruning (with git prune) the directories subx and\n main.git\n\nreduces immediately the object store of the subx to nearly nothing.\n\n> If you also configure gc.packrefs to never in shared.git and\n> symlink shared.git/refs into main.git/refs/shared and also into\n> subx.git/refs/shared and do this configuration on both server and\n> client systems you can have everyone transfer only the minimal\n> objects necessary.\n\nThank you also for this setting, my level of knowledge of git transfer\nmechanism is yet too low for understanding it without further\nexplanation/reading. If you can give some pointers they are welcome.\n\n\nThis solution seems great to implement some kind of submodules.\n\nI suppose we could also use this 3 tiers solution to do a more clever\nclone --share by the following scheme:\n\n# mkdir shared\n# cd shared\n# git init\n# cd ..\n# git clone --no-hardlinks --bare shared shared.git\n# rm -rf shared\n# cd shared.git\n# git remote add -f repo ../repo\n * [new branch]      master     -> repo/master\n# cd ..\n# git clone repo repo_copy\n# echo $PWD/shared.git/objects >> repo/.git/objects/alternates\n# cd shared.git\n# git remote add -f repo_copy ../repo_copy\n * [new branch]      master     -> repo_copy/master\n# cd ..\n# echo $PWD/shared.git/objects >> repo_copy/.git/objects/alternates\n\nthen the sequence of repack, gc, prune outlined above.\n\nBut I have not yet the experience in git, to allow me to foresee the\nconsequences of these settings.\n\n\nAll criticisms are welcome.\n\nMarc\n"}]}