how to reduce disk usage for large .git dirs?

How can I reduce the disk usage for multiple copies of the same repo?

Up to now I just made copies like this, but since .git alone is already 2GB it becomes expensive:

 # git clone git://host/repo.git repo-master
 # cp -a repo-master repo-branchA
 # cd repo-branchA
 # git checkout -b branchA origin/branchA
 # cd -
 # cp -a repo-master repo-branchB
 # cd repo-branchB
 # git checkout -b branchB origin/branchB
 # cd -
 # cp -a repo-master repo-branchB-feature
 # cd repo-branchB-feature
 # git checkout -b branchB-feature origin/branchB
 # cd -

Since each .git is almost identical I wonder if there is a reliable way to "share" it. The "git clone" man page mentions --shared as a dangerous way to do things. It does not give an advice how to manage such cloned trees.

So how can I reduce the disk usage needed for the four .git dirs above? I looked around in the docs that came with my git-2.1.3 package, but found nothing that answers my question. Maybe we can workout something and add it to one of the existing docs.

Thanks!

Olaf

On Thu, Nov 13, 2014 at 12:14:44PM +0100, Olaf Hering wrote:

You can use --local och --shared. As you say --shared can be dangerous. If you don't understand the man page enough to know how you should manage your clones you should probably not use it.

--local seems to be what you're looking for.

However as a side note I'm curious about what your use case is. Why do you need this many repos?

Your setup looks familiar to me for a subversion user switching to git and trying to use git as subversion. The common usecase is not to have multiple worktrees but to do a checkout to the worktree you need to work on. This is possible with git since it's very fast and I recommend you to try to use one worktree.

On Thu, Nov 13, Fredrik Gustafsson wrote:

Perhaps its more a lack of doc how to use the result.

I will try this.

Because I do work in each copy, poke around, or do commits and push them.

Switching branches will invalidate timestamps, causing a full rebuild.

Olaf

W dniu 2014-11-13 13:03, Olaf Hering pisze:
 > On Thu, Nov 13, Fredrik Gustafsson wrote:
[...]

Wouldn't a better way of solving "full rebuild" issue be to use ccache or similar solution?

Anyway, switching branches invalidates timestamps only on those files that change between branches -- it is to avoid unnecessary rebuilds.

You can always clone with --reference, and use alternates (alternate object store). Just don't delete objects in repository that other repositories borrow from; GitHub uses refs/borrowers/ namespace for that, IIRC.

HTH

On Thu, 2014-11-13 at 12:14 +0100, Olaf Hering wrote:

But you're not using clone you are using cp .

The clone man page also says this:-
      --local, -l
           When the repository to clone from is on a local machine, this flag bypasses the normal "Git aware" transport
           mechanism and clones the repository by making a copy of HEAD and everything under objects and refs directories. The
           files under .git/objects/ directory are hardlinked to save space when possible.

           If the repository is specified as a local path (e.g., /path/to/repo), this is the default, and --local is essentially
           a no-op. If the repository is specified as a URL, then this flag is ignored (and we never use the local
           optimizations). Specifying --no-local will override the default when /path/to/repo is given, using the regular Git
           transport instead.

Note the first sentence of the second paragraph.
 eg:
 # git clone git://host/repo.git repo-master
 # git clone repo-master repo-branchA
 # cd repo-branchA
 # git checkout -b branchA origin/branchA
 # cd -
 # git clone repo-master repo-branchB
 # cd repo-branchB
 # git checkout -b branchB origin/branchB
 # cd -
 # git clone repo-master repo-branchB-feature
 # cd repo-branchB-feature
 # git checkout -b branchB-feature origin/branchB
 # cd -

Should work better for you. And there is probably a way to do it less commands too.

On Thu, Nov 13, Roger Gammans wrote:

It fails right here because in this dir only "master" exists, but branchA is expected.

So far the sequence of commands is:

# git clone git://host/repo.git repo-master # cd repo-master # git checkout -b branchA origin/branchA # git checkout -b branchB origin/branchB # cd - # git clone -l -b branchA repo-master repo-branchA # git clone -l -b branchB repo-master repo-branchB

Next step will be: # $do_work ; git commit -avs ; git push

Will that work as expected? Will find out after lunch..

Olaf

On Thu, Nov 13, 2014 at 6:14 PM, Olaf Hering <olaf@aepfle.de> wrote:

If you know what you are doing, you can try git-new-workdir in contrib/workdir. A safe and reliable version of that is being worked on, hopefully it'll be released in 2.3.0.

On Thu, Nov 13, Olaf Hering wrote:

While playing around with this I made some notes, this is the result:

Manage multiple branches as separate copies

To preserve disk space for each clone, use a master copy of the reop and do local clones from such copy of a remote repository.

First clone the remote repository as usual. Then create a local branch for each remote branch that is supposed to be worked on: # git clone git://host/repo.git repo-master # cd repo-master # git checkout -b branchA origin/branchA # git checkout -b branchB origin/branchB # cd -

Now clone each work branch into its own directory. The work dir references the master repo. All changes come from and go into this repo, instead of the remote repo. # git clone -l -b branchA repo-master repo-branchA # git clone -l -b branchB repo-master repo-branchB

To make changs in a work dir, commit as usual. The changes will be pushed from the work copy into the local master repo. Its required to have some other branch than branchA active in repo-master, or push from work copy to repo-master will fail. # cd repo-master # git checkout master # cd - # cd repo-branchA # git commit -avs # git push origin branchA # cd -

To publish the outstanding changes its required to do this from the master repo. First checkout the work branch, then pull the local changes and finally push them to the remote repo. # cd repo-master # git checkout branchA # git pull # git push origin branchA # cd -

To receive changes from the remote repo its required to do this from the master repo. First checkout the work branch, then pull the outstanding remote changes into the local branch. And finally pull them into the work dir. # cd repo-master # git fetch --all (optional) # git checkout branchB # git pull # cd - # cd repo-branchB # git pull # cd -

# vim: set tw=72 et

Thanks for sharing your notes! A few comments:

On Thu, Nov 13, 2014 at 04:44:57PM +0100, Olaf Hering wrote:

That's one of the reason it's not recommended to push into a non-bare repository. You should clone your repo-master with the --bare option to avoid having a work dir there.

It's not. You could just add your remote repository as a remote to each of your clones of your master repo and push directly from them. It would be much simplier and it would allow you to directly fetch changes from your remote into your branches as well.

(however, I'm not sure but I think, that this will slowly increase the difference between your repositories when you develop. So that they won't change any new data since to local clone was made).

On Thu, Nov 13, 2014 at 5:03 PM, Fredrik Gustafsson <iveqy@iveqy.com> wrote:

Can you not do this much simpler with --reference? Like this:

  $ git clone --bare git://host/repo.git repo-master
  $ git clone -b branchA --reference repo-master git://host/repo.git
repo-branchA
  $ git clone -b branchB --reference repo-master git://host/repo.git
repo-branchB

All three repos now push/pull directly to/from git://host/repo.git, but repo-branchA and repo-branchB reference objects from within the bare repo-master. You have to make use to never delete objects from repo-master (if those objects happen to be referenced from repo-branchA|B). If you want to prevent the repos growing in size, you must devise a way to add new objects into repo-master before repo-branchA|B. (e.g. a nightly cron-job in repo-master that fetches from origin), so that when repo-branchA|B pulls, they will find most objects are already present in repo-master.

...Johan

On Thu, Nov 13, 2014 at 05:08:19PM +0100, Johan Herland wrote:

I think the "never delete" part is why we usually warn people off of using alternates. I think at the least you would have to "git config gc.auto 0" in the bare repository (otherwise your nightly fetches risk pruning). Of course you'd probably want to repack eventually for performance reasons. So maybe setting gc.pruneExpire is a better option (to something like "20.years.ago").

You can also fetch from the children into repo-master periodically. Like:

  cd repo-master &&
  for i in branchA branchB; do
    git fetch ../$i +refs/*:refs/remotes/$i/*
  done

after which it is actually safe to run "git gc" in the master (assuming there isn't simultaneous activity in the children). This is how we manage fork networks on GitHub (we take in objects to individual forks via push, and then migrate them to the master repo via fetch).

-Peff

On Thu, Nov 13, Fredrik Gustafsson wrote:

So my repo-master is now "bare". I pushed from repo-branchA into repo-master and see my commits in both repos. But pushing from repo-master to the remote fails because repo-master does not have outstanding remote commits. However, git fetch doesnt do anything:

Fetching origin
From host:/remote/dir
* branch            HEAD       -> FETCH_HEAD

Obviously I miss something. The man page of git clone or fetch does not mention how "bare" is supposed to be handled.

Olaf

On Fri, Nov 14, 2014 at 11:14:27AM +0100, Olaf Hering wrote:

Are you mixing up your branches? So that you're updating one branch in your master repo but trying to push an other branch to your remote repo?

On Fri, Nov 14, Fredrik Gustafsson wrote:

I dont think so. I have branchA in repo-branchA, and a 'git push origin branchA' puts it into repo-master. How is a bare repo supposed to be updated? Is a simple 'git fetch --all' supposed to work?

Olaf

On Fri, Nov 14, Olaf Hering wrote:

Is there s a slim chance that I have to fetch in repo-master before doing a git push in repo-branchA? git remote show origin shows that some branches are out of date.

Olaf

On Thu, Nov 13, Fredrik Gustafsson wrote:

Even if I do a fresh clone with --bare, the result can not be updated anymore with git fetch. What I'm doing wrong?

Olaf

Olaf Hering <olaf@aepfle.de> writes:

A --bare clone has no connection to its origin (there are no remotes). You want a --mirror.

Andreas.

On Fri, Nov 14, Andreas Schwab wrote:

Using --mirror for this purpose is dangerous because it will most likely overwrite changes on the remote side. Fortunately I used 'git push --dry-run origin' and the output was like:

To git://host/repo.git
 + abbrev1..abbrev2 branchA -> branchA (forced update)

Before that I pushed already to the remote repo without realizing where the push goes to.

Looks like using the --bare option for cloning the master and then doing something like "git clone --origin local_bare -b branchA --reference repo-master git://host/repo.git repo-branchA" will work better.

Olaf