threads / discuss / 27079

Re: is hosting a read-mostly git repo on a distributed file system practical?

Subject: Re: is hosting a read-mostly git repo on a distributed file system practical?

## tl;dr

2 messages between Apr 13, 2011 and Apr 13, 2011.

replies: 1people: 2as markdown or json

George Spelvin· Apr 13, 2011, 03:47 UTC · lore
> All clients, including the client that occasionally updates the
> read-mostly repo would be mounting the DFS as a local file system. My
> environment is one where DFS is easy, but establishing a shared server
> is more complicated (ie. bureaucratic).
> I guess I am prepared to put up with a slow initial clone (my developer
> pool will be relatively stable and pulling from a peer via git: or ssh:
> will usually be acceptable for this occasional need).
Show 5 quoted lines
> What I am most interested in is the incremental performance. Can my
> integrator, who occasionally updates the shared repo, avoid automatically
> repacking it (and hence taking the whole of repo latency hit) and can
> my developers who are pulling the updates do so reliably without a whole
> of repo scan?
I think the answers are yes, but I have to make a vouple of things clear:
* You can *definitely* control repack behaviour.  .keep files are the
  simplest way to prevent repacking.
* Are you talking about hosting only a "bare" repository, or one with
  the unpacked source tree as well?  If you try to run git commands on
  a large network-mounted source tree, things can get more than a bit
  sluggish; git recursively stats the whole tree fairly frequently.
  (There are ways to precent that, notably core.ignoreStat, but they
  make it less friendly.)
* You can clone from a repository mounted on the file system just as
  easily as you can from a network server.  So there's no need to set
  up a server if you find it onconvenient.
* Normally, the developers will clone from the integrator's repository
  before doing anything, so the source tree, and any changes they make,
  will be local.
* A local clone will try to hard link to the object directory.  I think
  it will copy them if it fails, or you can force that with "git clone
  --no-hardlinks".  For a more space-saving version, try "git clone
  -s", which will make a sort of soft link to the upstream repository.
  It's a git concept, so repacking upstream won't do any harm, but you
  Must Not delete objects from the upstream repository or you'll create
  dangling references in the downstream.
* If using the objects on the DFS mount turns out to be slow, you can
  just do the initial clone with --no-hardlinks.  Then the developers'
  day-to-day work is all local.

Indeed, you could easily do everything via DFS. Give everyone a personal "public" repo to push to, which is read-only to everyone else, and let the integrator pull from those.

> I understand that avoiding repacking for an extended period brings its
> own problems, so I guess I could live with a local repack followed by
> an rsync transfer to re-initial the shared remote, if this was
> warranted.

Normally, you do a generational garbage collection thing. You repack the current work frequently (which is fast to do, and to share, because it's small), and the larger, slower, older packs less frequently.

Anyway, I hope this helps!
Jon Seymour· Apr 13, 2011, 04:57 UTC · re: George Spelvin · lore
On Wed, Apr 13, 2011 at 1:47 PM, George Spelvin <linux@horizon.com> wrote:
> I think the answers are yes, but I have to make a vouple of things clear:
> * You can *definitely* control repack behaviour.  .keep files are the
>  simplest way to prevent repacking.
Good.
Show 6 quoted lines
> * Are you talking about hosting only a "bare" repository, or one with
>  the unpacked source tree as well?  If you try to run git commands on
>  a large network-mounted source tree, things can get more than a bit
>  sluggish; git recursively stats the whole tree fairly frequently.
>  (There are ways to precent that, notably core.ignoreStat, but they
>  make it less friendly.)
Bare. Developers use local disk for local repos and working tree.
> * You can clone from a repository mounted on the file system just as
>  easily as you can from a network server.  So there's no need to set
>  up a server if you find it onconvenient.

Are there advantages to using rsync for the initial clone? Will I get better restartability in the case that the network is less than 100% reliable?

I do remember trying to use a DFS-file system in the past, before I understood pack management properly and I seem to recall issues with network reliability.

> Indeed, you could easily do everything via DFS.  Give everyone a personal
> "public" repo to push to, which is read-only to everyone else, and let
> the integrator pull from those.
>

I'll probably use ssh-secured peer to peer for publishing purposes. The main thing I want the DFS-hosted repo for is to provide a single, always up, go-to point for the shared tag set.

>
> Anyway, I hope this helps!
>
Yep, thank you.
jon.

← back to recent threads