threads / discuss / 2652

Question about handling of heterogeneous repositories

Subject: Question about handling of heterogeneous repositories

## tl;dr

11 messages between Nov 22, 2005 and Dec 1, 2005.

replies: 10people: 5as markdown or json

Alex Riesen· Nov 22, 2005, 16:50 UTC · lore
Hi,

it is sometimes the case that a project consists of parts which are unrelated to each other, and only thing in common between them is that they all are used in that particular project. For example a program uses some library and the developer(s) of that program would like to have the source of that library somewhere close. Well, for this simple example one could just use two repositories, laid close to each other in a directory, like project/lib and project/prog. Now, if I make the example a bit more complex and say, that the developers of the program are the developers in that project and change everything under project/ directory, including project/library/. They are also good people and ready to give the changes to the library upstream.

How do they achieve that, without sending project/ and project/program/?

For everyone who have an experience with ClearCase or Perforce (I'm sorry for mentioning it) it is what the "mappings" are often used for: a project is build together from different parts, which can be worked on separately.

I'm trying to introduce git at work, but have to prepare myself for possible questions first, and this is one of them :)

Greetings, Alex Riesen

Andreas Ericsson· Nov 22, 2005, 19:40 UTC · re: Alex Riesen · lore

Re: Question about handling of heterogeneous repositories

Alex Riesen wrote:
Show 25 quoted lines
> Hi,
> 
> it is sometimes the case that a project consists of parts which are
> unrelated to each other, and only thing in common between them is that
> they all are used in that particular project. For example a program
> uses some library and the developer(s) of that program would like to
> have the source of that library somewhere close. Well, for this simple
> example one could just use two repositories, laid close to each other
> in a directory, like project/lib and project/prog.
> Now, if I make the example a bit more complex and say, that the
> developers of the program are the developers in that project and
> change everything under project/ directory, including
> project/library/. They are also good people and ready to give the
> changes to the library upstream.
> 
> How do they achieve that, without sending project/ and project/program/?
> 
> For everyone who have an experience with ClearCase or Perforce (I'm
> sorry for mentioning it) it is what the "mappings" are often used for:
> a project is build together from different parts, which can be worked
> on separately.
> 
> I'm trying to introduce git at work, but have to prepare myself for
> possible questions first, and this is one of them :)
> 
We do like this;

core core/gui core/lib

$ cat .gitignore gui lib

This is also nice because it lets the gui maintainers have the gui as the root with the core and lib parts as subdirectories. Everyone has their own responsibility checked out at top-level with other pieces below it. It's easy enough to script a pull of all repos so everyone's up to sync and everybody's happy.

It would certainly be nicer to have git ignore directories that have the ".git" directory (so long as it's not the top of the repo, that is), but I haven't had the energy to fix that when there's already a solution that's simple enough and quite adequate.

-- 
Andreas Ericsson                   andreas.ericsson@op5.se
OP5 AB                             www.op5.se
Tel: +46 8-230225                  Fax: +46 8-230231
Alex Riesen· Nov 22, 2005, 23:22 UTC · re: Andreas Ericsson · lore

Re: Question about handling of heterogeneous repositories

Andreas Ericsson, Tue, Nov 22, 2005 20:40:50 +0100:
Show 39 quoted lines
> >it is sometimes the case that a project consists of parts which are
> >unrelated to each other, and only thing in common between them is that
> >they all are used in that particular project. For example a program
> >uses some library and the developer(s) of that program would like to
> >have the source of that library somewhere close. Well, for this simple
> >example one could just use two repositories, laid close to each other
> >in a directory, like project/lib and project/prog.
> >Now, if I make the example a bit more complex and say, that the
> >developers of the program are the developers in that project and
> >change everything under project/ directory, including
> >project/library/. They are also good people and ready to give the
> >changes to the library upstream.
> >
> >How do they achieve that, without sending project/ and project/program/?
> >
> >For everyone who have an experience with ClearCase or Perforce (I'm
> >sorry for mentioning it) it is what the "mappings" are often used for:
> >a project is build together from different parts, which can be worked
> >on separately.
> >
> >I'm trying to introduce git at work, but have to prepare myself for
> >possible questions first, and this is one of them :)
> >
> 
> We do like this;
> 
> core
> core/gui
> core/lib
> 
> $ cat .gitignore
> gui
> lib
> 
> This is also nice because it lets the gui maintainers have the gui as 
> the root with the core and lib parts as subdirectories. Everyone has 
> their own responsibility checked out at top-level with other pieces 
> below it. It's easy enough to script a pull of all repos so everyone's 
> up to sync and everybody's happy.

And than, do you have to announce a special procedure everyone's is to execute after a clone so all subrepos are cloned? How do you handle common configs and clone options in subrepos? It also would be nice to have branches visible not only in in subrepos but in top-repo as well, and the other way around.

I'm also wondering, what implications core/lib/.git/{config,remotes/,refs/} under control in core/ can have?..

> It would certainly be nicer to have git ignore directories that have the 
> ".git" directory (so long as it's not the top of the repo, that is), but 
> I haven't had the energy to fix that when there's already a solution 
> that's simple enough and quite adequate.

BTW, will something like "*/.git/*" in info/exclude work? IOW, does * match a "/"?

Petr Baudis· Nov 27, 2005, 13:11 UTC · re: Alex Riesen · lore

Re: Question about handling of heterogeneous repositories

Dear diary, on Wed, Nov 23, 2005 at 12:22:28AM CET, I got a letter where Alex Riesen <raa.lkml@gmail.com> said that...

Show 23 quoted lines
> Andreas Ericsson, Tue, Nov 22, 2005 20:40:50 +0100:
> > >it is sometimes the case that a project consists of parts which are
> > >unrelated to each other, and only thing in common between them is that
> > >they all are used in that particular project. For example a program
> > >uses some library and the developer(s) of that program would like to
> > >have the source of that library somewhere close. Well, for this simple
> > >example one could just use two repositories, laid close to each other
> > >in a directory, like project/lib and project/prog.
> > >Now, if I make the example a bit more complex and say, that the
> > >developers of the program are the developers in that project and
> > >change everything under project/ directory, including
> > >project/library/. They are also good people and ready to give the
> > >changes to the library upstream.
> > >
> > >How do they achieve that, without sending project/ and project/program/?
> > >
> > >For everyone who have an experience with ClearCase or Perforce (I'm
> > >sorry for mentioning it) it is what the "mappings" are often used for:
> > >a project is build together from different parts, which can be worked
> > >on separately.
> > >
> > >I'm trying to introduce git at work, but have to prepare myself for
> > >possible questions first, and this is one of them :)

This is something e.g. Cogito wants to support, but does not yet. Patches welcome.

Show 7 quoted lines
> > It would certainly be nicer to have git ignore directories that have the 
> > ".git" directory (so long as it's not the top of the repo, that is), but 
> > I haven't had the energy to fix that when there's already a solution 
> > that's simple enough and quite adequate.
> 
> BTW, will something like "*/.git/*" in info/exclude work? IOW, does *
> match a "/"?

Nope, but try just '.git' - in case it is not a pathname but just a filename (or dirname, for that matter), it will recursively apply to all the subtrees.

-- 
				Petr "Pasky" Baudis
Stuff: http://pasky.or.cz/
VI has two modes: the one in which it beeps and the one in which
it doesn't.
Alex Riesen· Nov 29, 2005, 20:47 UTC · re: Petr Baudis · lore

Re: Question about handling of heterogeneous repositories

Petr Baudis, Sun, Nov 27, 2005 14:11:47 +0100:
Show 10 quoted lines
> > > >For everyone who have an experience with ClearCase or Perforce (I'm
> > > >sorry for mentioning it) it is what the "mappings" are often used for:
> > > >a project is build together from different parts, which can be worked
> > > >on separately.
> > > >
> > > >I'm trying to introduce git at work, but have to prepare myself for
> > > >possible questions first, and this is one of them :)
> 
> This is something e.g. Cogito wants to support, but does not yet.
> Patches welcome.

I wouldn't know what to patch, having no clear picture of the approach myself, and especially when I don't feel safe using the solution. For example, how do you go about moving/renaming files between subrepos? Rename detection will not work, which will be unexpected...

BTW, how does git-mv behave for out-of-tree renaming? How about inter-repo renaming (remove+add)?

Show 11 quoted lines
> > > It would certainly be nicer to have git ignore directories that have the 
> > > ".git" directory (so long as it's not the top of the repo, that is), but 
> > > I haven't had the energy to fix that when there's already a solution 
> > > that's simple enough and quite adequate.
> > 
> > BTW, will something like "*/.git/*" in info/exclude work? IOW, does *
> > match a "/"?
> 
> Nope, but try just '.git' - in case it is not a pathname but just a
> filename (or dirname, for that matter), it will recursively apply to all
> the subtrees.

well, it ignored the ".git"s in the subdirs, not _the_ subdirectories. I think that can be helped by putting the directories themselves into .gitignore lists.

Alex Riesen· Nov 30, 2005, 13:15 UTC · lore

Re: Question about handling of heterogeneous repositories

On 11/30/05, Josef Weidendorfer <Josef.Weidendorfer@gmx.de> wrote:
Show 9 quoted lines
> > BTW, how does git-mv behave for out-of-tree renaming? How about
> > inter-repo renaming (remove+add)?
>
> git-mv will fail. The source has to appear in git-ls-files, and the
> target simply is given to "git-update-index -add", which will return an
> 'outside repository' error. Hmmm... git-mv should check this itself.
>
> To detect inter-repository renaming, one has to see both repositories.
> Currently, git/cogito etc. commands only work with one repository only.
This case is somewhat special: one is _guaranteed_ to see both repositories.
PS: Im'restoring git@vger.kernel.org in cc-list, the discussion could be useful
Johannes Schindelin· Nov 30, 2005, 13:58 UTC · re: Alex Riesen · lore

Re: Question about handling of heterogeneous repositories

Hi,
On Wed, 30 Nov 2005, Alex Riesen wrote:
Show 8 quoted lines
> On 11/30/05, Josef Weidendorfer <Josef.Weidendorfer@gmx.de> wrote:
> >
> > To detect inter-repository renaming, one has to see both repositories. 
> > Currently, git/cogito etc. commands only work with one repository 
> > only.
> 
> This case is somewhat special: one is _guaranteed_ to see both 
> repositories.

It is even more so: it is guaranteed that git cannot reconstruct this as a move operation.

Hth, Dscho

Josef Weidendorfer· Nov 30, 2005, 14:40 UTC · re: Johannes Schindelin · lore

Re: Question about handling of heterogeneous repositories

On Wednesday 30 November 2005 14:58, Johannes Schindelin wrote:
Show 15 quoted lines
> Hi,
> 
> On Wed, 30 Nov 2005, Alex Riesen wrote:
> 
> > On 11/30/05, Josef Weidendorfer <Josef.Weidendorfer@gmx.de> wrote:
> > >
> > > To detect inter-repository renaming, one has to see both repositories. 
> > > Currently, git/cogito etc. commands only work with one repository 
> > > only.
> > 
> > This case is somewhat special: one is _guaranteed_ to see both 
> > repositories.
> 
> It is even more so: it is guaranteed that git cannot reconstruct this 
> as a move operation.

Say it in another way: because currently, git commands only work in the scope of one repository, there is no need to want to detect an inter-repository rename.

Git has no explicit support for renaming, but can do it. Why should it be a problem to write a tool which is able to detect inter-repository renames? Still, such a search probably would be very expensive.

git-mv could support this. Additional to deleting from one repository and adding to the new, it could generate a remark for the commit message, like:

 Added file xxx is from remote repo '/path/myrepo' with name 'yyy'
 (file SHA1: 123445... from remote commit SHA1: 345667...)

This info could be interesting both for humans and a history browsing tool. The tool still would have to know where to look for repo '/path/myrepo'. It could also work for moving subdirectories (ie. trees instead of files).

Josef
Alex Riesen· Nov 30, 2005, 15:01 UTC · re: Josef Weidendorfer · lore

Re: Question about handling of heterogeneous repositories

On 11/30/05, Josef Weidendorfer <Josef.Weidendorfer@gmx.de> wrote:
Show 6 quoted lines
> git-mv could support this. Additional to deleting from
> one repository and adding to the new, it could generate a remark for
> the commit message, like:
>
>  Added file xxx is from remote repo '/path/myrepo' with name 'yyy'
>  (file SHA1: 123445... from remote commit SHA1: 345667...)
That's sounds very tempting to special-case for nested repos :)
Maybe use .git/objects/info/alternates in top-level repo? It may
have side-effects of the new objects ending up in top-level repo
if commited from top-level directory. Even then, the tools can
be tought to stop looking for GIT_DIR in the directory closest
to the file being commited:
  $ pwd
  .../top
  $ git-commit sub1/dir/file
  ... uses repo in .../top/sub1 instead of $(pwd)/../top/.git

Looks very messy, with a potential for lots of corner cases and not that tempting anymore :(

> This info could be interesting both for humans and a history browsing tool.
> The tool still would have to know where to look for repo '/path/myrepo'.
> It could also work for moving subdirectories (ie. trees instead of files).

Well, these are just lots of files. I can't really think of case were I'd need to know about moving a directory without knowing what happened to the files.

>
> Josef
>
Josef Weidendorfer· Nov 30, 2005, 16:49 UTC · re: Alex Riesen · lore

Re: Question about handling of heterogeneous repositories

On Wednesday 30 November 2005 16:01, Alex Riesen wrote:
Show 9 quoted lines
> On 11/30/05, Josef Weidendorfer <Josef.Weidendorfer@gmx.de> wrote:
> > git-mv could support this. Additional to deleting from
> > one repository and adding to the new, it could generate a remark for
> > the commit message, like:
> >
> >  Added file xxx is from remote repo '/path/myrepo' with name 'yyy'
> >  (file SHA1: 123445... from remote commit SHA1: 345667...)
> 
> That's sounds very tempting to special-case for nested repos :)

If the repo path is an URL, this works quite general. I tried to avoid adding local head names to this remark...

> Maybe use .git/objects/info/alternates in top-level repo?

Or better the .git/remotes namespace... Still, this is misusing repo-local config for multiple repos. Seems quite confusing for me.

Show 12 quoted lines
> It may 
> have side-effects of the new objects ending up in top-level repo
> if commited from top-level directory. Even then, the tools can
> be tought to stop looking for GIT_DIR in the directory closest
> to the file being commited:
>   $ pwd
>   .../top
>   $ git-commit sub1/dir/file
>   ... uses repo in .../top/sub1 instead of $(pwd)/../top/.git
> 
> Looks very messy, with a potential for lots of corner cases and
> not that tempting anymore :(

git-mv would have to do exactly something like this to detect inter-repo renames at all ;-) But in general, this is really messy.

See my suggestion in a previous thread with similar topic
 http://www.gelato.unsw.edu.au/archives/git/0511/12028.html
Josef
Alex Riesen· Dec 1, 2005, 08:16 UTC · re: Josef Weidendorfer · lore

Re: Question about handling of heterogeneous repositories

On 11/30/05, Josef Weidendorfer <Josef.Weidendorfer@gmx.de> wrote:
> git-mv would have to do exactly something like this to detect
> inter-repo renames at all ;-)
> But in general, this is really messy.
That's is what I'm most afraid of.
> See my suggestion in a previous thread with similar topic
>  http://www.gelato.unsw.edu.au/archives/git/0511/12028.html

Oh, thanks! Pity, I missed that discussion, Pasky's subprojects is exactly what I am looking for.

← back to recent threads