threads / discuss / 32543

Moving (renaming) submodules, recipe/script

Subject: Moving (renaming) submodules, recipe/script

## tl;dr

11 messages between Jan 7, 2013 and Jan 8, 2013.

replies: 10people: 4as markdown or json

W. Trevor King· Jan 7, 2013, 00:36 UTC · lore

Today I had to move my first submodule, and I discovered that Git's support for this is pretty limited. There have been a few patch series attempting to address this [1,2], but none of them seems to have pushed through into master (although I can't put my finger on a reason for why). There are also some SO postings discussing this [3,4]. It would be nice if `git mv` worked out of the box on submodules. Failing that, there could be a `git submodule mv` command that casts the appropriate spell. Failing that, there could be a recipe in Documentation/git-submodule.txt. Here's the best I could come up with for a `git-submodule-mv.sh`:

  #!/bin/sh
  # usage: git-submodule-mv.sh OLD NEW
  OLD=$(realpath --relative-to . "$1")
  NEW=$(realpath --relative-to . "$2")
  SHA=$(git ls-files -s "$OLD" | sed 's|^[0-9]* \([0-9a-f]*\) .*|\1|')
  NAME=$(git config -f .gitmodules --get-regexp 'submodule\..*\.path' "$OLD" |
    sed -e 's|^submodule.||' -e "s|.path $OLD\$||")
  GITDIR=$(realpath --relative-to "$NEW" .git/modules/"$NAME")
  git config -f .gitmodules submodule."$NAME".path "$NEW"
  git config -f .git/modules/"$NAME"/config core.worktree "../../../$NEW"
  git rm --cached "$OLD"
  mv "$OLD" "$NEW"
  echo "gitdir: $GITDIR" > "$NEW/.git"
  git update-index --add --cacheinfo 160000 "$SHA" "$NEW"

This only works from the repository root directory, and I'm sure makes a number of poor assumptions (e.g. old-style submodules that don't use `gitdir` links are not supported). It does work for some simple test cases. The tricky parts (e.g. path -> name conversion) are already worked out more robustly git-submodule.sh, so adding a new cmd_mv shouldn't be very difficult.

Could something like this live somewhere in Git, or are we waiting for a more integrated solution?

Cheers, Trevor

[1]: http://thread.gmane.org/gmane.comp.version-control.git/88720 [2]: http://thread.gmane.org/gmane.comp.version-control.git/143250 [4]: http://stackoverflow.com/questions/4323558/moving-submodules-with-git [3]: http://stackoverflow.com/questions/4604486/how-do-i-move-an-existing-git-submodule-within-a-git-repository

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
Jonathan Nieder· Jan 7, 2013, 01:39 UTC · re: W. Trevor King · lore

Re: Moving (renaming) submodules, recipe/script

(just cc-ing Jens and Peter, who might be interested)
W. Trevor King wrote:
Show 43 quoted lines
> Today I had to move my first submodule, and I discovered that Git's
> support for this is pretty limited.  There have been a few patch
> series attempting to address this [1,2], but none of them seems to
> have pushed through into master (although I can't put my finger on a
> reason for why).  There are also some SO postings discussing this
> [3,4].  It would be nice if `git mv` worked out of the box on
> submodules.  Failing that, there could be a `git submodule mv` command
> that casts the appropriate spell.  Failing that, there could be a
> recipe in Documentation/git-submodule.txt.  Here's the best I could
> come up with for a `git-submodule-mv.sh`:
>
>   #!/bin/sh
>   # usage: git-submodule-mv.sh OLD NEW
>   OLD=$(realpath --relative-to . "$1")
>   NEW=$(realpath --relative-to . "$2")
>   SHA=$(git ls-files -s "$OLD" | sed 's|^[0-9]* \([0-9a-f]*\) .*|\1|')
>   NAME=$(git config -f .gitmodules --get-regexp 'submodule\..*\.path' "$OLD" |
>     sed -e 's|^submodule.||' -e "s|.path $OLD\$||")
>   GITDIR=$(realpath --relative-to "$NEW" .git/modules/"$NAME")
>   git config -f .gitmodules submodule."$NAME".path "$NEW"
>   git config -f .git/modules/"$NAME"/config core.worktree "../../../$NEW"
>   git rm --cached "$OLD"
>   mv "$OLD" "$NEW"
>   echo "gitdir: $GITDIR" > "$NEW/.git"
>   git update-index --add --cacheinfo 160000 "$SHA" "$NEW"
>
> This only works from the repository root directory, and I'm sure makes
> a number of poor assumptions (e.g. old-style submodules that don't use
> `gitdir` links are not supported).  It does work for some simple test
> cases.  The tricky parts (e.g. path -> name conversion) are already
> worked out more robustly git-submodule.sh, so adding a new cmd_mv
> shouldn't be very difficult.
>
> Could something like this live somewhere in Git, or are we waiting for
> a more integrated solution?
>
> Cheers,
> Trevor
>
> [1]: http://thread.gmane.org/gmane.comp.version-control.git/88720
> [2]: http://thread.gmane.org/gmane.comp.version-control.git/143250
> [4]: http://stackoverflow.com/questions/4323558/moving-submodules-with-git
> [3]: http://stackoverflow.com/questions/4604486/how-do-i-move-an-existing-git-submodule-within-a-git-repository
Jens Lehmann· Jan 7, 2013, 06:59 UTC · re: Jonathan Nieder · lore

Re: Moving (renaming) submodules, recipe/script

Am 07.01.2013 02:39, schrieb Jonathan Nieder:
> (just cc-ing Jens and Peter, who might be interested)
I´m currently working on teaching mv to move submodules and intend
to send those patches to the list after finishing submodule deinit.
Please see
  https://github.com/jlehmann/git-submod-enhancements/commits/mv-submodules
for the current state of this series.
Show 50 quoted lines
> W. Trevor King wrote:
> 
>> Today I had to move my first submodule, and I discovered that Git's
>> support for this is pretty limited.  There have been a few patch
>> series attempting to address this [1,2], but none of them seems to
>> have pushed through into master (although I can't put my finger on a
>> reason for why).  There are also some SO postings discussing this
>> [3,4].  It would be nice if `git mv` worked out of the box on
>> submodules.  Failing that, there could be a `git submodule mv` command
>> that casts the appropriate spell.  Failing that, there could be a
>> recipe in Documentation/git-submodule.txt.  Here's the best I could
>> come up with for a `git-submodule-mv.sh`:
>>
>>   #!/bin/sh
>>   # usage: git-submodule-mv.sh OLD NEW
>>   OLD=$(realpath --relative-to . "$1")
>>   NEW=$(realpath --relative-to . "$2")
>>   SHA=$(git ls-files -s "$OLD" | sed 's|^[0-9]* \([0-9a-f]*\) .*|\1|')
>>   NAME=$(git config -f .gitmodules --get-regexp 'submodule\..*\.path' "$OLD" |
>>     sed -e 's|^submodule.||' -e "s|.path $OLD\$||")
>>   GITDIR=$(realpath --relative-to "$NEW" .git/modules/"$NAME")
>>   git config -f .gitmodules submodule."$NAME".path "$NEW"
>>   git config -f .git/modules/"$NAME"/config core.worktree "../../../$NEW"
>>   git rm --cached "$OLD"
>>   mv "$OLD" "$NEW"
>>   echo "gitdir: $GITDIR" > "$NEW/.git"
>>   git update-index --add --cacheinfo 160000 "$SHA" "$NEW"
>>
>> This only works from the repository root directory, and I'm sure makes
>> a number of poor assumptions (e.g. old-style submodules that don't use
>> `gitdir` links are not supported).  It does work for some simple test
>> cases.  The tricky parts (e.g. path -> name conversion) are already
>> worked out more robustly git-submodule.sh, so adding a new cmd_mv
>> shouldn't be very difficult.
>>
>> Could something like this live somewhere in Git, or are we waiting for
>> a more integrated solution?
>>
>> Cheers,
>> Trevor
>>
>> [1]: http://thread.gmane.org/gmane.comp.version-control.git/88720
>> [2]: http://thread.gmane.org/gmane.comp.version-control.git/143250
>> [4]: http://stackoverflow.com/questions/4323558/moving-submodules-with-git
>> [3]: http://stackoverflow.com/questions/4604486/how-do-i-move-an-existing-git-submodule-within-a-git-repository
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
Junio C Hamano· Jan 7, 2013, 07:44 UTC · re: Jens Lehmann · lore

Re: Moving (renaming) submodules, recipe/script

Jens Lehmann <Jens.Lehmann@web.de> writes:
Show 5 quoted lines
> Am 07.01.2013 02:39, schrieb Jonathan Nieder:
>> (just cc-ing Jens and Peter, who might be interested)
>
> I´m currently working on teaching mv to move submodules and intend
> to send those patches to the list after finishing submodule deinit.
Thanks for a heads-up.

As a couple of recent "What's cooking" message has stated, I'll shortly kick jl/submodule-deinit topic out of 'next' back to 'pu', so please make an update a replacement, not an incremental. If I recall the discussion correctly, I think we agreed that deinit should clear the slate, which means the submodule working tree should be removed and made into an empty directory without ".git" in it, and the last round we saw on the list didn't do that.

Thanks.
Jens Lehmann· Jan 7, 2013, 08:18 UTC · re: Junio C Hamano · lore

Re: Moving (renaming) submodules, recipe/script

Am 07.01.2013 08:44, schrieb Junio C Hamano:
Show 17 quoted lines
> Jens Lehmann <Jens.Lehmann@web.de> writes:
> 
>> Am 07.01.2013 02:39, schrieb Jonathan Nieder:
>>> (just cc-ing Jens and Peter, who might be interested)
>>
>> I´m currently working on teaching mv to move submodules and intend
>> to send those patches to the list after finishing submodule deinit.
> 
> Thanks for a heads-up.
> 
> As a couple of recent "What's cooking" message has stated, I'll
> shortly kick jl/submodule-deinit topic out of 'next' back to 'pu',
> so please make an update a replacement, not an incremental.  If I
> recall the discussion correctly, I think we agreed that deinit
> should clear the slate, which means the submodule working tree
> should be removed and made into an empty directory without ".git" in
> it, and the last round we saw on the list didn't do that.

Right, and me thinks that would warrant a --force option for deinit to do that even if the submodule contains local changes (which would make deinit fail otherwise). Additionally Michael and Marc spoke up that they would rather have a --all option to deinit all initialized submodules and "git submodule deinit" without any arguments should just produce a usage message. As I saw no voices against it that'll be part of the next iteration too.

Junio C Hamano· Jan 7, 2013, 16:08 UTC · re: Jens Lehmann · lore

Re: Moving (renaming) submodules, recipe/script

Jens Lehmann <Jens.Lehmann@web.de> writes:
> Right, and me thinks that would warrant a --force option for deinit
> to do that even if the submodule contains local changes (which would
> make deinit fail otherwise).
Probably.
Show 5 quoted lines
> Additionally Michael and Marc spoke up
> that they would rather have a --all option to deinit all initialized
> submodules and "git submodule deinit" without any arguments should
> just produce a usage message. As I saw no voices against it that'll
> be part of the next iteration too.

Yeah, I forgot about that possible surprise of deiniting everything under the sun by default.

I am not sure if "--all" is a good way forward, though.

Can you defeat it with "git submodule deinit ./--all" or something to limit the target only to one submodule whose unfortunate location is named as such? If you have such a support, I have this suspicion that you already get a short and explicit way to say "everything under the current directory" with "git submodule deinit ." for free.

W. Trevor King· Jan 7, 2013, 12:08 UTC · re: Jens Lehmann · lore

Re: Moving (renaming) submodules, recipe/script

On Mon, Jan 07, 2013 at 07:59:53AM +0100, Jens Lehmann wrote:
Show 5 quoted lines
> I´m currently working on teaching mv to move submodules and intend
> to send those patches to the list after finishing submodule deinit.
> Please see
>   https://github.com/jlehmann/git-submod-enhancements/commits/mv-submodules
> for the current state of this series.
Ah, that will be much better :).

It looks like my script tries to do all the right things though, so if anyone needs to move a submodule before Jens' branch lands, you might want to give it a try.

Cheers, Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
W. Trevor King· Jan 8, 2013, 14:32 UTC · re: Jens Lehmann · lore

Re: Moving (renaming) submodules, recipe/script

On Mon, Jan 07, 2013 at 07:59:53AM +0100, Jens Lehmann wrote:
Show 8 quoted lines
> Am 07.01.2013 02:39, schrieb Jonathan Nieder:
> > (just cc-ing Jens and Peter, who might be interested)
> 
> I´m currently working on teaching mv to move submodules and intend
> to send those patches to the list after finishing submodule deinit.
> Please see
>   https://github.com/jlehmann/git-submod-enhancements/commits/mv-submodules
> for the current state of this series.

Thinking about this a bit more, I'm not clear on how out-of-tree updates (i.e. worktree in .git/modules/*/config) propogated during branch checkouts (merges, rebases, etc.). I just got a broken rebase trying to move a trivial patch back before the submodule move, and Git was confused about what had happened to the submodules. Here's a simple script that illustrates the problem:

  #!/bin/sh
  rm -rf a b c
  mkdir a
  (cd a
   git init
   echo a > README
   git add README
   git commit -am 'a'
  )
  git clone a b
  (cd b
   git submodule add ../a submod-1
   git commit -am 'add submodule at submod-1'
  )
  git clone b c
  (cd c
   git submodule update --init
  )
  (cd b
   git-submodule-mv.sh submod-1 submod-2
   git commit -am 'move submodule from submod-1 to submod-2'
  )
  (cd c
   git pull
   ls -d .git/modules/*
   cat .git/modules/submod-1/config
   ls -a submod*
  )

The end result is that `c` gets the `.gitmodules` path updates and new gitlinked directory from the submodule move in `b` (using my git-submodule-mv.sh posted earlier in this thread), but `submod-1` is left lying around (because Git doesn't know that it can remove submod-1/.git and submod-1/README). The Git directory for the submodule stays in .git/modules/submod-1/ (good), but the worktree in .git/modules/submod-1/config still points to ../../../submod-1 (bad).

This means that submodule moves are possible, but anyone trying to share them between several repositories (or trying to rebase across the move within their own repository) is in for a world of suffering ;). I'm not sure how this should be addressed, but I didn't see anything handling it in Jens' new series.

Thanks, Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
W. Trevor King· Jan 8, 2013, 15:59 UTC · re: W. Trevor King · lore

Re: Moving (renaming) submodules, recipe/script

On Tue, Jan 08, 2013 at 09:32:14AM -0500, W. Trevor King wrote:
> Thinking about this a bit more, I'm not clear on how out-of-tree
> updates (i.e. worktree in .git/modules/*/config) propogated during
> branch checkouts (merges, rebases, etc.).

Actually, I don't understand why storing `worktree` in .git/modules/*/config is useful at all…. This may be related to my lack of clarity on the "why can't we have multiple working directories checked out at the same time" issue:

On Tue, Oct 23, 2012 at 06:09:55PM -0400, W. Trevor King wrote:
Show 13 quoted lines
> On Tue, Oct 23, 2012 at 10:57:57PM +0200, Jens Lehmann wrote:
> > Am 22.10.2012 14:37, schrieb W. Trevor King:
> > > but cloning a remote repository (vs. checking out a local branch)
> > > seems to be baked into the submodule implementation.  Should I be
> > > thinking about generalizing git-submodule.sh, or am I looking under
> > > the wrong rock?  My ideal syntax would be something like
> > > 
> > >   $ git submodule add -b c --local dir-for-c/
> > 
> > But then we'd have to be able to have two (or more) work trees using
> > the same git directory, which current submodule code can't.
> 
> And that's the problem I'm trying to solve ;).

Can someone with a better feeling for why this won't work. Is is just that there's only one `.git/HEAD`?

Cheers, Trevor

-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy
Jens Lehmann· Jan 8, 2013, 17:12 UTC · re: W. Trevor King · lore

Re: Moving (renaming) submodules, recipe/script

Am 08.01.2013 15:32, schrieb W. Trevor King:
Show 51 quoted lines
> On Mon, Jan 07, 2013 at 07:59:53AM +0100, Jens Lehmann wrote:
>> Am 07.01.2013 02:39, schrieb Jonathan Nieder:
>>> (just cc-ing Jens and Peter, who might be interested)
>>
>> I´m currently working on teaching mv to move submodules and intend
>> to send those patches to the list after finishing submodule deinit.
>> Please see
>>   https://github.com/jlehmann/git-submod-enhancements/commits/mv-submodules
>> for the current state of this series.
> 
> Thinking about this a bit more, I'm not clear on how out-of-tree
> updates (i.e. worktree in .git/modules/*/config) propogated during
> branch checkouts (merges, rebases, etc.).  I just got a broken rebase
> trying to move a trivial patch back before the submodule move, and Git
> was confused about what had happened to the submodules.  Here's a
> simple script that illustrates the problem:
> 
>   #!/bin/sh
>   rm -rf a b c
>   mkdir a
>   (cd a
>    git init
>    echo a > README
>    git add README
>    git commit -am 'a'
>   )
>   git clone a b
>   (cd b
>    git submodule add ../a submod-1
>    git commit -am 'add submodule at submod-1'
>   )
>   git clone b c
>   (cd c
>    git submodule update --init
>   )
>   (cd b
>    git-submodule-mv.sh submod-1 submod-2
>    git commit -am 'move submodule from submod-1 to submod-2'
>   )
>   (cd c
>    git pull
>    ls -d .git/modules/*
>    cat .git/modules/submod-1/config
>    ls -a submod*
>   )
> 
> The end result is that `c` gets the `.gitmodules` path updates and new
> gitlinked directory from the submodule move in `b` (using my
> git-submodule-mv.sh posted earlier in this thread), but `submod-1` is
> left lying around (because Git doesn't know that it can remove
> submod-1/.git and submod-1/README).

That's just what current git does with removed submodules (but my recursive submodule update series will handle that just fine).

>  The Git directory for the
> submodule stays in .git/modules/submod-1/ (good), but the worktree in
> .git/modules/submod-1/config still points to ../../../submod-1 (bad).

You'll not only have to update the gitfile but also the core.worktree setting in the repo. Sorry I missed that when you posted your script.

Show 5 quoted lines
> This means that submodule moves are possible, but anyone trying to
> share them between several repositories (or trying to rebase across
> the move within their own repository) is in for a world of suffering
> ;).  I'm not sure how this should be addressed, but I didn't see
> anything handling it in Jens' new series.

If you adjust core.worktree properly you'll just have the old submodule work tree lying around (just like you do after you rm'd it) and everything apart from that should just work.

As I mentioned that will be fixed by recursive submodule checkout. I'll see if I can polish my preliminary branch so that interested people can play around with it if anyone is interested.

W. Trevor King· Jan 8, 2013, 17:48 UTC · re: Jens Lehmann · lore

Re: Moving (renaming) submodules, recipe/script

On Tue, Jan 08, 2013 at 06:12:13PM +0100, Jens Lehmann wrote:
Show 7 quoted lines
> Am 08.01.2013 15:32, schrieb W. Trevor King:
> >  The Git directory for the
> > submodule stays in .git/modules/submod-1/ (good), but the worktree in
> > .git/modules/submod-1/config still points to ../../../submod-1 (bad).
> 
> You'll not only have to update the gitfile but also the core.worktree
> setting in the repo. Sorry I missed that when you posted your script.

My git-submodule-mv.sh script does update core.worktree. The problem is that `git checkout`, `git merge`, etc. do not.

Show 13 quoted lines
> > This means that submodule moves are possible, but anyone trying to
> > share them between several repositories (or trying to rebase across
> > the move within their own repository) is in for a world of suffering
> > ;).  I'm not sure how this should be addressed, but I didn't see
> > anything handling it in Jens' new series.
> 
> If you adjust core.worktree properly you'll just have the old
> submodule work tree lying around (just like you do after you rm'd
> it) and everything apart from that should just work.
> 
> As I mentioned that will be fixed by recursive submodule checkout.
> I'll see if I can polish my preliminary branch so that interested
> people can play around with it if anyone is interested.

Sounds like a fix will be in here. I'll definitely help put the branch through its paces ;).

Trevor
-- 
This email may be signed or encrypted with GnuPG (http://www.gnupg.org).
For more information, see http://en.wikipedia.org/wiki/Pretty_Good_Privacy

← back to recent threads