threads / rfc / 43273

patchRe: [RFC/PATCH] Implement poor-man's submodule support using commit hooks

Subject: Re: [RFC/PATCH] Implement poor-man's submodule support using commit hooks

## tl;dr

14 messages between Dec 20, 2006 and Dec 20, 2006. Diffs are folded; open one to read it.

replies: 13people: 7as markdown or json

Andy Parkins· Dec 20, 2006, 13:09 UTC · lore

[RFC/PATCH] Implement poor-man's submodule support using commit hooks

Make a file called .gitmodules. In it, list the paths containing a submodule. Add that file to the index.

This patch adds a check to the pre-commit hook finds that file and pulls the HEAD hash out of each of the listed submodule repositories. That hash is then listed to the .gitmodules file along with the submodule name and .gitmodules is added back to the repository.

You've now got poor-man's submodule support. Any commits to the submodule will change the hash and hence the .gitmodules file will be different and therefore will show up as "modified" to git.

It's not got any nice UI for checking out (obviously) or merging; but it does at least record the state of a project. With a bit of manual work you can easily check out the right commit in the submodule. If there were a post-checkout hook script, this could probably be automated.

To prevent git-prune in the submodule from removing references that the supermodule refers to the post-commit hook reads the .gitmodules file and creates a file in submodule/.git/refs/superrefs/ that refers to the hash we've references. git-prune in the submodule will find that reference and hence won't remove it from under us.

Problems:
 - git-prune in the supermodule doesn't clean the supermodule refs in
   the submodule
 - no checkout support
 - no reset support
 - no merge support (other than what git provides for the .gitmodule
   file)
 - no check for dirty submodule before commit
Signed-off-by: Andy Parkins <andyparkins@gmail.com>
---
This is in replacement of the previous patch.

I've used the --show-cdup option of git-rev-parse to make it work even in subdirectories. I've also added git-prune protection by adding a post-commit script to reference the hash in the submodule.

What'd you reckon? Might be useful until real submodule support arrives. If there is no .gitmodules file in the root, then git behaves as it always did.

I suppose if this were actually found to be really useful, it should go in git-commit.sh itself, rather than the hooks.

 templates/hooks--post-commit |   29 +++++++++++++++++++++++++-
 templates/hooks--pre-commit  |   47 +++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 74 insertions(+), 2 deletions(-)
Show changes to 2 files +74 −2

templates/hooks--post-commit, templates/hooks--pre-commit

diff --git a/templates/hooks--post-commit b/templates/hooks--post-commit
index 8be6f34..551d928 100644
--- a/templates/hooks--post-commit
+++ b/templates/hooks--post-commit
@@ -5,4 +5,31 @@
 #
 # To enable this hook, make this file executable.
 
-: Nothing
+# Poor-man's Submodules
+# ---------------------
+# If we're here, then a commit has succeeded. If submodule support is enabled
+# then we need a way of telling the submodule that we now reference a hash
+# owned by it, so that it is not pruned.
+WORKINGTOP=$(git-rev-parse --show-cdup)
+GITMODULES="${WORKINGTOP}.gitmodules"
+if [ -f "$GITMODULES" ]; then
+	cat "$GITMODULES" |
+	while read subdir hash
+	do
+		SUBMODULEPATH="$WORKINGTOP$subdir/.git/refs/superrefs"
+
+		# XXX: check if the line is a comment
+
+		# check if the subdir is a repository
+		if [ ! -d "$WORKINGTOP$subdir/.git" ]; then
+			continue;
+		fi
+
+		# Write the hash to a file of the same name - this means that if we get
+		# multiple commits that refer to the submodule, we only get one file in
+		# the submodule, as the submodule hash is constant across supermodule
+		# commits
+		mkdir -p "$SUBMODULEPATH"
+		echo $hash > "$SUBMODULEPATH/$hash"
+	done
+fi
diff --git a/templates/hooks--pre-commit b/templates/hooks--pre-commit
index 723a9ef..7718369 100644
--- a/templates/hooks--pre-commit
+++ b/templates/hooks--pre-commit
@@ -67,5 +67,50 @@ perl -e '
 	}
     }
     exit($found_bad);
-'
+' || exit 1
+
+# Poor-man's Submodules
+# ---------------------
+# Enable poor-man's submodule support when .gitmodules is present
+# Simply create a .gitmodules file listing the paths in your repository
+# that contain other git repositories; each line will be replaced with the
+# path followed by the hash of the current HEAD of the submodule.
+# When the submodule changes hash this file will be different from the
+# version in the repository and a change will be noted - voila, submodules.
+# Of course there is no checkout support, but at least the current state
+# will be accurately recorded
+WORKINGTOP=$(git-rev-parse --show-cdup)
+GITMODULES="${WORKINGTOP}.gitmodules"
+if [ -f "$GITMODULES" ]; then
+	cat "$GITMODULES" |
+	while read subdir hash
+	do
+		# check if the line is a comment and output it anyway
+		if (expr "x$subdir" : "x#" >/dev/null) then
+			echo "$subdir $hash"
+			continue;
+		fi
+
+		# check if the subdir is a repository
+		if [ ! -d "$WORKINGTOP$subdir/.git" ]; then
+			echo "$subdir is not a git repository, so it can't be a submodule"
+			exit 1;
+		fi
+
+		# XXX: really need a check here and quit if the submodule is
+		# dirty
+
+		echo "$subdir $(GIT_DIR=$WORKINGTOP$subdir/.git git-rev-parse HEAD)"
+	done > newgitmodules
+	# Update
+	mv newgitmodules "$GITMODULES"
+
+	# This relies on the .gitmodules file having already been added to
+	# the repository - perhaps this should be automated?
+	git-update-index "$GITMODULES" ||
+	(
+	echo "FATAL: Submodule tracker file, $GITMODULES, is not tracked in this repository." >&2
+	exit 1
+	)
+fi
 
-- 
1.4.4.2.g0d2a
Johannes Sixt· Dec 20, 2006, 13:29 UTC · re: Andy Parkins · lore
Andy Parkins wrote:
> +WORKINGTOP=$(git-rev-parse --show-cdup)
> +GITMODULES="${WORKINGTOP}.gitmodules"
> +if [ -f "$GITMODULES" ]; then
> +       cat "$GITMODULES" |
useless-use-of-cat-syndrome
> +       while read subdir hash

Wouldn't it be better to have the order of subdir and hash swapped? That way subdir may contain blanks, and it gives nicer alignment in the file because of the constant length of the hashes.

-- Hannes
Andy Parkins· Dec 20, 2006, 13:47 UTC · re: Johannes Sixt · lore
On Wednesday 2006 December 20 13:29, Johannes Sixt wrote:
> > +       cat "$GITMODULES" |
>
> useless-use-of-cat-syndrome
I tried 
 while
 do 
 done < file1 > file1

And that didn't work. However, it was racy anyway using the same file, so I changed to as it is now, but forgot to switch back to input redirection.

Show changes to 2 files +1 −2

git-commit.sh, templates/hooks--pre-commit

diff --git a/git-commit.sh b/git-commit.sh
diff --git a/templates/hooks--pre-commit b/templates/hooks--pre-commit
index 7718369..74edfe2 100644
--- a/templates/hooks--pre-commit
+++ b/templates/hooks--pre-commit
@@ -82,7 +82,6 @@ perl -e '
 WORKINGTOP=$(git-rev-parse --show-cdup)
 GITMODULES="${WORKINGTOP}.gitmodules"
 if [ -f "$GITMODULES" ]; then
-   cat "$GITMODULES" |
    while read subdir hash
    do
        # check if the line is a comment and output it anyway
@@ -101,7 +100,7 @@ if [ -f "$GITMODULES" ]; then
        # dirty
 
        echo "$subdir $(GIT_DIR=$WORKINGTOP$subdir/.git git-rev-parse HEAD)"
-   done > newgitmodules
+   done < "$GITMODULES" > newgitmodules
    # Update
    mv newgitmodules "$GITMODULES"

> > +       while read subdir hash
>
> Wouldn't it be better to have the order of subdir and hash swapped? That
> way subdir may contain blanks, and it gives nicer alignment in the file
> because of the constant length of the hashes.

Unfortunately, it is the hash that is optional.  When you create the file, you 
don't list the hashes, you list the subdirectories.  I suppose I could make 
it so you have to give "000000" or something first?



Andy
-- 
Dr Andy Parkins, M Eng (hons), MIEE
Jakub Narebski· Dec 20, 2006, 14:07 UTC · re: Andy Parkins · lore
Andy Parkins wrote:
> On Wednesday 2006 December 20 13:29, Johannes Sixt wrote:
Show 9 quoted lines
>>> +       while read subdir hash
>>
>> Wouldn't it be better to have the order of subdir and hash swapped? That
>> way subdir may contain blanks, and it gives nicer alignment in the file
>> because of the constant length of the hashes.
> 
> Unfortunately, it is the hash that is optional.  When you create the file, you 
> don't list the hashes, you list the subdirectories.  I suppose I could make 
> it so you have to give "000000" or something first?
That's the convention git uses in git-diff, reflog, etc.
-- 
Jakub Narebski
Warsaw, Poland
ShadeHawk on #git
Andy Parkins· Dec 20, 2006, 14:20 UTC · re: Jakub Narebski · lore
On Wednesday 2006 December 20 14:07, Jakub Narebski wrote:
Show 5 quoted lines
> > Unfortunately, it is the hash that is optional.  When you create the
> > file, you don't list the hashes, you list the subdirectories.  I suppose
> > I could make it so you have to give "000000" or something first?
>
> That's the convention git uses in git-diff, reflog, etc.

They are all outputs though; this is an input. I did it that way round so that I could say:

cat .gitmodules <<EOF submodule1 submodule2 EOF

And that was it. Of course, this doesn't work when there are spaces in the filename, so I guess it's got to be

cat .gitmodules <<EOF 000 submodule1 000 submodule2 EOF

I'll wait to see if there are more objections before posting an updated patch.
Andy
-- 
Dr Andy Parkins, M Eng (hons), MIEE
Rogan Dawes· Dec 20, 2006, 14:33 UTC · re: Andy Parkins · lore
Andy Parkins wrote:
Show 12 quoted lines
> On Wednesday 2006 December 20 13:29, Johannes Sixt wrote:
> 
>>> +       cat "$GITMODULES" |
>> useless-use-of-cat-syndrome
> 
> I tried 
> 
>  while
>  do 
>  done < file1 > file1
> 
> And that didn't work. 

It wouldn't, since the redirection overwrites the file at the same time as trying to read from it. A better way is to redirect to a temp file, and rename it if the previous operation was successful.

e.g.

0 $ echo true > t 0 $ sed 's/true/false/g' < t > t~ && mv t~ t || rm t~ 0 $ cat t false

and

0 $ echo true > t 0 $ (sed 's/true/false/g' < t > t~; false) && mv t~ t || rm t~ 0 $ cat t true 0 $ ls t~ ls: t~: No such file or directory 2 $

Andy Parkins· Dec 20, 2006, 14:40 UTC · re: Rogan Dawes · lore
On Wednesday 2006 December 20 14:33, Rogan Dawes wrote:
> It wouldn't, since the redirection overwrites the file at the same time

Of course you are correct; however it wasn't completely crazy. Before now, I've replaced files in C by doing

 fd_old = open( "someexistingfile" );
 unlink( "someexistingfile" );
 fd_new = open( "someexistingfile" );
So I was really just trying it in case bash did a similar thing internally.
> as trying to read from it. A better way is to redirect to a temp file,
> and rename it if the previous operation was successful.

That's what I ended up doing; as you say it's better anyway because the error conditions don't mean that the .gitmodules file ends up half written.

Andy
-- 
Dr Andy Parkins, M Eng (hons), MIEE
Randal L. Schwartz· Dec 20, 2006, 15:42 UTC · re: Andy Parkins · lore
>>>>> "Andy" == Andy Parkins <andyparkins@gmail.com> writes:

Andy> fd_old = open( "someexistingfile" ); Andy> unlink( "someexistingfile" ); Andy> fd_new = open( "someexistingfile" );

This is dangerous if the process aborts suddenly.
Might be better to do this:

mumble_command < thefile > thefile.$$ mv thefile.$$ thefile

The second move will be atomic. The only problem will be if the process aborts, which will leave stale temp files around.

-- 
Randal L. Schwartz - Stonehenge Consulting Services, Inc. - +1 503 777 0095
<merlyn@stonehenge.com> <URL:http://www.stonehenge.com/merlyn/>
Perl/Unix/security consulting, Technical writing, Comedy, etc. etc.
Johannes Schindelin· Dec 20, 2006, 16:09 UTC · re: Andy Parkins · lore
Hi,
On Wed, 20 Dec 2006, Andy Parkins wrote:
Show 5 quoted lines
>  Before now, I've replaced files in C by doing
> 
>  fd_old = open( "someexistingfile" );
>  unlink( "someexistingfile" );
>  fd_new = open( "someexistingfile" );
Lucky you! Obiously you are not stuck in the 10th circle called Windows.
Ciao,
Martin Waitz· Dec 20, 2006, 13:36 UTC · re: Andy Parkins · lore
hoi :)
On Wed, Dec 20, 2006 at 01:09:01PM +0000, Andy Parkins wrote:
> 
> You've now got poor-man's submodule support.  Any commits to the
> submodule will change the hash and hence the .gitmodules file will be
> different and therefore will show up as "modified" to git.

I started with something similiar, too. You can have a look at http://git.admingilde.org/tali/git.git/module which tries to implement submodules without changing the core.

-- 
Martin Waitz
Andy Parkins· Dec 20, 2006, 13:48 UTC · re: Martin Waitz · lore
On Wednesday 2006 December 20 13:36, Martin Waitz wrote:
> I started with something similiar, too.
> You can have a look at http://git.admingilde.org/tali/git.git/module
> which tries to implement submodules without changing the core.

I wasn't really trying to do away with the need for core changes; I just wanted something simple that I could use today.

If this is old news - no problem.  Drop it.
Andy
-- 
Dr Andy Parkins, M Eng (hons), MIEE
Andy Parkins· Dec 20, 2006, 14:35 UTC · re: Martin Waitz · lore
On Wednesday 2006 December 20 13:36, Martin Waitz wrote:
> I started with something similiar, too.
> You can have a look at http://git.admingilde.org/tali/git.git/module
> which tries to implement submodules without changing the core.

I had a look at that; and my previous objection still applies: it blends two repositories. I'd rather they were separate.

Having them separate is what makes my little hook scripts really simple; there is no special initialisation script needed, and the neither the supermodule nor the submodule needs special treatment; and there is no need for any alternates or new object type. If I want to forget about submodule support I just delete the .gitmodules file and it's all back to normal.

I don't say you're wrong in your chosen method, and I don't say I'm right. I wasn't really planning on going much further with these scripts, they actually represent all that I personally want from a submodule system, I am sure others would want far more sophistication.

Andy
-- 
Dr Andy Parkins, M Eng (hons), MIEE
Martin Waitz· Dec 20, 2006, 15:44 UTC · re: Andy Parkins · lore
hoi :)
On Wed, Dec 20, 2006 at 02:35:48PM +0000, Andy Parkins wrote:
> I had a look at that; and my previous objection still applies: it blends two 
> repositories.  I'd rather they were separate.
yes, but one can see what is needed and where the real problems are.

And I really think we should only start adding submodule code to mainline git when we are sure that we found an approach where we can see that it works and that we don't have to change it later.

-- 
Martin Waitz
Johannes Sixt· Dec 20, 2006, 14:18 UTC · re: Andy Parkins · lore
Andy Parkins wrote:
> +               # check if the subdir is a repository
> +               if [ ! -d "$WORKINGTOP$subdir/.git" ]; then
> +                       echo "$subdir is not a git repository, so it can't be a submodule"

You must redirect the message to >&2 otherwise it remains invisible (it goes into file newgitmodules).

-- Hannes

← back to recent threads