[howto] Kernel hacker's guide to git, updated

Just updated my KHGtG to include the latest goodies available in git-core, the Linux kernel standard SCM tool:

	http://linux.yyz.us/git-howto.html

Several changes in git-core have made working with git a lot easier, so be sure to re-familiarize yourself with the development process.

Comments, corrections, and notes of omission welcome. This document mainly reflects my typical day-to-day git activities, and may not be very applicable outside of kernel work.

	Jeff

On 9/29/05, Jeff Garzik <jgarzik@pobox.com> wrote:

Can you update the date on that page to reflect your latest updates? I was digging around with git yesterday and had a few surprises... like checking out the kernel and being told I modified a bunch of files I never touched.

- Dave

On 9/29/05, David Leimbach <leimy2k@gmail.com> wrote:

The  KHGtG page is updated to September 29, 2005.

-- Don't be evil!!!

On 9/29/05, Alberto Patino <pato.lukaz@gmail.com> wrote:

Awesome! Thanks. :-) I'm looking at cogito and darcs-git for working with v9fs and other goodies.

Dave

Am Donnerstag, 29. September 2005 13:03 schrieb Jeff Garzik:

Unfortunately, following the instructions to the letter produces this: oliver@oenone:~/linux-2.6> git checkout usage: read-tree (<sha> | -m <sha1> [<sha2> <sha3>])

	Regards
		Oliver

On Thu, 2005-09-29 at 14:08, Oliver Neukum wrote:

Yeah. See if you still have a .git/HEADS that symlinks to a valid place or not...?

jdl

Am Donnerstag, 29. September 2005 21:34 schrieb Jon Loeliger:

oliver@oenone:~/linux-2.6> ls -la .git/ insgesamt 14 drwxrwxr-x 6 oliver users 224 2005-09-29 21:06 . drwxr-xr-x 3 oliver users 72 2005-09-29 20:45 .. -rw-rw-r-- 1 oliver users 19 2005-05-02 01:02 description lrwxrwxrwx 1 oliver users 17 2005-09-29 21:06 HEAD -> refs/heads/master -rw------- 1 oliver users 32 2005-09-29 21:06 index drwxrwxr-x 2 oliver users 104 2005-09-11 21:41 info drwxr-xr-x 260 oliver users 6240 2005-09-29 19:05 objects drwxrwxr-x 4 oliver users 96 2005-05-02 02:15 refs drwxr-xr-x 2 oliver users 72 2005-09-29 21:05 remotes oliver@oenone:~/linux-2.6> ls -la .git/refs/ insgesamt 1 drwxrwxr-x 4 oliver users 96 2005-05-02 02:15 . drwxrwxr-x 6 oliver users 224 2005-09-29 21:06 .. drwxrwxr-x 2 oliver users 72 2005-09-29 19:05 heads drwxrwxr-x 2 oliver users 600 2005-09-20 05:02 tags oliver@oenone:~/linux-2.6> ls -la .git/refs/heads/ insgesamt 4 drwxrwxr-x 2 oliver users 72 2005-09-29 19:05 . drwxrwxr-x 4 oliver users 96 2005-05-02 02:15 .. -rw-rw-r-- 1 oliver users 41 2005-09-29 19:05 master oliver@oenone:~/linux-2.6> cat .git/refs/heads/master aa55a08687059aa169d10a313c41f238c2070488

	Regards
		Oliver

Oliver Neukum <oliver@neukum.org> writes:

I wonder what that 32-byte small file called 'index' is doing there. If this is the kernel tree, its index file should weigh about 1.7MB if I recall correctly.

Am Freitag, 30. September 2005 09:37 schrieb Junio C Hamano:

I have no idea, but the instructions as stated didn't work for me. Can you replicate that?

	Regards
		Oliver

On Fri, 30 Sep 2005, Junio C Hamano wrote:

32 bytes is what you get for an empty index (it's a 12-byte header, followed by a 20-byte SHA1 of the contents).

In general, such an index file it should be 100% equivalent to not having an index file at all.

		Linus

On Thu, Sep 29, 2005 at 07:03:05AM -0400, Jeff Garzik wrote:
 > 
 > Just updated my KHGtG to include the latest goodies available in 
 > git-core, the Linux kernel standard SCM tool:
 > 
 > 	http://linux.yyz.us/git-howto.html
 > 
 > Several changes in git-core have made working with git a lot easier, so 
 > be sure to re-familiarize yourself with the development process.
 > 
 > Comments, corrections, and notes of omission welcome.  This document 
 > mainly reflects my typical day-to-day git activities, and may not be 
 > very applicable outside of kernel work.

You wrote..

$ git clone rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git linux-2.6
$ cd linux-2.6
$ rsync -a --verbose --stats --progress \
  rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git/ \
  .git/

Could be just..

$ git clone rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git linux-2.6 $ cd linux-2.6 $ git pull

Likewise, in the next section, git pull doesn't need an argument if pulling from the repo it cloned.

		Dave

On Thu, 29 Sep 2005, Dave Jones wrote:

That is not actually the same. "git pull" for example will not download Linus' tags whilst the rsync would get everything.

Now that is definitely correct.  (-:

Best regards,

	Anton

On Thu, Sep 29, 2005 at 09:07:29PM +0100, Anton Altaparmakov wrote:

 > > $ git clone rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git linux-2.6
 > > $ cd linux-2.6
 > > $ rsync -a --verbose --stats --progress \
 > >   rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git/ \
 > >   .git/
 > > 
 > > Could be just..
 > > 
 > > $ git clone rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git linux-2.6
 > > $ cd linux-2.6
 > > $ git pull
 > 
 > That is not actually the same.  "git pull" for example will not download 
 > Linus' tags whilst the rsync would get everything.

Ah. I didn't know this. Thanks. Hmm, it'd be nice to have a shorthand 'not have to type the url, pull everything'. Something like 'git pull all'.

		Dave

On Thu, 29 Sep 2005, Dave Jones wrote:

Something like this?

Except it's called "git fetch --all", and it's obviously totally untested.

 		Linus
--

On Thu, 29 Sep 2005, Linus Torvalds wrote:

Gaah. Using a new pine version, and it is back to corrupting whitespace.

Damn all mailer writers. Do they think email is just for grandma?

 		Linus

On Thu, Sep 29, 2005 at 02:26:42PM -0700, Linus Torvalds wrote:

 > On Thu, 29 Sep 2005, Linus Torvalds wrote:
 > >
 > >Something like this?
 > 
 > Gaah. Using a new pine version, and it is back to corrupting whitespace.
 > 
 > Damn all mailer writers. Do they think email is just for grandma?

What I find amusing is that it was a patch rejection mail from you *years* back (circa 2000 iirc), telling me my pine corrupted whitespace, that made me switch MUA ;-)

All these years later, and it's still buggered ?

		Dave

On Thu, 29 Sep 2005, Dave Jones wrote:

Actually, it seems better. It seems to be buggered by default, but it used to be that you had to actually recompile pine to make it behave. Now you can just disable "strip-whitespace-before-send" and _enable_ "quell-flowed-text" and those together seem to do the trick. No extra patches or recompiles necessary.

So there's progress. 

Of course, pico is still pico. Which I find a bit sad: my editor of choise is still an improved version of uemacs, and pico actually comes from the same uemacs history, but has different key-bindings for just enough keys to be slightly confusing.

Still, that shared history means that I find pico a lot more to my taste than just about any other emailer editor out there. It may have a few differences, but it has more things in common..

		Linus

On Thu, 29 Sep 2005, Linus Torvalds wrote:

Indeed.  I use those two options like that, too.  (-:

Why don't you enable "enable-alternate-editor-implicitly" and set editor = "your-editor-of-choice" in the pine config? It is integrated in a quite seamless way.

Best regards,

	Anton

On Thu, 29 Sep 2005, Anton Altaparmakov wrote:

You think so? I don't find it that way.

With an alternate editor you have to edit the headers separately, and things like postponing a message suddenly turns into a big deal, not just a trivial ^O. In fact, almost everything gets more involved.

And pico _is_ pretty close to uemacs.

		Linus

On Thu, 29 Sep 2005, Linus Torvalds wrote:

Well using vim as alternate editor a postpone turns into a "ZZ" followed 
by ^O which I don't think is such a big deal but in general I agree that 
it adds hassle.  It is a shame that the text based headers do not appear 
as part of the message in the editor in particular...
 

I wouldn't know about that. Never used uemacs. I am a vim addict myself. (-;

Best regards,

	Anton

Anton Altaparmakov <aia21@cam.ac.uk> writes:

Anybody interested in adding entry to SubmittingPatches MUA Specific Help section?

Hi,

On Thu, 29 Sep 2005, Junio C Hamano wrote:

Did I miss something? Straight from my local git directory:

-- snip --

Pine ----

(Johannes Schindelin)

I don't know how many people still use pine, but for those poor souls it may be good to mention that the quell-flowed-text is needed for recent versions.

... the "no-strip-whitespace-before-send" option, too. AFAIK it was introduced in 4.60.

-- snap --

Hth, Dscho

On Thu, Sep 29, 2005 at 02:26:42PM -0700, Linus Torvalds wrote:
 > 
 > 
 > On Thu, 29 Sep 2005, Linus Torvalds wrote:
 > >Something like this?
 > 
 > Gaah. Using a new pine version, and it is back to corrupting whitespace.
 > 
 > Damn all mailer writers. Do they think email is just for grandma?

Of course - grandma uses pine, hackers use mutt ;)

*hides*

Best, Elfyn

On Thu, 29 Sep 2005, Linus Torvalds wrote:

Ok, disabling "text flowing" seems to have fixed it. It still leaves empty spaces at the end of lines when doing normal word-wrapping in the editor (and then seems to use those empty spaces as a "marker" for flowing), but that's at least just a small silly detail.

So how about this patch now? With it you can do

	git fetch --tags <linus-kernel-repo>

and it should fetch all my tags automatically.

		Linus
---

On Thu, Sep 29, 2005 at 02:35:15PM -0700, Linus Torvalds wrote:
 > 
 > 
 > On Thu, 29 Sep 2005, Linus Torvalds wrote:
 > > 
 > > Gaah. Using a new pine version, and it is back to corrupting whitespace.
 > 
 > Ok, disabling "text flowing" seems to have fixed it. It still leaves empty 
 > spaces at the end of lines when doing normal word-wrapping in the editor 
 > (and then seems to use those empty spaces as a "marker" for flowing), but 
 > that's at least just a small silly detail.
 > 
 > So how about this patch now? With it you can do
 > 
 > 	git fetch --tags <linus-kernel-repo>
 > 
 > and it should fetch all my tags automatically.

Seems to work. I blew away git/refs/tags/v2.6.13* and git fetch --tags fetched them all back just fine.

		Dave

Dave Jones wrote:

Nope. It intentionally includes the manual rsync because clone/pull doesn't seem to grab tags. Or at least last time I checked...

	Jeff

Jeff Garzik <jgarzik@pobox.com> writes:

clone does (or should), fetch (and hence pull) does not automatically. Running the following one-liner every once in a while would sync your set of tags with Linus:

git fetch origin `git-ls-remote --tags origin | sed -ne 's|^.*refs/tags/|tag |p'`

Thanks for all the comments. I just updated the KHGtG with the feedback I received. Go to

	http://linux.yyz.us/git-howto.html

and click reload.  Continued criticism^H^H^Hcomments welcome!

Two items of note:

1)

that's way too long and convoluted to deal with. Once the 'git fetch --tags' changes make it into the official repository (are they there already?), I'll remove all the remaining direct references to running rsync.

2) What is the easiest way to obtain a list of changes present in 
repository B, that are not present in repository A?  I used to use 
git-changes-script [hacked cg-log script] for this:

	$ cd /repo/netdev-2.6
	$ git-changes-script -L ../linux-2.6

would display all changes in /repo/netdev-2.6 which are not present in /repo/linux-2.6. This is similar to 'git log master..HEAD', except for repositories rather than branches.

	Jeff

Jeff Garzik <jgarzik@pobox.com> writes:

Sounds like a thinly veiled threat and/or very effective prodding ;-).

It is not there yet only because I simply have not got around to it, but it will happen before 0.99.8.

I suspect the version Linus posted has a funny interaction with 'git-pull'; 'git pull --tags' by mistake, or intentionally to file a bug report to annoy me ;-), would create an Octopus out of those tags, if I am not mistaken.

I think I still have the copy you sent to the list. If you do not mind me placing in the master branch just holler -- better yet please send a patch with commit log and signoff to add the latest, and I will apply it.

-jc

Junio C Hamano wrote:

It's archived here: http://www.kernel.org/pub/linux/kernel/people/jgarzik/git-changes-script

but it needs a git expert (read: not me :)) to fix it up for the very latest git-core stuff.

Currently, using 'git-changes-script -L ../linux-2.6' spits out

Regards,

	Jeff

Jeff Garzik <jgarzik@pobox.com> writes:

I haven't really *read* that script, but I think it is trying to make a list of commits from both repositories and trying to find the set that are in one side and not in the other using diff (a real shell programer probably would have used "comm" for this kind of task, not "diff"), then doing a handcrafted git-log on each of them.

Attached is my quick hack, based on your original question, without really trying to understand what the script is doing, so I cannot claim it is a rewrite nor even attempting to be compatible. Please take a look at it and tell me if this is any close to what you need.

I have a suspition that this might be better done as a natural extension of git-log, though.

------------ #!/bin/sh # # Copyright (c) 2005 Junio C Hamano #

. git-sh-setup || die "Not a git archive"

usage () {
	echo >&2 "$0 ( -L | -R ) <dir> [<ref>] [<ref>]

-L shows changes in local not in remote. -R shows changes in remote not in local. <dir> names the remote repository.

If given no refs, local and remote HEADs are compared.
If given one ref, local HEAD and named remote ref are compared.
If given two refs, the first names a local ref, and the second names
remote ref to be compared.
"
	exit 1
}

case "$1" in
-L | -R)
	;;
*)
	usage ;;
esac

other="$2"

(
	unset GIT_DIR GIT_OBJECT_DIRECTORY
	cd "$other" && . git-sh-setup ||
	die "$other is not a valid git repository."
)

local=${3:-HEAD} remote=${4:-HEAD}

# Basic validation. local=$(git-rev-parse --verify "$local^0" 2>/dev/null) || die "local ref $local is not valid." remote=$(GIT_DIR="$other" git-rev-parse --verify "$remote^0" 2>/dev/null) || die "remote ref $remote is not valid."

case "$1" in
-L)
	list_args="$local ^$remote" ;;
-R)
	list_args="^$local $remote" ;;
esac

GAOD="$GIT_ALTERNATE_OBJECT_DIRECTORIES"

GIT_ALTERNATE_OBJECT_DIRECTORIES="$other/.git/objects:$GAOD" \ git-rev-list --pretty $list_args | LESS=-S ${PAGER:-less}

On Fri, 30 Sep 2005, Junio C Hamano wrote:

Hey, even more impressive is "git pull --all", which will happily try to create an octopus of every single ref available at the other end.

Now, I think that octopus merges in _general_ are likely to be driver error, and that it might make sense to have a separate flag to enable them in the first place. That would solve the confusion..

So then you could do

	git pull --all --octopus xyzzy

if you _really_ meant to do that. 

		Linus

Linus Torvalds <torvalds@osdl.org> writes:

True.

However, I think --all is a mistake even if you use it without merging in 'git fetch', so I am not planning to do refs/heads/ side, at least not yet. Even if you prevent an Octopus, what would you do then? If you choose to merge one of them, which one? Not merging any that is not explicitly specified on the command line, seems to me the most sensible and safe option.

The rule for 'pull' to decide which refs to merge is:

 (1) if command line has explicit refspecs (--tags and --heads
     do not count), they are all merged.

 (2) if command line has no explicit refspecs (--tags and
     --heads do not count), the default one found from either
     remotes or branches file is merged.

Notice that I am forbidding remotes file to say "by default I always merge these three heads from there to make an Octopus" by the above rule (branches file cannot even name more than one head so this is not an issue). Since everybody seems to agree that Octopus is not something that is done mechanically and routinely anyway [*1*], I think this is a sensible way to guard against accidental Octopus.

We could consider fetching all heads, by minimally renaming remote master to origin and getting everything else under the same name, but I'd really want to keep the local namespace for branches isolated from each other. Many kernel.org public repositories seem to have 'test' and 'release' branches and if you are a maintainer of such a tree, and if you are interested in another maintainer's tree, and if that other maintainer has the 'test' and 'release' branches, --heads (or --tags) overwriting your 'test' with his 'test' is obviously not what you want.

Possibly, something like this could be arranged later:

	* git fetch --heads=$ns $remote "$@"

	In addition to the usual refspecs (the rest of the
	command line arguments), fetch all remote heads and
	store remote refs/heads/$a under local refs/heads/$ns/$a
	for all $a.  If $ns is empty, remote "master" is renamed
	"origin".

	* git fetch --heads $remote "$@"

        shorthand for empty $ns

[Footnote]

*1* I do make many Octopus merges, but they happen across my local topic branches. Topics merged change day-by-day, and even the set of topics alive at the time changes everyday. IOW, it is not something I would want to do with the same sets of heads every time by describing them in the remotes file.

Am Freitag, 30. September 2005 13:15 schrieb Jeff Garzik:

The error is still there.

oliver@oenone:~/linux-2.6> git checkout -f master usage: read-tree (<sha> | -m <sha1> [<sha2> <sha3>])

	Regards
		Oliver

Oliver Neukum wrote:

Something's wrong with your installation, not the instructions. Works fine here.

	Jeff

On 9/30/05, Jeff Garzik <jgarzik@pobox.com> wrote:

Right , you must have a corrupted copy of the git repo. Do the initial cloning of the linux repository again, and then retry the git checkout.

On Fri, Sep 30, 2005 at 07:15:41AM -0400, Jeff Garzik wrote:

FYI, the rsync command to get the tags blows away .git/branches/origin, so on the next "git pull", git will tell you "Where do you want to fetch from today?".

In the mean time, better make it two separate rsync commands (sorry for the long lines):

rsync -a --delete --verbose --stats --progress \
  rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git/tags/ \
  .git/tags/

rsync -a --delete --verbose --stats --progress \
  rsync://rsync.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git/heads/ \
  .git/heads/

Erik

Erik Mouw wrote:

Agreed, though I always run 'git pull $url' each time, so I never noticed this.

Since git clone pulls the tags, I have eliminated the first rsync completely.

	Jeff

Jeff Garzik <jgarzik@pobox.com> : [...]

The basic tasks make no use of git-whatchanged.

Even if it is a CVSism, it is quite handful to retrieve the history of the patchsets by aiming at a specific file.

-- Ueimor

Jeff Garzik wrote:

i just used the instructions at the bottom of your web page to grab new tags, and it completely wiped all my Stacked Git metadata. (fortunately i was trying this in a test repository, and it will be simple to recover the missing files).

so, this is probably something that should have a warning, or maybe you might consider providing an example that won't wipe out existing files and directories under .git/ ...

begin:vcard fn:Chuck Lever n:Lever;Charles org:Network Appliance, Incorporated;Linux NFS Client Development adr:535 West William Street, Suite 3100;;Center for Information Technology Integration;Ann Arbor;MI;48103-4943;USA email;internet:cel@citi.umich.edu title:Member of Technical Staff tel;work:+1 734 763 4415 tel;fax:+1 734 763 4434 tel;home:+1 734 668 1089 x-mozilla-html:FALSE url:http://www.monkey.org/~cel/ version:2.1 end:vcard