Last mile to 1.0?

I do not know what release plan Linus has in mind, and also expect things to be quieter next week during OLS and kernel summit, but I think we are getting really really close.

Here are the things I think we would want to see before we hit 1.0:

 - Remaining feature enhancements and fixes.

   - Anonymous pull from packed archives on remote sites via
     non-rsync, non-ssh transport.  Many people are behind
     corporate firewalls that do not pass anything but outgoing
     http(s) and some do not even pass outgoing ssh.  The recent
     addition of git-daemon by Linus would greatly alleviate the
     situation, but we may also end up wanting something HTTP
     reachable.

 - Documentation.

   - Many files under Documentation/ directory have become
     stale.  I've tried to do one pass of full sweep recently,
     but I'd like somebody else to make another pass to make
     sure that the usage strings in programs, what the programs
     do, and what Documentation says they do match.  Also, the
     spelling and grammar fixes, which I am very bad at and have
     not done any attempt, needs to be done.

     Volunteers?

   - Are all the files in Documentation/ reachable from git(7)
     or otherwise made into a standalone document using asciidoc
     by the Makefile?  I haven't looked into documentation
     generation myself (I use only the text files as they are);
     help to update the Makefile by somebody handy with asciidoc
     suite is greatly appreciated here.

     Volunteers?

   - We may want to describe more Best Current Practices, along
     the lines of "Working with Others" section in the tutorial.
     Please write on your faviorite topic and send patches in
     ;-))

 - Publicity.  I would be very happy to see somebody with good
   writing and summarizing skills to prepare an article to be
   published on LWN.NET to coincide with the 1.0 release.  An
   update to GIT traffic would also be nice.

Junio C Hamano <junkio@cox.net> writes:

For this we need a cgi script that will generate an appropriate pack. Although stupid http fetching may have some potential if we ditch libcurl and use pipelining for http 1.1. Bandwidth wise that will never equal a custom pack because it will not do deltas. But in the common case of an incremental pull it should be able to equal rsync.

Do we want to put some porcelain around, git-fsck-cache --tags? So we can discover the tag objects in the archive and place them someplace usable. Jeff Garzik in his howto is still recommending:

Which feels like something is missing. Given that tags are sha1-indexed objects we should be pulling them. And I believe you can have a tag as a parent of a commit, so even with the pack optimized clients we should be pulling them now.

Eric

ebiederm@xmission.com (Eric W. Biederman) writes:

I agree that nothing would beat a pack customized for each puller from the bandwidth point of view. I like the general idea of git-daemon Linus did and the cgi script you suggest, but I wonder what the CPU/disk load implications for the server.

On Sat, 16 Jul 2005, Junio C Hamano wrote:

I think you need to nail down the various scenerios that people will be useing here.

a very common one will be prople who want to setup a cron job to update their local tree nightly, in this case having a pre-generated pack file with each day's updates will save a significant amount of processing power.

would it make sense to have it do something along the lines of sending the day;s pack file plus a small number of individual object (even if the pack file will partially duplicate object the puller already has)

David Lang

David Lang <david.lang@digitalinsight.com> writes:

I think that would be a reasonable thing to do. The server for anonymous puller, git-daemon, may need some extending. As people criticised, git-upload-pack/git-fetch-pack protocol being not easily extensible, we would end up doing another protocol, though, that's OK.

Fortunately [*1*], git-daemon itself is written to be extensible if you are willing to introduce a totally new protocol driver, so if this on-demand pack generation turns out to be too much a resource hog, we have a reasonably easy way out.

[Footnote]

*1* We are fortunate but that is not by dumb luck. When he did daemon.c::execute(), Linus must have thought things through.

Satturday, 16 July 2005 21:46 Junio C Hamano wrote:

I'd add the UTF-8 native support. Currently neither commit nor gitk doesn't support that. Probably this should be done at as low as possible level.

Alexey Nezhdanov <snake@penza-gsm.ru> writes:

I do not understand your proposal. Care to clarify? You can write your commit messages in UTF-8 today and "git-cat-file commit $commit_ID", which is as low level as you can go, gives you that commit message in UTF-8.

Monday, 18 July 2005 09:35 Junio C Hamano wrote:

Yes. But it have not integration with user environment. Personally I use koi8-r environment and doing "cg commit" (and probably "git commit" too) results in koi8-r text written as commit text. I remember that this issue were discussed on this list and all come to conclusion that utf-8 is the only sane encoding that can be used in this case. But this should not be user's problem - it's just the UI that doesn't understands when transcoding should be done. Similarly - when I start gitk then I see only weird symbols instead of commit text, not matter what encoding I use for commits - koi8-r or utf-8. BTW russian koi8-r encoded text _in_diff_contents_ show correctly. Wonder why...

Alexey Nezhdanov <snake@penza-gsm.ru> writes:

I disagree.

Yes, while you _could_ do something like this to feed text in local encoding to the VISUAL/EDITOR, and convert it back to UTF-8:

On Sat, Jul 16, 2005 at 10:46:00AM -0700, Junio C Hamano wrote:

Looking at the set of patches we just all dumped on Linus, I think they pretty much show us that we don't have any major issues.

As I see it, the status is currently like this:

Revision control - Stable Pulling locally or over rsync - Stable Pushing over ssh - Stable

Remote, anonymous pulls not using rsync - Beta Usability features[1] - Beta

Documentation - Alpha

My feeling is that we're pretty well set to do a 1.0 release.

1 - Usability features are all the things around git-apply, git-format-patch, etc, that we're clearly working on to make life more pleasant, but aren't really critical.

On Saturday 23 July 2005 04:15, Ryan Anderson wrote:

One old farts comment re the docs here folks.

This will need to be improved considerably if you want all us lurking frogs to be able to use it without drowning this list with what really should be RTFMable questions. Specifically, we should be able to dl one package and install something that, by reading the manpages, can be made to work OOTB given an adequate network connection.

My lurking & reading the mail here tends to give me the impression that while it can be made to work, there are yet rough edges for the new user. Potentially discouraging rough edges...

Several days ago there was a discussion on discovering remote tags and branches. This was somewhat related to supporting packed repositories on a dumb server I have been toying with, so here is my current status.

The first three patches deal with the discovery of remote tags and heads:

  [PATCH 1/6] git-peek-remote: show tags and heads from a remote repository.
  [PATCH 2/6] Documentation: git-peek-remote.
  [PATCH 3/6] git-ls-remote: show and optionally store remote refs.

The discovery over http transport against a dumb server needs to have a couple of files to support it, which is prepared with the next two patches.

  [PATCH 4/6] Add update-server-info.
  [PATCH 5/6] Document update-server-info.

The git-update-server-info command introduced by these two patches prepare not just the list of tags and heads, but the list of packs and commit ancestry information. Using that, cloning a packed repository over http transport from a dumb server becomes trivial.

  [PATCH 6/6] Support cloning packed repo from dumb http servers.

The next task would be to add support of the same to git-fetch.

On Sat, Jul 16, 2005 at 10:46:00AM -0700, Junio C Hamano wrote:

How is this for a start?

Source Code Management with Git

Git, sometimes called "global information tracker", is a "directory content manager". Git has been designed to handle absolutely massive projects with speed and efficiency, and the release of the 2.6.12 and (soon) the 2.6.13 version of the Linux kernel would indicate that it does this task well.

Git falls into the category of distributed source code management tools, similar to Arch or Darcs (or, in the commercial world, BitKeeper). This means that every working directory is a full-fledged repository with full revision tracking capabilities.

Git uses the SHA1 hash algorithm to provide a content-addressable pseudo
filesystem, complete with its own version of fsck.
  o Speed of use, both for the project maintainer, and the end-users, is
    a key development principle.
  o The history is stored as a directed acyclic graph, making long-lived
    branches and repeated merging simple.
  o A collection of related projects are building on the core Git
    project, either to provide an easier to use interface on top (Darcs,
    Mercurial, StGit, Cogito), or to take some of the underlying concepts
    and reimplement them directly into another system (Arch 2.0).
  o Two, interchangeable, on-disk formats are used:
    o An efficient, packed format that saves spaced and network
      bandwidth.
    o An unpacked format, optimized for fast writes and incremental
      work.

Git results from the inspiration and frustration of Linus Torvalds, and the enthusiastic help of over 300 participants on the development mailing list.[1]

1 - Generated with the following, in a maildir folder:
	find . -type f | xargs grep -h "^From:" | perl -ne \
	'tr#A-Z#a-z#; m#<(.*)># && print $1,"\n";' | sort -u | wc -l 

Duplicate send, had typo in orif address line :( On Saturday 23 July 2005 04:50, Ryan Anderson wrote:

A very good start for the overview preamble.  Do carry on.

Ryan Anderson wrote:

That's not actually what "distributed" means. There are several distributed SCM tools[1] that store repo information outside the actual working directory.

Perhaps that last sentence could be something like "This means that each developer has a local full-fledged repository with full revision tracking capabilities, not dependent on network access to a central server." I'm sure there are better wordings, but I hate to point out an problem without offering at least one possible improvement.

Kevin

[1] I believe that ArX, monotone, codeville, and svk all fall into this category. Possibly even Arch itself, although I haven't researched that.

Ryan Anderson <ryan@michonline.com> writes:

A very good start indeed.  Thanks.

I think Kevin's comment is valid and his description is reasonable.

I think you would want to drop Darcs and Mercurial from the "on top" list. If I understand correctly, Mercurial is independently written with its own on-disk formats [*1*]. It would be very unfair to put Darcs in "building on" category. They've been there for quite some time with their own repository format and the tools and interfaces are reasonably mature.

Instead, please add gitk and gitweb to the list. We should not forget that these "mostly read-only" things are Porcelains.

[Footnote]

*1* It feels that actually it is done right. Its misfortune is that many core kernel people have already switched to git.

Satturday, 23 July 2005 21:09 Junio C Hamano wrote:

Then add qgit too to provide fair coverage.

Dear diary, on Sat, Jul 16, 2005 at 07:46:00PM CEST, I got a letter where Junio C Hamano <junkio@cox.net> told me that...

I hope to get to it tomorrow but it now occurred to me that I don't know when do you actually want to release 1.0 and I think it's crucial for it to support some sensible HTTP transport - I saw some scripts going in etc, but what's its current state? Is it usable?

Note that I really _loved_ the Daniel's tools while they lasted. What I loved most about them was that they really only pulled objects I needed and not a single worthless one. Does the current HTTP transport share this property?

Note that I also want to setup a simple "proof-of-concept" GIT homepage tomorrow. Well, write it, where it should be hosted can be worked out later and I have places for it to reside at for now. (Suggestions for final hosting welcome. In reality, how nice (and persistent) the URL gets is probably the only thing that really matters. My attempt will live at http://git.or.cz/.)

Petr Baudis <pasky@suse.cz> writes:

I am a big fan of Barkalow puller, too. It is conceptually simple, very easy to explain, and quite nicely done to be transport independent. Performance sucks, but that is not Dan's fault.

If we are talking about a dumb HTTP server that has packed and then prune-packed its repository, "not a single worthless one" is asking for moon. If Jeff packed all his 50 branches into a single pack and prune packed his repository, the only thing a dumb server could do when you ask for one of his branches is to give you that statically prepared single pack which contains everything, because there would be nothing in .git/objects/??/. You need some CGI support that pulls only needed objects out of that pack and talks a moral equivalent of the upload-pack protocol for that.

Barkalow puller is still useful when all the objects you still need to pull from the remote are unpacked on the remote end. That's how I resurrected "git clone" over http with packed dumb servers. For "clone" case, I just slurp all the available packs, and have Barkalow puller take over the rest.

I have an early WIP for "git fetch", but I have backburnered it for quite some time. I'll push it in its current form into my proposed updates branch, so interested people can hack on it.

I hope nobody starts another SCM project called CZ ;-).