If you would write git from scratch now, what would you change?

If you would write git from scratch now, from the beginning, without concerns for backwards compatibility, what would you change, or what would you want to have changed?

Yes, I know, I know. "Worse is better". It is better to release early and get feedback what is really needed, as opposed to what do you think is needed.

I think git is a wonderful example of "evolved" software, evolving practically from the very beginnings.

On Sun, Nov 25, 2007 at 09:48:27PM +0000, Jakub Narebski wrote:

  * reset/checkout/revert. The commands to wonderful things, but this UI
    is a mess for the newcomer.

  * pull/fetch/push: I would have had pull being what fetch is, and
    added some --merge option to actually "do the obvious merge". But
    pull encourage "bad" behavior from the user, and confuses newcomers
    a lot.

  * I would have hidden plumbing more, using a really distinguished
    namespace (stupid example, there are probably better ways, but we
    could have git-_rev-parse or git-plumbing-rev-parse instead of
    git-rev-parse) so that it's clear to the user that those are really
    internal commands, and that he doesn't need to understand them.

    This is a big issue with git: the list of commands of git is the top
    of the iceberg from the UI point of view. People _feel_ they are
    comfortable with a tool if they get say 75% of the UI. I don't say
    it's true that understanding 75% of the UI makes you a $tool expert,
    but it's how people feel it. With git, 75% of the commands (and
    don't get me started with the options ;P) is a _lot_. bzr is way
    better at that game: there are at least as many commands, but those
    are completely hidden to the user.

    Of course having our guts easy to grok and find is a big advantage
    for the git gurus. But for the newcomer it's a disconcerting.

  There is probably more things I'd change, but those were the first UI
rumblings from me :)

On Sun, Nov 25, 2007 at 11:23:14PM +0100, Pierre Habouzit wrote:

Heartily seconded. I think checkout is the most egregrious of the three. git-checkout can be used to:

    * Switch branches
    * Create a branch
    * Change the state of all files to a particular commit
    * Change the state of a particular file to that of the index
    * Change the state of a particular file (and index) to a particular
      commit

To makes things more complicated, several of these tasks can be done with other commands. Short of rewriting git from scratch, what can be done to simplify the many-to-many mapping of tasks to commands?

Steven Walter <stevenrwalter@gmail.com> writes:

Come on. The second one is just to give a short-hand side-effet for commonly used operation and you do not have to use it nor learn it.

Also, you have written the last three in a more confusing way than it is necessary. They are all the same thing but with variations --- your way of writing them is like enumerating "change the state of files whose name starts with A", "change the state of files whose name starts with B", etc. as if they are distinctly different and confusing operations.

Let's clear the confusion. Although it is not bad like the above "random 5 different operations", checkout does serve 2 quite different purposes:

 (1) checkout a revision.

     This primarily affects the notion of where your HEAD is.  Is it
     pointing at a branch, or detached at a particular commit?  In
     either case, the objective from the user's point of view here is "I
     want to change on which commit and/or branch I'd build the next
     commit, if I were to issue git-commit command".

     "I started modifying but realized that I wanted to build not on top
     of master but a separate topic", is a typical use case, and this
     form will let you take your local changes with you exactly for this
     reason.

     Obviously when people say "I checkout this commit", they mean the
     state of the work tree and they mean the whole tree.  It is
     hopefully clear that is what you are doing from the fact that you
     do not give any pathspec to the command to trigger this mode of
     operation.

 (2) checkout selected paths out of a commit (or the index).

     "I screwed up.  I want to start over modifications to these files
     from the state of the previous commit (or the last state I
     staged)." is a typical use case for this mode.  For this reason,
     the named paths are updated in the work tree and the work tree and
     the index are made to match.

     Again, it hopefully is clear enough that you need to give some
     pathspec to it for the operation to make sense, if you understand
     the purpose of the command.  Like "." to mean the whole tree, "*.c"
     to mean all C files, or "directory/" to mean everything underneath
     it.

So yes, it does two quite different things, and that's mostly because the verb "to check out" has overloaded meanings.

Hopefully it is clear which one you are using by thinking about the reason WHY you are "checking out", and by looking at the way you form the command line.

Junio C Hamano wrote:

I think the overwhelming majority of git users learn `git checkout -b`. The cases where you do want to switch to a branch you just created seem far more common than the cases where you don't (particularly for new users), which is the whole reason the -b option exists in the first place. So I don't think it's reasonable to say "you can choose not to be confused by ignoring this incredibly useful command."

Given the above, I'd argue that it serves 3 purposes:

   (1) check out a revision
   (2) check out selected paths out of a commit (or the index)
   (3) start working on a new branch

It's true that (1) and (3) are very closely related, but I think in the minds of many git users (particularly new ones) they are distinct. (2) really seems the most out of place here, and has the most potential for finding a new home (perhaps within git-reset).

-Adam

On Nov 26, 2007 7:36 AM, Adam Roben <aroben@apple.com> wrote:

I think this is mostly due to the idea of a branch as a separated box (like a directory) instead of a line of development like the notion which comes from thinking in a branch as the place where HEAD is pointing to.

Personally, it is always difficult for me to understand git as a whole, because I'm not sure what is the common use case for each command in the most-usual-way-of-doing-the-things when using git, despite of having long and complete documentation for each individual command. The question is if we can give the power of git to their users in the same way they think, or how git could be able to teach their users to think in the way it works.

An idea would be to study (and document) the most successful use cases that git supports and check if it is already providing unique and/or clear commands for them.

--Carlos

On Mon, 26 Nov 2007, Carlos Rica wrote:

I think that part of git's oddity comes from the fact that the UI is organized around use cases rather than commands. That is, for each thing that people commonly do, the sequence of commands is as short as possible and each of the names makes sense in the context of this sequence. But then the commands and options, in the list of commands and options outside of the context of a use case, don't make any sense as a whole.

It's like trying to document the "take" command in a text adventure, where "take [noun]" means to pick it up, "take off [noun]" means to remove it as clothing, and "take off" means to leave.

There's a set of primitive git operations, but the git commands aren't those; the git command schemas (not just the "command" part, but the type of arguments following it) are semi-natural-language interfaces to collections of primitive operations, and are set up to have a core "what the user is saying to do" and all of the reasonable analogous extensions to that. This means that the very same result can often be reached with multiple entirely different commands, because there are different ways of conceptualizing what you're doing that overlap. (E.g., "git checkout HEAD ." will check out the current directory from the current branch, discarding local changes; "git reset --hard HEAD" will move the current branch to its current state, bringing the working copy in line with it as well; both of these have the effect of discarding all local changes while keeping the branch state the same, but that's just because the aspects of the two operations which are different don't matter with those particular arguments)

I think git's UI design is, by and large, very good, but I'm not sure how to document it so as to make it easy to learn, aside from giving a quick explanation of how to use reflogs to recover from mistakes and telling users to just try stuff in their local repository.

	-Daniel
*This .sig left intentionally blank*

Jakub Narebski wrote:

Erm... (it's much harder to come with lists like these lately :-))

 - "index", "cached" and "stage" are a definite source of confusion
 - "git add" and "git rm" would be nicer as "git stage" and "git unstage"
   (or something similar)
 - libgit would have come first
 - "git revert" should be called "git invert"
 - "git revert" would (maybe) be "git reset"
 - "git clone" wouldn't exist
 - "git-gui" would be written in Qt (ducks)
 - git-apply et al wouldn't be a disaster when the log message contains a   
   diff (change to git diff format?)
 - empty directories in the repository (ducks again)

Andy

On Nov 26, 2007, at 5:46 PM, Andy Parkins wrote:

I warmly second that.

But here, I have to disagree. Why would you want to call "git- revert" "git-reset"?

I know it's annoying that commands with the same name do different things in SVN/CVS but I don't think it's a reason to necessarily adapt to them. There are plenty of misnomers already anyway (checkout, commit, add).

While we're discussing bad names, as someone already pointed out, I agree it's sad that "git push" is almost always understood as being the opposite of "git pull".

On Mon, Nov 26, 2007 at 18:10:10 +0100, Benoit Sigoure wrote:

Well, it is an oposite of pull. Compared to it, it is limited in that it will not do a merge and on the other hand extended to *also* be an oposite of fetch, but still oposite of pull is push.

Jan Hudec <bulb@ucw.cz> writes:

With the same reasoning the opposite of a duck is a lobster, since a lobster has not only fewer wings, but also more legs.

On Mon, Nov 26, 2007 at 20:12:37 +0100, David Kastrup wrote:

No.

The basic pull/push actions are:

git pull: Bring the remote ref value here. git push: Put the local ref value there.

Are those not oposites?

Than each command has it's different features on top of this -- pull merges and push can push multiple refs -- but in the basic operation they are oposites.

Jan Hudec writes:

I think that is in absolute agreement with David: Ducks swim on the surface of the water and lobsters swim underneath. Why consider the different features on top of where they swim?

The thing about git-pull that surprises so many users is the merge. There's a separate command to do that step, and git-pull had a fairly good excuse to do the merge before git's 1.5.x remote system was in place, but now the only really defensible reason for its behavior is history.

Michael Poole

On Mon, Nov 26, 2007 at 14:50:35 -0500, Michael Poole wrote:

When I first looked at hg -- and that was long before I looked at git -- I was surprised that their pull did NOT merge and you had to do a separate step. Partly because doing those two steps is quite common.

Jan Hudec writes:

Frequency of use is a good argument for having one command that does both. It is not a good argument that "fetch, then merge" should be called "pull" or is the opposite of "push".

Michael Poole

On 11/26/07, Michael Poole <mdpoole@troilus.org> wrote:

I'm starting to think that things oriented around the default names of master and origin needs rethinking. Everything should use explicitly named remotes. You could always do something like set a default remote repository, but that is different than using the magic name 'origin'.

On Monday 2007, November 26, Benoit Sigoure wrote:

I don't; you're reading it the wrong way around. I think current revert should actually be called invert. "revert" means to move back to a previous point. That is not at all what git-revert does, what it actually does is to apply the opposite of the given commit - i.e. an inversion.

Revert on the other hand would be the perfect name for current git-reset.

I know that very well; nor did I pick any of those suggestions based on what svn does. There is no other VCS that uses "invert" as far as I know. Nor does any other VCS use "revert" the way I'd like; so I'm really not sure where you're getting the idea that I chose those as a copy of another VCS. What I would like (ideally, with a time machine) is those words, which have a well defined meaning in English, to match more closely with their function in the VCS. "revert" is definitely not right.

I know; but the question was "what if we could start again". I don't see too many problems with checkout and commit as it happens. They both seem like adequate verbs to describe the operations they perform. We could perhaps quibble about the extra functionality that they've both gained, but that isn't a naming fault, and the extensions are natural extensions for users of git.

Andy

On Nov 26, 2007 5:46 PM, Andy Parkins <andyparkins@gmail.com> wrote:

But...wait...Qt would require...(I'm scared to say!)... that awful, painful, hopeless thing called C++. Probably you didn't mean what you said ;-)

Marco

Marco Costalba <mcostalba@gmail.com> wrote:

Heh.

I'll never port git-gui to Qt. Because of that awful, painful thing called C++ that it uses. I despise C++. No, please don't start a C++ language war again on the list. :-)

I recently considered porting git-gui to XUL, as nobody has ever said "Firefox isn't native enough on my OS!". It also (maybe) has the benefit of having a large developer base (everyone and their dog has coded in HTML and Javascript before, except maybe Linus).

But XUL doesn't support launching a process and connecting pipes to its stdin and stdout. I started to try and create an XPCOM extension to provide that functionality from NSPR and started to run into major problems compiling the XPCOM plugin, getting the necessary interfaces implemented, etc.

In the end I was able to recreate the bulk of the main git-gui UI in XUL in just an hour or so, but spent days trying to just do a basic thing like "git diff-index --cached -z HEAD" and consume the result. I never even got that to work so I just gave up on the idea.

So git-gui is in Tcl/Tk for the long-term. However I'm going to try and port git-gui over to the Tcl/Tk 8.5 "tiles" extension (if it is available on your system) so we can get better looking native widgets. I'll still fall back to the old style widgets for Tcl/Tk 8.4 so existing users aren't forced to upgrade to 8.5 just to use the latest git-gui. (But really, 8.5 isn't that hard to build and install...)

Shawn O. Pearce wrote:

[git-gui in XUL]

What about Ajax / Comet support in XUL, Can this be used for that? (Just an [perhaps stupid] idea).

Jakub Narebski <jnareb@gmail.com> wrote:

Yes, XUL fully supports AJAX. If it didn't Google Maps and its cool interface wouldn't exist. :-)

The problem there is that AJAX requires HTTP. So I'd have to create a "micro HTTP server" that runs on the loopback interface and listens for HTTP requests from the GUI, parses them, runs the necessary Git action, then sends the results back to the GUI.

Sort of ugly.

My bigger concern is also for a shared machine; how do I secure the HTTP server so only the git-gui process that is supposed to be using it is able to access it? I guess I could create a 600 ~/.gitguicookie file or some such entity and throw random data into it to initialize it. That's basically all xauth is doing.

Actually I might revisit this XUL concept using an HTTP server and AJAX. I could actually link the damn HTTP server against libgit.a (Junio will hate me). If the server dies XUL can notice it and simply restart it. But there's a whole suite of actions that I can run through the internal APIs with high chances of success, and a lot quicker than forking the corresponding plumbing process, especially on fork challenged machines like Windows.

Hi,

On Mon, 26 Nov 2007, Shawn O. Pearce wrote:

But if you can restart the HTTP server via XUL, you can start other git programs directly.

What you'd have to do is (urgh) write a wrapper via start_command() which would recognize that the second process die()d.

All in all, I think if you want to switch from Tcl/Tk to another language for git-gui, for the sake of attracting more developers, it might be wiser to go Java than XUL.

Ciao, Dscho

Johannes Schindelin wrote:

Wont we get with the same problems as egit/jgit?

---- This is proposed set of questions for git-gui mini survey...

1. What language and what toolkit should git-gui be written in?
   (single choice)

   a. Tcl/Tk    (current implementation)
   b. C++/Qt
   c. C/GTK+
   d. Python    (native)
   e. Python/PyQt
   f. Python/PyGTK
   g. Ruby
   h. Java/Swing
   i. Java/SWT
   j. XUL+JavaScript+CSS/XULRunner
   k. other
   l. no opinion

2. If you have chosen "other" in question above, what language and
   toolkit should it be? C/XForms? C#/Mono? C/wxWidgets? XAML+Silverlight?
   GTK2-Perl? C/OpenGL? ;-)

3. Do you contribute to git-gui?
   Yes/No

4. If git-gui would use other language/toolkit, would you contribute?
   Yes/No

5. What languages and what toolkits you are proficient with (to send
   patches)? 
   (multiple choice)

   a. Tcl/Tk    (current implementation)
   b. C++/Qt
   c. C/GTK+
   d. Python    (native)
   e. Python/PyQt
   f. Python/PyGTK
   g. Ruby
   h. Java/Swing
   i. Java/SWT
   j. XUL+JavaScript+CSS/XULRunner
   k. other
   l. N/A

6. What other?

Hi,

On Wed, 28 Nov 2007, Jakub Narebski wrote:

My idea was not to get the same problems, but to use jgit. After all, Shawn made a point of separating the both.

I am pretty comfortable with a), but rather than go [b-gi-l] I would prefer h).

Yes (sort of; not half as much as I'd like to.)

Yes, as long as it is a language/toolkit that is available on all platforms that I (have to) work. That pretty much excludes C# and Python as a language.

[abchk]

Personally, I am quite comfortable with the existing implementation, and IMHO people dismiss contributing to git-gui too easily; Tcl is not all that complicated, and it is not hard at all to change/imitate existing code.

Ciao, Dscho

On Nov 28, 2007 7:32 AM, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

Since we're listing off a bunch of toolkits, I should pitch FLTK, which is well-supported across platforms, reasonably featured, and pretty lightweight (probably much smaller than any of the other ones listed, in terms of dependency installs)

That said...

Agreed. I don't know much about Tcl/Tk, but I think that git-gui is fine as-is. It's not very "pretty" compared to all of the fancy Gtk apps the make up my system, but that's not an obstacle for me. (The fonts are pretty bad, though)

Jason

On Wed, Nov 28, 2007 at 12:32:10 +0000, Johannes Schindelin wrote:

Out of interest, where does neither of those two work and Qt and tcl/tk do? Mono and python both seem to be quite portable.

Hi,

On Thu, 29 Nov 2007, Jan Hudec wrote:

IRIX (an ancient one).

Besides, Mono is darned slow.  Even Tcl/Tk is faster.

Furthermore, my complaint was not about a platform where neither C# nor Python work. That is irrelevant. If you have one platform where only one works, and another platform where only the other works, you cannot have a single program for both platforms. Right?

Hth, Dscho

On Wed, Nov 28, 2007 at 23:48:12 +0000, Johannes Schindelin wrote:

On the shootout Mono seems to be an order of magnitude faster in most tests. But maybe they are performing very poorly on some strange platform where they don't have JIT.

Right.

I probably shouldn't be surprised that mono does not work on older unices, but I am a bit surprised python does not.

Hi,

On Thu, 29 Nov 2007, Jan Hudec wrote:

*Sigh* I managed again to make myself misunderstood.

Even if newer Python does not easily compile on that IRIX, I have an old Python there (2.2). But I don't have any Python on MSys. (Yes, there is a _MinGW_ port, but no _MSys_ one.) So for me, Python is out.

Hth, Dscho

On Thu, Nov 29, 2007 at 12:01:47 +0000, Johannes Schindelin wrote:

While it would be a problem, but is it really fatal? AFAIK MSys uses unixy paths inside the program, but accepts arguments and calls other processes using the windows convention, so mingw python should have no problem calling msys programs and vice versa. It might be more problematic to compile a shared module for it, but .dlls are quite well isolated, so even compiling a plugin linked with msys for mingw python might not be impossible.

Nevertheless, I actually think git-gui is quite well in Tcl/Tk and rewriting it in python nor any other language would probably help it in any way.

On Nov 30, 2007 6:50 PM, Jan Hudec <bulb@ucw.cz> wrote:

A little provocation: I've never seen in open source a discussion on what language to use for an application and then the development of the application from scratch.

What I see daily instead is the effort of one (or a very little number of people) to develop something in the language he choose and then , _after_ some code has been produced, the effort embraced by other people that join the project.

Some near examples? gitk, gitweb, stgit, git itself especially for shell parts (why shell should be a better prototyping language then other prototyping languages? portability? easy to learn? performance? library support? syntax? probably no one of the above in general terms).

I would say this thread, although very interesting from a learning point of view, it's a a little bit academic.

Marco

Jan Hudec <bulb@ucw.cz> wrote:

UNIX (really X11) users think git-gui looks like cr*p on their systems as Tk draws with 1980s widgets, not 2007 style widgets. They have every right to complain about the look and feel of the application, its utter crap. Tk 8.5's tiles extension may help that, but I haven't tried.

On Windows 2000/XP and Mac OS X I think I've gotten git-gui to (almost) fit into the rest of the desktop. It fits into the Windows UI better than it does Mac OS X, there are still some rough edges where it is really obvious its not a native Mac OS X application.

On all platforms Tk has some "features" that are less than desirable. For example it has been an absolute nightmare to get split pane divider things to work on all systems. I can't tell you how many days I spent just getting the main window to not react stupidly on each system. And it *still* doesn't act right everywhere. Sometimes if you resize the window the status bar on the bottom disappears and Tk just clips it right out of the UI (no, I didn't ask it to do that, Tk has bugs).

Building context sensitive menus isn't fun. Managing some data structures in Tcl isn't fun. The list of why I'm currently unhappy with Tcl/Tk for git-gui is actually pretty long.

On Dec 1, 2007 3:35 AM, Shawn O. Pearce <spearce@spearce.org> wrote:

Not to advertise, just my two cents, but Qt with whatever language binding you want to use, it's really powerful, easy to learn, documentation is great, easy to create GUI forms, actually you don't even need to program because Qt Designer let you create a form graphically, the result is a XML like file that a Qt tool called UIC transforms in a compilable file.

Qt library is consistent and complete and very portable, especially Qt4 works and installs under different OS with no hassles. And the Qt community (http://www.qtcentre.org/forum/) is very helpful and supportive.

I really don't want to advertise, but after reading your list of Tcl/Tk cons I was not able to stay quiet.

Marco

Jakub Narebski <jnareb@gmail.com> writes:

[...]

What's this? Tkinter? If so, it's better to be spelled "Python/Tk" here, and probably should be removed anyway as there is no apparent reason to re-implement current Tcl/Tk in Python/Tk.

Anyway, for Python as a language, a realistic choice of GUI seems to be between PyGtk, PyQt, and WxPython.

Marco Costalba wrote:

Actually although I like C++, that's not the reason, the reason is that Qt is a significantly (IMHO) better toolkit than Tk. It's more cross platform and looks a lot nicer. The fact that it's C++ is neither here nor there.

Personally I find these language wars a bit distasteful; to me programming is programming - the language is a purely secondary point.

Andy

On Nov 27, 2007 9:45 AM, Andy Parkins <andyparkins@gmail.com> wrote:

Actually there exist a Python bindings for Qt if you prefer.

I was just joking about C++, never meant to start a "language war" that I personally consider as very very un-useful and very pity.

Marco

On Tue, Nov 27, 2007 at 14:15:21 +0100, Marco Costalba wrote:

I tried to write something in them and got a bit burned. Qt has it's idea of memory management (delete children with parent) and the bindings don't protect from accessing pointers to objects deleted this way, which can cause rather hard to debug crashes.

Gtk seems to be much better for use from various scripting languages.

Hi,

On Tue, 27 Nov 2007, Andy Parkins wrote:

Tcl/Tk was easier to install on a lot more platforms in my life than Qt.

Ciao, Dscho

Johannes Schindelin wrote:

I wasn't really thinking of the install; that's a packaging problem. I was speaking of the toolkit itself. I know what you mean, but I wasn't even thinking of cross-platform in a "number of places it can run" sense. What I meant (although my point is irrelevant and way off the original question) was the facilities available in the toolkit with a cross-platform interface.

Qt puts a common face on threading, process control, networking, file systems, internationalisation, rendering, openGL, and of course the GUI itself. Tcl/Tk (to take the most wicked example) gives you applications that are much harder to make run on Windows than on UNIX.

Anyway, I don't want to sound like a strange Qt fan boy; the above is simply my justification for putting "git-gui in Qt" on my wish list.

Andy

Quoting Andy Parkins <andyparkins@gmail.com>:

Hear, hear.

Not sure it would be that easy. (As I have just learned recently) "git rm" is the opposite of "git add" _only_ in the case of files-not-previously-tracked. And the opposite of "git add <file>" for files-already-being-tracked is "git reset HEAD -- <file>", which is probably where you were going with "git unstage" 8-) .

Why? AFAIC, git clone works out quite well - both functionality and naming wise.

Cheers.

On 11/25/07, Jakub Narebski <jnareb@gmail.com> wrote:

I would sit down and carefully design the command syntax. git's biggest criticism is that it is hard to use and this is mainly caused by the seemingly very complex commands. Much of this complexity could be hidden from the user.

I'd also integrated a patch management system like stgit. I'm using stgit commands for 90% of my tasks and it has a different syntax than git (its trying to fix some of the problems).

Most current git users are knowledgeable programmers and could handle a rework of the git command syntax. The sooner the syntax is reworked the better in my opinion. The current syntax grew organically as we learned what git needed. Now's the time to use this knowledge and design an optimal command structure.

Jakub Narebski <jnareb@gmail.com> writes:

Get rid of plumbing at the command line level. It is confusing to users, and command line arguments, exec calls and I/O streams are not efficient and reasonably typed mechanisms for the kind of operations done in plumbing. Instead using a good extensible portable scripting language (I consider Lua quite suitable in that regard, but it is conceivable that something with a native list type supporting easy sorts, merges and selections could be more efficient) and implementing plumbing in that or in C would have been preferable for creating the porcelain.

That would keep plumbing out of the hair of users and make it easier to cobble together extensions and variations with non-trivial internal dataflow.

Shell scripts have also proven to be a constant hassle with regard to portability and bugs (like underquoting).

On Mon, Nov 26, 2007 at 18:11:43 +0100, David Kastrup wrote:

No, please. It's extremely useful. It should be a bit more hidden, but it's a big advantage of git that the plumbing is available.

POSIX shell is really the best extensible portable scripting language available for the job. Because the whipuptitude is the most important property and shell is simply best at one-liners. And since you use it for regular work (running editor, compiler, git porcelain), it is the obvious choice for whiping up a short function.

On Nov 26, 2007, at 8:27 PM, Jan Hudec wrote:

Perl seems pretty portable. If we had a decent, complete libgit, it would be easy to create bindings for various languages and script Git in other languages than Shell script.

On Mon, Nov 26, 2007 at 21:11:41 +0100, Benoit Sigoure wrote:

Perl might be good for the lower level stuff (and is indeed used for that in git a lot), but most useful tools on top of git gather few bigish bits (contents of whole files and such) and pass them to some application. And this is what shell is really good at.

So yes, more direct interfaces for various languages would certainly be good, but it would never be a full replacement for the process interface. It is most generic and for many hacks the easiest thing to use.

On Mon, 26 Nov 2007, David Kastrup wrote:

We can't get rid of plumbing. It is part of Git probably forever and is really really convenient for scripting in any language you want.

The only valid argument IMHO is the way too large number of Git commands directly available from the cmdline.

The solution: make purely plumbing commands _not_ directly available from the command line. Instead, they can be available through 'git lowlevel <blah>' instead of 'git <blah>' and only 'git lowlevel' would stand in your shell default path.

Such a scheme can be implemented in parallel with the current one for a release while the direct plumbing commands are deprecated in order to give script authors a transition period to fix their code.

Nicolas

Nicolas Pitre <nico@cam.org> writes:

What about "at the command line level" did you not understand?

On Mon, Nov 26, 2007 at 20:34:25 +0100, David Kastrup wrote:

Which part of we neither can nor want did you not understant?

The availability of plumbing is really big part of a reason why git is so good and has so many scripts and tool built on top of it. Bzr and hg boast with their ability to add plugins, but git ability to use plumbing simply beats that hands down, because the plugins are python-only and writing them requires understanding the internal API, while git plumbing can be used from any language and can usually be understood by running it interactively a few times.

That's why we don't want (and really can't because there is a huge amount of code in various languages using it) to get rid of plumbing at the command level. What we may do is hide it from the casual user.

To do that, we'd want to get rid of the git-* commands and links in bin (remove the builtins altogether and move the non-builtin to libexec -- that seems to be the plan for 1.6 or 1.7 already) and than hiding the plumbing from --help and completion hides it from the user.

Jan Hudec <bulb@ucw.cz> writes:

Which is the reason I proposed making the plumbing available at a scripting level, not at the command line level.

The actual trend we are getting nowadays is locking the porcelaine, previously available as shell scripts, down into C code, _without_ making use of a reasonable plumbing layer suitable for any scripting at all.

So the git community at the same time praises shell scripting and simultanouesly replaces it without even using the available plumbing, _and_ claims that _both_, exclusive and incompatible approaches, are the perfect solution. At the same time. While fighting the shell portability fight continuously, on Unix as well as Windows.

I may have a big mouth, but swallowing all of this at once is beyond me.

On Mon, Nov 26, 2007 at 21:35:56 +0100, David Kastrup wrote:

But scripting in the first place means *SHELL* scripting. Or you normally use Lua command line for your daily work?

For myself I would say I don't think C is an appropriate tool for the job. It is nice when you need to optimize things to the last instruction, but for my taste it's unwieldy for the high-level stuff.

Well, the builtins *do* use the plumbing. They just use the C functions without using streams and forks. Isn't that what you wanted?

But the key reason for keeping the plumbing around is prototyping and especially tailoring. Junio has many scripts (you can look at them in the todo branch in git repo) to support his particular workflow and plumbing is useful there. And shell is really the right tool for such things.

On Mon, 26 Nov 2007, David Kastrup wrote:

You're mixing two orthogonal issues, namely: 1) the scripting language, and 2) the too large number of Git command accessible through your default path.

#1 is a non issue really. We don't want to lock plumbing to any particular scripting language, and the current interface is the most universal one in that regard.

#2 can be solved through a single multiplexer such as 'git low-level'.

That 'git low-level foo' may just look up git-foo in some libexec directory, and only 'git-low-level' need to be in the path instead of all those plumbing commands.

Need only to have both forms ('git foo' and 'git low-level foo') to work for a transition period.

Nicolas

El 26/11/2007, a las 20:57, Jan Hudec escribió:

Yes, the plumbing is really lovely when it comes time to whipping together a quick tool for a special task; much nicer than writing a plugin.

For the benefit of newcomers, I just wish the plumbing was kept a little bit out of sight. You know, porcelain in /usr/bin and plumbing in /usr/libexec or other such place.

It's fine once you've learnt your workflows and know the 10 or 15 Git tools that you'll be using day-to-day; but for people who are just starting off this can be a little bit intimidating:

$ git-<tab> Display all 146 possibilities? (y or n)

Cheers, Wincent

Wincent Colaiuta <win@wincent.com> writes:

I'd agree to that but I've always considered this an issue for distros. We've supported an ability for them to specify a gitexecdir separate from /usr/bin in our Makefile for almost two years.

The tab completion for bash and zsh would also help you here, but I see there are quite a few commands that should not be there, and it's time to clean it up.

	$ git <tab>
        add                   fetch                 push
        am                    filter-branch         rebase
        annotate              format-patch          rebase--interactive
        apply                 fsck                  relink
        archive               gc                    remote
        bisect                get-tar-commit-id     repack
        blame                 grep                  request-pull
        branch                gui                   reset
        bundle                imap-send             resolve
        checkout              init                  revert
        checkout-index        instaweb              rm
        cherry                less                  send-email
        cherry-pick           lg                    shortlog
        citool                log                   show
        clean                 lost-found            show-branch
        clone                 ls-files              show-ref
        co                    ls-remote             stash
        commit                ls-tree               status
        config                merge                 submodule
        convert-objects       mergetool             svnimport
        count-objects         mv                    tag
        describe              name-rev              var
        diff                  pickaxe               verify-pack
        diff-stages           pull                  whatchanged

Perhaps this list can be a starting point...

 contrib/completion/git-completion.bash |    9 +++++++++
 1 files changed, 9 insertions(+), 0 deletions(-)

On Mon, 26 Nov 2007, Junio C Hamano wrote:

Would probably be a good thing to start enforcing that by default. It's easier to follow such policies when they're coordinated from the project origin.

Nicolas

Nicolas Pitre <nico@cam.org> writes:

Not really. The project origin ships the Makefile to install under $HOME, but I do not see any distros following that.

On Mon, 26 Nov 2007, Junio C Hamano wrote:

What about the default RPM spec file?

Nicolas

Junio C Hamano <gitster@pobox.com> wrote:

...

Ack'd-by: Shawn O. Pearce <spearce@spearce.org>

;-)

"Shawn O. Pearce" <spearce@spearce.org> writes:

Seriously, speaking I find this "negative" list ugly. I am wondering if it makes more sense to use positive "Porcelain" list, or perhaps even "The most commonly used" list from "git help" output.

Here is an alternate attempt to it.

---

 Documentation/cmd-list.perl            |   59 ++++++++++++------------
 Makefile                               |    2 +-
 contrib/completion/git-completion.bash |   77 ++------------------------------
 generate-cmdlist.sh                    |   34 ++------------
 4 files changed, 40 insertions(+), 132 deletions(-)

On Nov 26, 2007, at 7:35 PM, Junio C Hamano wrote:

Yes, a positive list makes much more sense. If for no other reason than that it will require the author of a new command to make a conscious decision before that command will be suggested to users.

-Steve

Hi,

On Mon, 26 Nov 2007, Nicolas Pitre wrote:

I agree, but that's not even the complete truth. Git would be not even half as useful as it is without its scriptability.

So it is not only convenience, but very much a reason that git development is so fast. That, and that more people than elsewhere let code talk. Which is also much easier when you have a scriptable system.

Ciao, Dscho

On Mon, 26 Nov 2007, Johannes Schindelin wrote:

Sure, but this is missing the point.

The issue at hand is about the fact that way too many Git commands are to be found in the default command path. Diverging on whether or not plumbing is useful is the wrong question.

Nicolas

Hi,

On Mon, 26 Nov 2007, Nicolas Pitre wrote:

Ah, thanks. I use a spam filter here, so I did not get the complete context.

Sorry, Dscho

Nicolas Pitre wrote:

The "git-cmd" form of writing commands was deemed obsolete round about the time git.sh was rewritten in C. There's just no reason for it anymore.

It's unfortunate that git-sh-setup makes it equally valid for scripts to use either form, as we can never get rid of the dashed form when so many scripts in the core distribution uses it.

Ah well.

Andreas Ericsson wrote:

I think it would be enough to have "git" and perhaps "git-sh-setup" in PATH, and the rest of git-cmd in EXEC_PATH != PATH.

On Nov 25, 2007 1:48 PM, Jakub Narebski <jnareb@gmail.com> wrote:

Currently data can be quickly copied from pack to pack,
but data cannot be quickly copied blob->pack or pack->blob
(there was an alternate blob format that supported this,
 but it was deprecated).  Using the pack format for blobs
would fix this.  It would also mean blobs wouldn't need to
be uncompressed to get the blob type or size I believe.

So far this has prevented me from deploying git here (and is half the reason I have not been active recently). Currently we use p4 and we have large files. When a large file is checked in (submitted), it is compressed *once* and sent over the network -- these are the only delays that end-users experience.

The equivalent operation in git would require the creation of the blob, and then of a temporary pack to send to the server. This requires 3 calls to zlib for each blob, which for very large files is not acceptable at my site.

Yes, git has much better features. But 80%+ of my workgroup will not use them, and only notice that git is "slower".

Thanks,

On Mon, 26 Nov 2007, Dana How wrote:

I don't see why you would need the pack->blob copy normally.

Then you can do just that for big enough blobs where "big enough" is configurable: encapsulate them in a pack instead of a loose object. Problem solved. Sure you'll end up with a bunch of packs containing only one blob object, but given that those blobs are so large to be a problem in your work flow when written out as loose objects, then they certainly must be few enough not to cause an explosion in the number of packs.

They already don't.

I currently count 2 calls to zlib, not 3. And with big blobs as packs, as suggested above then you'd have only one call when actually staging their content. This should be really straight forward to implement given that pack-objects is already a built-in.

Nicolas

On Nov 26, 2007 11:52 AM, Nicolas Pitre <nico@cam.org> wrote:

True,  but that doesn't change the main point.

Are you suggesting that "git add" create a new pack containing one blob when the blob is big enough? Re-using (part of) the pack format in a blob (or maybe only some blobs) seems like less code change.

It looks like sha1_file.c:parse_sha1_header() works on a buffer filled in by sha1_file.c:unpack_sha1_header() by calling inflate(), right?

It is true you don't have to uncompress the *entire* blob.

I count 3:

Call 1: git-add calls zlib to make the blob.

Call 2: builtin-pack-objects.c:write_one() calls sha1_file.c:read_sha1_file() calls :unpack_sha1_file() calls :unpack_sha1_{header,rest}() calls inflate() to get the data from the blob into a buffer.

Call 3: Then write_one() calls deflate to make the new buffer to write into the pack. This is all under the "if (!to_reuse) {" path, which is active when packing a blob.

Remember, I'm comparing "p4 submit file" to "git add file"/"git commit"/"git push", which is the comparison the users will be making.

On the other hand, I'm looking at code from June; but I haven't noticed big changes since then on the list.

Calls 2 and 3 go away if the blob and pack formats were more similar.

On Mon, 26 Nov 2007, Dana How wrote:

Sure, but let's not go overboard either.

Exactly.

Don't know what you mean exactly here, but what I mean is to do something as simple as:

	pretend_sha1_file(...);
	add_object_entry(...);
	write_pack_file();

when the buffer to make a blob from is larger than a configured treshold.

Right.  Only the first 16 bytes or so need to be uncompressed.

Oh, you're right.  Somehow I didn't count the needed decompression.

... which my suggestion should provide with a minimum of changes, maybe less than 10 lines of code.

Nicolas

On Nov 26, 2007 12:55 PM, Nicolas Pitre <nico@cam.org> wrote:

I will think about your suggestion (and the number of packs that might result), but I confess I am surprised by it.

When I proposed automatically extracting large blobs from source packs when creating a new pack under a blob size limit while pack-objects was running, you objected on the grounds that pack-objects only creates packs and should not create blobs (this proposal had other problems too, but this is the one you didn't like).

Now it's OK for git-add to sometimes create packs instead of blobs? I would not have predicted that!

;-)

On Mon, 26 Nov 2007, Dana How wrote:

Going back to loose objects from packs is indeed something I object to if it becomes part of a work flow. Objects should move from the loose space towards the packed space and not the other way around. Sure there is fetch.unpackLimit, but with the auto-repack recently added to Git this variable could probably be set even lower.

But having a pack created for huge blobs up front has many advantages, the most obvious is the fact that later repack can combine and/or send those single-blob packs with almost no cost.

Loose objects are meant to be blazingly fast to create. Once repacked they have no advantage being loose again. Obviously when your blob is huge you won't benefit much from a loose object.

Nicolas

On Mon, 26 Nov 2007, Dana How wrote:

Could you do some benchmark for repository with your large objects as loose objects created with and without core.legacyHeaders (created with git pre 1.5.3), and as single blob packs, perhaps kept, with _undocumented_ (except for RelNotes) gitattribute delta unset for those files?

  - We used to have core.legacyheaders configuration, when
    set to false, allowed git to write loose objects in a format
    that mimicks the format used by objects stored in packs.  It
    turns out that this was not so useful.  Although we will
    continue to read objects written in that format, we do not
    honor that configuration anymore and create loose objects in
    the legacy/traditional format.

  - "pack-objects" honors "delta" attribute set in
    .gitattributes.  It does not attempt to deltify blobs that
    come from paths with delta attribute set to false.

  - diff-delta code that is used for packing has been improved
    to work better on big files.

The last part is thanks to your comments, complaints and efforts, Dana.

On Nov 26, 2007 12:17 PM, Jakub Narebski <jnareb@gmail.com> wrote:

First of all, this is a very reasonable request and what I should be doing. Unfortunately, I only have the cycles at the moment to point out this issue, which appears to be a problem from my perspective.

Currently, a user who wants to publish some (large) files does the following: git add (calls deflate) git commit git push (builds a pack to stdout, calling inflate and deflate on each blob).

So if the blob and pack formats were more similar (different blob format, big blobs are singleton packs, etc) the zlib calls in git push go away. The deflate call could be sped up by using 1 for compression level, but it still takes time.

Another "solution" is to make each workgroup member's .git/objects
be a symlink to a tree with a lot of sticky bits and do some scripting.
(This means "git push" doesn't push any data and only alters stuff
 in .git/refs/heads on the server.)
I'm not entirely enthusiastic about this,  and when I mentioned it a while
ago it did cause some retching...

Yes,  there have been some very useful improvements recently.

However, I didn't actually push for the first "-" you list; I was pushing for the "mimic" option even then but some argument was presented to me against it, to which I had no counter-argument until I understood git better later.

Thanks,

Dana How <danahow@gmail.com> wrote:

I agree with Nico's comment that you probably don't need pack->loose object as its just not something you want to do. But otherwise above you mean "loose->pack" or "pack->loose" as blob is one type of loose object but there are others (tree, commit, tag).

The alternate format for loose objects *was* the packfile format, but without the packfile header or trailer as that was really quite unnecessary for a single object storage.

Unfortunately we removed that alternate format from the system. We can't create it anymore. We can't efficiently copy it to the packfile anymore. But we can still read it in case someone still has loose objects using that alternate format in their repository.

I was sad when Nico removed the format in 726f852b0ed7e. I can understand why he did so but I think it was a move in the wrong direction.

On Mon, 26 Nov 2007, Shawn O. Pearce wrote:

What I'm suggesting, though, is to actually create a real pack for those blobs where the recompression is really an issue. all the code is there and only needs to be called.

In most usage cases, though, the proportion of blobs that gets copied directly into a pack is minimal, and even then they don't amount to a lot of cycles compared to the majority of deltified objects.

(yeah, "deltified" is said to be wrong by some, but it is really 
 convenient a word.)

I wish I could convince you otherwise by now.

Nicolas

Jakub Narebski <jnareb@gmail.com> wrote:

- Sort tree entries by name, *not* by name+type

  This has got to be my biggest gripe with Git.  I think Linus really
  screwed the pooch with this.  We've talked it over a few times
  on the list and he and I have just agreed to disagree on this.

  Ask any database person and they'll tell you how wrong the
  current tree ordering is.  Or they are nuts and don't get
  the concept of data integrity.

  Linus' excuse is that the current ordering makes working with
  the flat index faster as its just one index file.  That doesn't
  mean that the flat index file can't contain tree information.
  Like it does in say that new fangled cache-tree extension.  :-)

  This particular "design decision" has brought all sorts of bugs
  into the system, like the D/F merge conflict issues, and even one
  from Linus himself when he first introduced the submodule support.
  Lets not even talk about ugly that made things in jgit.

- Loose objects storage is difficult to work with

  The standard loose object format of DEFLATE("$type $size\0$data")
  makes it harder to work with as you need to inflate at least
  part of the object just to see what the hell it is or how big
  its final output buffer needs to be.

  It also makes it very hard to stream into a packfile if you have
  determined its not worth creating a delta for the object (or no
  suitable delta base is available).

  The new (now deprecated) loose object format that was based on
  the packfile header format simplified this and made it much
  easier to work with.

- No proper libgit

  Already been stated but we don't have a great library and we
  don't have a good way to build one right now either.  A lot of
  our internal code assumes die() will abort the process.  That's a
  very bad assumption to be making inside of a library.

- Binary packed-refs representation

  I probably wouldn't have done an ASCII based packed-refs file,
  or heck, even loose refs.  I probably would have just gone with
  a binary file that we wholesale rewrite every time there is any
  sort of ref update.

  We already do this with the index.  So every time we update a
  file path we are rewriting the entire index.  And we update
  file paths a heck of a lot more often than we update branch
  heads.  Or tags.

  But tools like for-each-ref get invoked heavily, and fast access
  to the ref database is important to overall performance.

- No GIT_OBJECT_DIRECTORY vs. GIT_DIR distinction

  This is causing problems with $GIT_DIR/objects/info/alternates
  and then try to repack repositories.  Not having the ref space of
  the alternates and/or borrowers considered during repacking can
  cause all sorts of fun breakage that may be hard to recover from.
  Plus it means you have to do funny "refs/forkee" hacks just to
  avoid pushing unnecessary objects over the wire when the other
  end is borrowing objects.

  I probably would have had the object directory unified with its
  ref database, so that they cannot be accessed individually.

All of the above is written with 20/20 hindsight and all that.

Looking back (and knowing myself well) I think the only item I would have gotten right if I had written Git from scratch is the first one above (the tree entry ordering). I probably would have done something equally "as bad" as what we have today for all of the others...

"Shawn O. Pearce" <spearce@spearce.org> writes:

... not to mention countless others you would get wrong that you did not list in the above, as the current git got them right ;-)

Junio C Hamano <gitster@pobox.com> wrote:

Indeed.

Which is why nobody is looking to rewrite Git from scratch.

Except myself and a few other nuts who want a pure Java implementation for Eclipse plugins. :-)

Shawn O. Pearce wrote:

And the project to implement git in C# / Mono (I wonder what is the status of those implementations...)

Hi,

On Tue, 27 Nov 2007, Jakub Narebski wrote:

See for yourself. I started listing some plumbings in http://git.or.cz/gitwiki/Plumbings, but it seems that the homepage points to a wrong URL for the repo. The correct one is http://repo.or.cz/w/Widgit.git.

Hth, Dscho

On Mon, 26 Nov 2007, Shawn O. Pearce wrote:

It is a bit cumbersome indeed, but I'm afraid we're really stuck with it since every object SHA1 depends on that format.

Not really. Since separate zlib compression levels for loose objects and packed objects were introduced, there was a bunch of correctness issues. What do you do when both compression levels are different? Sometimes ignore them, sometimes not? Because the default loose object compression level is about speed and the default pack compression level is about good space reduction, the correct thing to do by default would have been to always decompress and recompress anyway when copying an otherwise unmodified loose object into a pack.

Nicolas

On Nov 26, 2007 8:58 PM, Nicolas Pitre <nico@cam.org> wrote:

Yes, now I remember: this was the same argument you used to convince me that losing the "new" (deprecated) loose format was OK.

However, if we changed WRITE(DEFLATE(SHA1("$type $size\0$data"))) (where SHA1(x) = x but has the side-effect of updating the SHA-1) to WRITE($pack_style_object_header) SHA1("$type $size\0") WRITE(DEFLATE(SHA1($data))) then the SHA-1 result is the same but we get the pack-style header, and blobs can be sucked straight into packs when not deltified. The SHA-1 result is still usable at the end to rename the temporary loose object file (and put it in the correct xx subdirectory).

Because we can't change the SHA-1 result we unfortunately can never drop the 2nd call above [this is something that could have been different, to respond to the email that started this thread]. You didn't like the duplication between the 1st and 2nd call, but I can't say I see that as a big deal.

Not exactly. I did think about this. When you are packing to stdout, and only sending the resulting packfile locally, you don't want to bother with recompressing everything. [This is the "workgroup" case that concerns me.] Other cases, sure, recompression could help (e.g., packing to a file means the file will probably be around for a while, so you want to recompress if the levels are unequal; and you probably want to recompress as well if the packfile will be sent over a "slow" link).

Thanks,

Dana How <danahow@gmail.com> wrote:

Hah.  That's exactly what the "new" (deprecated) format was, and what
its code for creating such objects looked like in sha1_file.c. :-)
 

On Mon, 26 Nov 2007, Nicolas Pitre wrote:

No. 

The SHA1 itself just depends on "$type $size\0$data" (no deflate phase), and that one is easy and cheap to calculate. How we then *encode* the data on disk is totally immaterial.

In fact, pack-files obviously do not encode it in that form at all, they in fact use two different forms of "$binaryhdr$DEFLATE($data)" or "$binaryhdr$basesha$DEFLATE($delta)" (that's from memory, so don't rely on that).

So we could easily change the on-disk format, and we obviously have - the alternate (but deprecated) format for unpacked objects already did. In fact, we could - and probably should - add some kind of "back end interface" for alternate encoding formats, in case somebody wants to do something really crazy like use a database for object tracking.

(Side note: using an actual database would really be insane. There is absoluely zero point. But what *could* be interesting would be to have a "cluster back-end" for the git object store, where objects get hashed to different nodes. If you have a really fast network, it may actually be beneficial to spread the objects out, and get better disk throughput by that kind of strange "git object RAID-0 striping" setup)

		Linus

(*) Honesty in advertising: the really *original* format did the SHA1 after the deflate, but that was quickly fixed and was a really stupid choice. The main point for doing that was that it meant that loose objects could be verified by just running "sha1sum" on them, and comparing the result with their name.