Difficulties of scripting git

Hello, git.

I want to write a bash script (more precisely, correct an existing script) which uses git, and it is DIFFICULT!!!

I need a git pull in the script. Fine, but how does the script know if it's worked or not? I couldn't find any description of a return code in the git-pull man page (or in the git-merge man page).

The big problem is when I have modified, uncommitted files in the target repo, and those same files are to be updated in commits coming from the source repo. Sadly, git is unable to merge these changes. It just fails, putting an error message onto stderr, but doesn't tell the calling script in any way that I can see.

One idea would be always to call git stash before doing the pull, then git stash pop afterwards. Trouble is, git stash is unreliable - it doesn't always add a new stash entry, so the stash pop at the end would sometimes/often pop off an entry it shouldn't. git stash doesn't have a --force argument. git stash doesn't set a result code, either, that I can see. One way around this would be to do

    $ git stash list | wc -l

both before and after the git stash and compare the answers, but really?

So, next idea, feed the output from git status --porcelain through grep before and after the git pull, so as to find out whether there are any modified files before the git pull (thus making a stash necessary) and any files with conflicts after the git stash pop. Shouldn't be too difficult.

Except, how does one recognise a file with conflicts from this git status output? The man page says that

    "   For paths with merge conflicts, `X' and `Y' show the modification
    states of each side of the merge.  For paths that do not have merge
    conflicts, `X' shows the status of the index, and `Y' shows the status
    of the work tree.  For untracked paths, `XY' are `??'.  Other status
    codes can be interpreted as follows:  ...."

I've spent nearly an hour trying to make sense of this bit of man page. How is one meant to distinguish an XY of a merge conflict from the XY of an index/work tree entry? I can't find that key bit of information anywhere.

What am I missing? Why does writing scripts using git have to be so hard?

On 2021-01-09 at 19:14:13, Alan Mackenzie wrote:

git stash doesn't consider there being nothing to stash to be an error, so it doesn't set a nonzero error code. Think of it as, "I want a clean working tree," not, "I want to create a stash." In that sense, what you wanted is already done, so it's not an error.

You are, however, not the only person who's been unhappy with this behavior. A --force option may be a useful option to have if someone wanted to create it.

Using "git status --porcelain" before and after the stash is an easy way to find out whether there was anything to stash. If there's a difference, then a stash was created.

In the chart, the options in the first section are "normal" cases that occur without a merge conflict. The second section is things that happen when there's a conflict (the "unmerged" states).

So what you're looking for to find conflicts is something maybe a little like this:

  git status --porcelain | grep -E '^(AA|DD|[AUD]U|U[AUD])'

If you believe that you're likely to need to handle names with newlines, then of course you'll need -z as well.

I do agree that this is a little unclear; I went ahead and simulated a merge conflict to verify. I'll try to send a patch making the documentation reflect what I mentioned about the chart above.

The table describing the porcelain format in git-status(1) is helpful, but it's not completely clear what the three sections mean, even to some contributors. As a result, users are unable to find how to detect common cases like merge conflicts programmatically.

Let's improve this situation by describing what each section means: non-conflicted, conflicted, or untracked files.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
 Documentation/git-status.txt | 4 ++++
 1 file changed, 4 insertions(+)

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

I agree that the addition clarifies, but it is a bit sad that we already have a beginning of the explanation; I wonder if we should improve the existing description in addition, even if it may not be sufficient to eliminate the need for this new paragraph. Here is what we already have:

    For paths with merge conflicts, `X` and `Y` show the modification
    states of each side of the merge. For paths that do not have merge
    conflicts, `X` shows the status of the index, and `Y` shows the status
    of the work tree.  For untracked paths, `XY` are `??`.  Other status
    codes can be interpreted as follows:

This introductory text does sort-of hint that there are three classes (merged paths, unmerged paths and untracked paths), but (1) the order the three classes are described do not match that of the table, and (2) the explanation of the untracked paths predates the addition of ignored ones to the untracked class, so the description is added after the legends as if an afterthought.

I am actually tempted to suggest rewriting the whole section, starting from the paragraph above and ending at the table, with something like this:

    Three different classes of paths are shown in the same format,
    but the meaning of `XY` are different:

    * For merged paths, `X` shows the status of the index, and `Y`
      shows the status of the working tree.

    * For unmerged paths, `X` and `Y` show the modification states
      of each side of the merge, relative to the common ancestor.

    * For untracked paths, `X` and `Y` do not convey different
      meaning (as, by definition, they are not known to the index);
      `??` is shown for untracked paths, and when `--ignored` option
      is in effect, ignored paths are shown with `!!`.

    In the following table, these three classes are shown in
    separate sections, and these characters are used for `X` and `Y`
    fields for the first two sections that show tracked paths:

    * ' ' = unmodified
    * 'M' = modified

       ...

    ....
    X        Y       Meaning
    ------------------------------------------------------------
            [AMD]    not updated

       ...

Hmm?

On 2021-01-10 at 01:41:23, Junio C Hamano wrote:

Sure, I can reroll with that. I noticed that we're using a text diagram instead of a table, so maybe I can fix that up as well in v2, depending how the output looks.

Hello, Junio and Brian.

Thanks for such rapid and helpful responses to my post.

On Sat, Jan 09, 2021 at 17:41:23 -0800, Junio C Hamano wrote:

Another problem I tripped over was that word "Other", which is ambiguous. For quite some time I read it wrongly as meaning "Other than the codes already described in this paragraph" rather than its intended meaning "Other than for untracked paths". So I was searching through the man page looking for a description of codes for conflicts.

When you amend this documentation, would you please consider using the term "conflicts" instead of, or as well as "unmerged". "Conflicts" is something definite and clear - "unmerged" feels a bit vague and indefinite, to me at any rate. Why might a file be "unmerged" still?

Also missing from the current doc, I think, is a description of which "side" of the difference is represented by X, and which by Y. In my use case (having conflicts after doing a git stash pop) those "sides" would be the work tree and the repository. In other scenarios (say a merge conflict after git pull) I think they would be something else (though my head is hurting a bit, here).

I would appreciate such a new formulation.  Thanks!

On 2021-01-10 at 12:28:40, Alan Mackenzie wrote:

The two sides are the two heads that are being merged and don't include the working tree at all. I think the answer about which side is which in a merge depends on whether you're merging or rebasing. Rebases do things the opposite way of how merges do them.

In your case, you're interested in the fact that there _is_ an unresolved conflict, which means you'll either get DD, AA, or something with a U in it. That will happen regardless of how you do it, with git pull, git merge, git rebase -m, or git cherry-pick, and it indicates that the working tree is "broken" (i.e., has conflict markers) and the index is in a conflicted state. Since the state of the working tree and the index are known, we use this field for listing the two heads instead.

The table describing the porcelain format in git-status(1) is helpful, but it's not completely clear what the three sections mean, even to some contributors. As a result, users are unable to find how to detect common cases like merge conflicts programmatically.

Let's improve this situation by rephrasing to be more explicit about what each of the sections in the table means, to tell users in plain language which cases are occurring, and to describe what "unmerged" means.

Signed-off-by: brian m. carlson <sandals@crustytoothpaste.net>
---
This uses text from Junio's email, so his sign-off will be required
here.  I assume that won't be a problem, but I can send a v3 if it is.

 Documentation/git-status.txt | 28 ++++++++++++++++++++--------
 1 file changed, 20 insertions(+), 8 deletions(-)

"brian m. carlson" <sandals@crustytoothpaste.net> writes:

Actually I find the text so vastly improved from "here is my attempt" version in the discussion that, other than the paragraph structure, there is nothing left that I can call my contribution. I'll sign the resulting commit off anyway, though ;-)

Even if rebase uses the good-old "format-patch | am -3" pipeline, it would result in an index with entries at higher stages. So I am not sure if this "Note that" helps the reader.

This iteration has improved the "Three different classes of paths are shown" in the "here is my attempt" version to "Three different types of states ..."; shouldn't we be doing the same here with s/classes/types of states/?

Thanks.

On 2021-01-11 at 20:22:10, Junio C Hamano wrote:

Well, I did use some of your text verbatim, though, and the sign-off is necessary. Not that it's easily distinguishable in our history since you're the maintainer, but I also like to be meticulous about sign-offs because I think it makes sure that people get due credit for their contributions.

I'll rephrase.

Can do.