git log -z doesn't separate commits with NULs

> Hello all.
>
> I wanted to generate several files with some statistics using "git log
> -z" command.
> I did something like this:
> git log -z --patch HEAD~10..HEAD -- SomePathHere | xargs -0
> --max-chars=1000000 ~/1.sh
>
> If I put
> echo "started"
> into the file ~/1.sh I see that the file is called only once instead of
> multiple times.
>
> I'm newbie to xargs, thus I tested with and that worked as I expected.
> find . -type f -print0 | xargs -0 ./1.sh
> That produced a lost of "started" lines.
>
> Thus I suspect there is a but in git log -z command and that doesn't
> "Separate the commits with NULs instead of with new newlines." as
> promised in the documents.
> Is my understanding correct or I don't understand the documentation or
> somehow pass wrong parameters into git log?

>
> Thank you.
> Best regards,
> Nikolaj
> --
> To unsubscribe from this list: send the line "unsubscribe git" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html

> >Thus I suspect there is a but in git log -z command and that doesn't
> >"Separate the commits with NULs instead of with new newlines." as
> >promised in the documents.
> >Is my understanding correct or I don't understand the documentation or
> >somehow pass wrong parameters into git log?
> 
> Just a guess, but doesn't the "--patch" option to git log ask it to
> produce a patch output? Surely that will override the -z: patch will
> not be expecting NULs.

> I wanted to generate several files with some statistics using "git log
> -z" command.
> I did something like this:
> git log -z --patch HEAD~10..HEAD -- SomePathHere | xargs -0
> --max-chars=1000000 ~/1.sh
> 
> If I put
> echo "started"
> into the file  ~/1.sh I see that the file is called only once instead of
> multiple times.

> Am 23.02.2012 10:14, schrieb Nikolaj Shurkaev:
>> I wanted to generate several files with some statistics using "git log
>> -z" command.
>> I did something like this:
>> git log -z --patch HEAD~10..HEAD -- SomePathHere | xargs -0
>> --max-chars=1000000 ~/1.sh
>>
>> If I put
>> echo "started"
>> into the file  ~/1.sh I see that the file is called only once instead of
>> multiple times.
> That is because xargs calls the program with as many arguments as
> possible, unless directed otherwise. Put this in the script:
>
> 	echo "started $*"
>
> and repeat. Then try this:
>
>   ... | xargs -0 -n 1 ~/1.sh
>
> -- Hannes
>

> I wanted to generate several files with some statistics using "git
> log -z" command.
> I did something like this:
> git log -z --patch HEAD~10..HEAD -- SomePathHere | xargs -0
> --max-chars=1000000 ~/1.sh

> On Thu, Feb 23, 2012 at 12:14:23PM +0300, Nikolaj Shurkaev wrote:
>
>> I wanted to generate several files with some statistics using "git
>> log -z" command.
>> I did something like this:
>> git log -z --patch HEAD~10..HEAD -- SomePathHere | xargs -0
>> --max-chars=1000000 ~/1.sh
> I'm not sure what "1.sh" is expecting to take as input, but that will
> feed entire commits, including their commit message and entire diff, to
> the script on its command line.
>
> That seems like an awkward interface, but we don't really know what your
> script intends to do. Maybe it is worth sharing the contents of the
> script.
>
>> If I put echo "started" into the file  ~/1.sh I see that the file is
>> called only once instead of multiple times.
> Yes. The point of xargs is usually to cram as many arguments into each
> invocation of "1.sh" as possible, splitting into multiple invocations
> only when we hit the argument-list memory limit that the OS imposes.
>
> If you want xargs to give each argument its own invocation of the
> script, use "xargs -n1".
>
>> I'm newbie to xargs, thus I tested with and that worked as I expected.
>> find . -type f -print0 | xargs -0  ./1.sh
>> That produced a lost of "started" lines.
> If you instrument your 1.sh more[1], you will find that is not executing
> once per file, but rather getting a large chunk of files per invocation.
>
> [1] Try adding: echo "got args: $*"
>
>> Thus I suspect there is a but in git log -z command and that doesn't
>> "Separate the commits with NULs instead of with new newlines." as
>> promised in the documents.
> You could verify that assertion by looking at the output. Try piping
> your "git log" command through "cat -A | less". When I try it, I see a
> NUL between each commit (cat -A will show it as "^@").
>
> -Peff
>

> Thank you very much for your tips. They really helped me. I was trying
> to create patches that would affect only some given files or
> folders. By this moment I have the following:
> 
> GeneratePatches.sh
> ---------------------
> #!/bin/bash
> #parameter 1 - <since>..<to>
> #parameter 2 - path to file
> git log -z --reverse --format=email --patch "$1" -- "$2" | xargs
> --null --max-args=1 ./CreatePatchFile.sh
> ---------------------
> 
> and CreatePatchFile.sh
> ---------------------
> #!/bin/bash
> 
> myPatchNumber=$(ls ./*-patch.patch 2>/dev/null | wc -l)
> let "myPatchNumber += 1"
> 
> patchFile="./"$(printf "%04d" $myPatchNumber)"-patch.patch"
> echo "$@" > "$patchFile"
> ---------------------
> 
> I call
> ./GeneratePatches.sh HEAD~3..HEAD SomePath
> and that produces something very similar to what I want.
> 
> Perhaps there is a better way to do that.

> Nikolaj Shurkaev<snnicky@gmail.com>  writes:
>
>> Thank you very much for your tips. They really helped me. I was trying
>> to create patches that would affect only some given files or
>> folders. By this moment I have the following:
>>
>> GeneratePatches.sh
>> ---------------------
>> #!/bin/bash
>> #parameter 1 -<since>..<to>
>> #parameter 2 - path to file
>> git log -z --reverse --format=email --patch "$1" -- "$2" | xargs
>> --null --max-args=1 ./CreatePatchFile.sh
>> ---------------------
>>
>> and CreatePatchFile.sh
>> ---------------------
>> #!/bin/bash
>>
>> myPatchNumber=$(ls ./*-patch.patch 2>/dev/null | wc -l)
>> let "myPatchNumber += 1"
>>
>> patchFile="./"$(printf "%04d" $myPatchNumber)"-patch.patch"
>> echo "$@">  "$patchFile"
>> ---------------------
>>
>> I call
>> ./GeneratePatches.sh HEAD~3..HEAD SomePath
>> and that produces something very similar to what I want.
>>
>> Perhaps there is a better way to do that.
> So what git-format-patch is lacking?
>

> For example there are commits that affect not only files in folder A
> but files in folder B, C and so on.  If I do format-patch that will
> give me nice patches, but there are modifications of folders B, C and
> so on there.  I do not know a way to generate patches via format-patch
> that affect only files in folder A.

> On Thu, Feb 23, 2012 at 04:48:43PM +0300, Nikolaj Shurkaev wrote:
>
> Doesn't:
>
>   git format-patch HEAD~3..HEAD SomePath
>
> do what you want? It is certainly designed to, and it seems to work for
> me.

> I'll propose to put something like this into git documentation
> --------------------------------------------------------------------------------------------
> diff --git a/Documentation/git-format-patch.txt
> b/Documentation/git-format-patch.txt
> index 6ea9be7..63267c6 100644
> --- a/Documentation/git-format-patch.txt
> +++ b/Documentation/git-format-patch.txt
> @@ -23,6 +23,7 @@ SYNOPSIS
> [--cover-letter] [--quiet]
> [<common diff options>]
> [ <since> | <revision range> ]
> + [[\--] <path>...]
> 
> DESCRIPTION
> -----------
> @@ -219,6 +220,12 @@ you can use `--suffix=-patch` to get
> `0001-description-of-my-change-patch`.
> range are always formatted as creation patches, independently
> of this flag.
> 
> +[\--] <path>...::
> + Put in patches only those modifications that affect specified files
> + and folders. It's important to understand that log message of the
> + commit may become inappropriate because some parts of patch may be
> + cut off.
> +

>> DESCRIPTION
>> -----------
>> @@ -219,6 +220,12 @@ you can use `--suffix=-patch` to get
>> `0001-description-of-my-change-patch`.
>> range are always formatted as creation patches, independently
>> of this flag.
>> 
>> +[\--] <path>...::
>> + Put in patches only those modifications that affect specified files
>> + and folders. It's important to understand that log message of the
>> + commit may become inappropriate because some parts of patch may be
>> + cut off.
>> +
>
> I think that text looks OK. But to my mind, it is not that format-patch
> accepts a path parameter, but rather that it takes arbitrary log-like
> arguments.

> The above text is not telling the entire truth, though.
>
> When the command is run with the "--full-diff" option, seleted commits
> will be shown in full.  This is useful for example when you want to pick
> commits that add a new "frotz" driver, which obviously needs to include
> "drivers/frotz/" subdirectory, without missing necessary changes to the
> Makefiles in the higher level (e.g. "drivers/Makefile"), e.g.
> 
> 	git format-patch --full-diff v1.0..v1.1 -- drivers/frotz
> 
> In such a case, "some parts may be cut off" does not make the log message
> inappropriate.

> On the other hand, people often do not use the resulting history of taking
> partial patches (i.e. without --full-diff) and feeding them to "am" as-is.
> The operation is used merely to give them a starting point for working on
> (possibly) an unrelated topic, and the history is further tweaked with
> "rebase -i" or even "commit --amend".  It is not "inappropriate" that the
> log says more than what the patch does in such a use case.  What the log
> says is irrelevant.

> > I don't know how well tested every option is, though, so maybe it's not
> > a good idea to encourage the use of random options.
> 
> I obviously agree and also suspect that the real question is not "how well
> tested" but "if it makes sense".
> 
> I am reasonably sure that over time the options and features that make
> sense in the context of producing something that is useful with "am" have
> been already made to work well, but I also am fairly certain that the
> coverage of the code to explicitly reject options that do not make sense
> in that context would be spotty at best.  For example, did we carefully
> design and implement how format-patch should behave when "-z" is given,
> or does the code happen to do whatever it happens to do?  If the latter,
> did we consider rejecting "-z" when given from the command line and
> implement such safety?

> True. That is also a slightly dangerous thing to do, though, because you
> are omitting full patches in the middle that touch the same paths as the
> patches you include....
> ... So
> perhaps we are better off to refer the user to git-log(1), say that
> commit limiting options in general would work, but be careful with
> sending a partial result.

> > True. That is also a slightly dangerous thing to do, though, because you
> > are omitting full patches in the middle that touch the same paths as the
> > patches you include....
> > ... So
> > perhaps we are better off to refer the user to git-log(1), say that
> > commit limiting options in general would work, but be careful with
> > sending a partial result.
> 
> You seem to have spelled out everything I originally wrote in my reply
> that I later deleted before sending it out, and I think the reason that
> brought you to the three-line conclusion is the same one that made me I
> delete them ;-).

> Jeff King <peff@peff.net> writes:
> 
> > True. That is also a slightly dangerous thing to do, though, because you
> > are omitting full patches in the middle that touch the same paths as the
> > patches you include....
> > ... So
> > perhaps we are better off to refer the user to git-log(1), say that
> > commit limiting options in general would work, but be careful with
> > sending a partial result.
> 
> You seem to have spelled out everything I originally wrote in my reply
> that I later deleted before sending it out, and I think the reason that
> brought you to the three-line conclusion is the same one that made me I
> delete them ;-).
> 
> Using a partial patch essentially has the same risk as cherry-picking a
> commit into different context, and it is a more generic issue that this
> particular manual page should not waste tons of space to teach readers
> about.  I think "Be careful and clueful" is sufficient and the best we can
> do without writing a textbook on distributed software development
> disciplines.

> I did something like this:
> git log -z --patch HEAD~10..HEAD -- SomePathHere | xargs -0 
> --max-chars=1000000 ~/1.sh
>
> If I put
> echo "started"
> into the file  ~/1.sh I see that the file is called only once instead of
> multiple times.

> Nikolaj Shurkaev<snnicky@gmail.com>  writes:
>
>> I did something like this:
>> git log -z --patch HEAD~10..HEAD -- SomePathHere | xargs -0
>> --max-chars=1000000 ~/1.sh
>>
>> If I put
>> echo "started"
>> into the file  ~/1.sh I see that the file is called only once instead of
>> multiple times.
> If you want the command to be called once for each commit you need to
> pass --max-args=1 to xargs.  Otherwise xargs will cumulate the arguments
> until --max-chars is reached, and the command is expected to loop over
> them.
>
> Andreas.
>