threads / patch / 55299

patchgit-p4: fix failed submit by skip non-text data files

Subject: [PATCH] git-p4: fix failed submit by skip non-text data files

## tl;dr

9 messages between Mar 12, 2021 and Jun 29, 2021. Diffs are folded; open one to read it.

replies: 8people: 5as markdown or json

dorgon chang via GitGitGadget· Mar 12, 2021, 07:47 UTC · lore
From: "dorgon.chang" <dorgonman@hotmail.com>
If the submit contain binary files, it will throw exception and stop submit when try to append diff line description.
This commit will skip non-text data files when exception UnicodeDecodeError thrown.
Signed-off-by: dorgon.chang <dorgonman@hotmail.com>
---
    git-p4: fix failed submit by skip non-text data files
    
    git-p4: fix failed submit by skip non-text data files
    
    If the submit contain binary files, it will throw exception and stop
    submit when try to append diff line description.
    
    This commit will skip non-text data files when exception
    UnicodeDecodeError thrown.
    
    I am using git-p4 with UnrealEngine game projects and this fix works for
    me.
    
    Signed-off-by: dorgon.chang dorgonman@hotmail.com
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-977%2Fdorgonman%2Fdorgon%2Ffix_gitp4_get_diff_description-v1
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-977/dorgonman/dorgon/fix_gitp4_get_diff_description-v1
Pull-Request: https://github.com/git/git/pull/977
 git-p4.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
Show changes to git-p4.py +5 −2
diff --git a/git-p4.py b/git-p4.py
index 4433ca53de7e..29a8c202399a 100755
--- a/git-p4.py
+++ b/git-p4.py
@@ -1977,8 +1977,11 @@ def get_diff_description(self, editedFiles, filesToAdd, symlinks):
                 newdiff += "+%s\n" % os.readlink(newFile)
             else:
                 f = open(newFile, "r")
-                for line in f.readlines():
-                    newdiff += "+" + line
+                try:
+                    for line in f.readlines():
+                        newdiff += "+" + line
+                except UnicodeDecodeError:
+                    pass # Fond non-text data
                 f.close()
 
         return (diff + newdiff).replace('\r\n', '\n')

base-commit: d4a392452e292ff924e79ec8458611c0f679d6d4
-- 
gitgitgadget
Simon Hausmann· Jun 17, 2021, 11:18 UTC · re: dorgon chang via GitGitGadget · lore

Re: [PATCH] git-p4: fix failed submit by skip non-text data files

On Fri, Mar 12, 2021 at 07:47:49AM +0000, dorgon chang via GitGitGadget wrote:
Show 7 quoted lines
> From: "dorgon.chang" <dorgonman@hotmail.com>
> 
> If the submit contain binary files, it will throw exception and stop submit when try to append diff line description.
> 
> This commit will skip non-text data files when exception UnicodeDecodeError thrown.
> 
> Signed-off-by: dorgon.chang <dorgonman@hotmail.com>

As suggested on https://github.com/git/git/pull/977#issuecomment-862197824, I'm happy to state that the patch looks good to me. IIRC the diff there is solely for the submit template, so it should only include text. That your patch ensures in what seems an idiomatic way.

Signed-off-by: Simon Hausmann <simon@lst.de>
Simon
Johannes Schindelin· Jun 18, 2021, 13:24 UTC · re: Simon Hausmann · lore

Re: [PATCH] git-p4: fix failed submit by skip non-text data files

Hi Simon,
On Thu, 17 Jun 2021, Simon Hausmann wrote:
Show 14 quoted lines
> On Fri, Mar 12, 2021 at 07:47:49AM +0000, dorgon chang via GitGitGadget wrote:
> > From: "dorgon.chang" <dorgonman@hotmail.com>
> >
> > If the submit contain binary files, it will throw exception and stop submit when try to append diff line description.
> >
> > This commit will skip non-text data files when exception UnicodeDecodeError thrown.
> >
> > Signed-off-by: dorgon.chang <dorgonman@hotmail.com>
>
> As suggested on
> https://github.com/git/git/pull/977#issuecomment-862197824, I'm happy to
> state that the patch looks good to me. IIRC the diff there is solely for
> the submit template, so it should only include text. That your patch
> ensures in what seems an idiomatic way.
Thank you for reviewing and chiming in.
> Signed-off-by: Simon Hausmann <simon@lst.de>

The typical way to record your review is to say `Reviewed-by:`. The `Signed-off-by:` footer is usually used to indicate that you wrote the patch, or that you shepherd it onto the Git mailing list.

Sorry to be so nit-picky...

Thanks, Dscho

Junio C Hamano· Jun 29, 2021, 00:52 UTC · re: Johannes Schindelin · lore

Re: [PATCH] git-p4: fix failed submit by skip non-text data files

Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:
>> ... IIRC the diff there is solely for
>> the submit template, so it should only include text. That your patch
>> ensures in what seems an idiomatic way.

This is a crucial piece of information lacking in the proposed commit log message that would help readers understand why this is a safe change. An updated patch with a better log message would be appreciated.

Show 7 quoted lines
> Thank you for reviewing and chiming in.
>
>> Signed-off-by: Simon Hausmann <simon@lst.de>
>
> The typical way to record your review is to say `Reviewed-by:`. The
> `Signed-off-by:` footer is usually used to indicate that you wrote the
> patch, or that you shepherd it onto the Git mailing list.
Yes to both.

It is unusual to see "reviewed-by" from those whose names do not appear even once in output of "git shortlog --no-merges git-p4.py" on a patch that touches "git-p4.py", but this is a fringe area (compared to the more core-ish part of the system) where people touch to scratch their own itch without staying around for a long haul, so it is understandable that we do not always have resident experts in the area. A review like this is highly appreciated.

Thanks, all.
Simon Hausmann· Jun 18, 2021, 14:54 UTC · re: dorgon chang via GitGitGadget · lore

Re: [PATCH] git-p4: fix failed submit by skip non-text data files

On Fri, Mar 12, 2021 at 07:47:49AM +0000, dorgon chang via GitGitGadget wrote:
Show 22 quoted lines
> From: "dorgon.chang" <dorgonman@hotmail.com>
> 
> If the submit contain binary files, it will throw exception and stop submit when try to append diff line description.
> 
> This commit will skip non-text data files when exception UnicodeDecodeError thrown.
> 
> Signed-off-by: dorgon.chang <dorgonman@hotmail.com>
> ---
>     git-p4: fix failed submit by skip non-text data files
>     
>     git-p4: fix failed submit by skip non-text data files
>     
>     If the submit contain binary files, it will throw exception and stop
>     submit when try to append diff line description.
>     
>     This commit will skip non-text data files when exception
>     UnicodeDecodeError thrown.
>     
>     I am using git-p4 with UnrealEngine game projects and this fix works for
>     me.
>     
>     Signed-off-by: dorgon.chang dorgonman@hotmail.com

As suggested on https://github.com/git/git/pull/977#issuecomment-862197824, I'm happy to state that the patch looks good to me. IIRC the diff there is solely for the submit template, so it should only include text. That your patch ensures in what seems an idiomatic way.

Reviewed-by: Simon Hausmann <simon@lst.de>
Simon
Junio C Hamano· Jun 19, 2021, 06:47 UTC · re: dorgon chang via GitGitGadget · lore

Re: [PATCH] git-p4: fix failed submit by skip non-text data files

"dorgon chang via GitGitGadget" <gitgitgadget@gmail.com> writes:
> From: "dorgon.chang" <dorgonman@hotmail.com>
>
> If the submit contain binary files, it will throw exception and
> stop submit when try to append diff line description.
OK, that explains how the program fails.
> This commit will skip non-text data files when exception
> UnicodeDecodeError thrown.

If there are changes in aText and aBinary file and you try to submit a cl that contains both changes, you do want changes to both files go together, no? If you skip non-text, does that mean you ignore the changes to aBinary file and submit only the changes to aText file?

I guess my confusion comes from not understanding what you exactly mean by "append diff line description". Whatever that means, if that is purely informational and does not affect what is actually submit in the resulting cl, then the patch would be an improvement. If not, and if for example it loses changes to binary files, then it is merely sweeping the problem under the rug.

In short the explanation of the solution does not build confidence in the readers minds. You'd need to explain why such a skipping is a safe thing to do a bit better.

Even if we assuming that what happens in the loop you threw in try/except block is purely cosmetic and optional thing that does not affect the correct operation of the program or its outcome, I wonder if we can do better. When you get a decode error, you'd have an early part of the change (which could be empty) before you hit the error in newdiff, and that is returned to the caller without any sign that it is a truncated output. I wonder something like

	except UnicodeDecodeError:
		newdiff = '<<new binary file>>'

may be more helpful to the user. Assuming that this is purely for human consumption without affecting the correctness or outcome of the program and we can place pretty much any text there, that is. But because the proposed commit log message does not explain why skipping is safe, I do not know if that assumption holds in the first place.

Thanks.
Show 15 quoted lines
> diff --git a/git-p4.py b/git-p4.py
> index 4433ca53de7e..29a8c202399a 100755
> --- a/git-p4.py
> +++ b/git-p4.py
> @@ -1977,8 +1977,11 @@ def get_diff_description(self, editedFiles, filesToAdd, symlinks):
>                  newdiff += "+%s\n" % os.readlink(newFile)
>              else:
>                  f = open(newFile, "r")
> -                for line in f.readlines():
> -                    newdiff += "+" + line
> +                try:
> +                    for line in f.readlines():
> +                        newdiff += "+" + line
> +                except UnicodeDecodeError:
> +                    pass # Fond non-text data
s/Fond/Found/ I would think.
Show 5 quoted lines
>                  f.close()
>  
>          return (diff + newdiff).replace('\r\n', '\n')
>
> base-commit: d4a392452e292ff924e79ec8458611c0f679d6d4
dorgon.chang· Jun 20, 2021, 07:56 UTC · re: Junio C Hamano · lore

Re: [PATCH] git-p4: fix failed submit by skip non-text data files

Show 21 quoted lines
> [On the Git mailing list](https://lore.kernel.org/git/xmqq35tel5ad.fsf@gitster.g), Junio C Hamano wrote ([reply to this](https://github.com/gitgitgadget/gitgitgadget/wiki/ReplyToThis)):
> 
> ```
> "dorgon chang via GitGitGadget" <gitgitgadget@gmail.com> writes:
> 
> > From: "dorgon.chang" <dorgonman@hotmail.com>
> >
> > If the submit contain binary files, it will throw exception and
> > stop submit when try to append diff line description.
> 
> OK, that explains how the program fails.
> 
> > This commit will skip non-text data files when exception
> > UnicodeDecodeError thrown.
> 
> If there are changes in aText and aBinary file and you try to submit
> a cl that contains both changes, you do want changes to both files
> go together, no?  If you skip non-text, does that mean you ignore
> the changes to aBinary file and submit only the changes to aText
> file?
> 
Show 7 quoted lines
> I guess my confusion comes from not understanding what you exactly
> mean by "append diff line description".  Whatever that means, if
> that is purely informational and does not affect what is actually
> submit in the resulting cl, then the patch would be an improvement.
> If not, and if for example it loses changes to binary files, then it
> is merely sweeping the problem under the rug.
> 

The skip will not affect actual submit files in the resulting cl, the diff line description will only appear in submit template, so you can review what changed before actully submit to p4.

Show 15 quoted lines
> In short the explanation of the solution does not build confidence
> in the readers minds.  You'd need to explain why such a skipping is
> a safe thing to do a bit better.
> 
> Even if we assuming that what happens in the loop you threw in
> try/except block is purely cosmetic and optional thing that does not
> affect the correct operation of the program or its outcome,  I
> wonder if we can do better.  When you get a decode error, you'd have
> an early part of the change (which could be empty) before you hit
> the error in newdiff, and that is returned to the caller without any
> sign that it is a truncated output.  I wonder something like
> 
> 	except UnicodeDecodeError:
> 		newdiff = '<<new binary file>>'
> 

I don't know if add any message here will be helpful for users, so I choose to just skip binary content, since it already append filename previously.

Show 6 quoted lines
> may be more helpful to the user.  Assuming that this is purely for
> human consumption without affecting the correctness or outcome of
> the program and we can place pretty much any text there, that is.
> But because the proposed commit log message does not explain why
> skipping is safe, I do not know if that assumption holds in the
> first place.
Show 20 quoted lines
> Thanks.
> 
> > diff --git a/git-p4.py b/git-p4.py
> > index 4433ca53de7e..29a8c202399a 100755
> > --- a/git-p4.py
> > +++ b/git-p4.py
> > @@ -1977,8 +1977,11 @@ def get_diff_description(self, editedFiles, filesToAdd, symlinks):
> >                  newdiff += "+%s\n" % os.readlink(newFile)
> >              else:
> >                  f = open(newFile, "r")
> > -                for line in f.readlines():
> > -                    newdiff += "+" + line
> > +                try:
> > +                    for line in f.readlines():
> > +                        newdiff += "+" + line
> > +                except UnicodeDecodeError:
> > +                    pass # Fond non-text data
> 
> s/Fond/Found/ I would think.
>
Just fixed the typo, thanks.
Junio C Hamano· Jun 21, 2021, 03:43 UTC · re: dorgon.chang · lore

Re: [PATCH] git-p4: fix failed submit by skip non-text data files

"dorgon.chang" <dorgon.chang@gmail.com> writes:
Show 6 quoted lines
> The skip  will not affect actual submit files in the resulting cl,
> the diff line description will only appear in submit template, 
> so you can review what changed before actully submit to p4.
> ...
> I don't know if add any message here will be helpful for users, 
> so I choose to just skip binary content, since it already append filename previously. 
These are both good things to write in the proposed log message.
Thanks.
dorgon chang via GitGitGadget· Jun 21, 2021, 05:16 UTC · re: dorgon chang via GitGitGadget · lore

[PATCH v2] git-p4: fix failed submit by skip non-text data files

From: "dorgon.chang" <dorgonman@hotmail.com>
If the submit contain binary files, it will throw exception and stop submit when try to append diff line description.
This commit will skip non-text data files when exception UnicodeDecodeError thrown.

The skip will not affect actual submit files in the resulting cl, the diff line description will only appear in submit template, so you can review what changed before actully submit to p4.

I don't know if add any message here will be helpful for users, so I choose to just skip binary content, since it already append filename previously.

Signed-off-by: dorgon.chang <dorgonman@hotmail.com>
---
    git-p4: fix failed submit by skip non-text data files
    
    git-p4: fix failed submit by skip non-text data files
    
    If the submit contain binary files, it will throw exception and stop
    submit when try to append diff line description.
    
    This commit will skip non-text data files when exception
    UnicodeDecodeError thrown.
    
    I am using git-p4 with UnrealEngine game projects and this fix works for
    me.
    
    Signed-off-by: dorgon.chang dorgonman@hotmail.com
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-977%2Fdorgonman%2Fdorgon%2Ffix_gitp4_get_diff_description-v2
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-977/dorgonman/dorgon/fix_gitp4_get_diff_description-v2
Pull-Request: https://github.com/git/git/pull/977
Range-diff vs v1:
 1:  19b59f40b183 ! 1:  606729bda112 git-p4: fix failed submit by skip non-text data files
     @@ Commit message
      
          This commit will skip non-text data files when exception UnicodeDecodeError thrown.
      
     +    The skip will not affect actual submit files in the resulting cl,
     +    the diff line description will only appear in submit template,
     +    so you can review what changed before actully submit to p4.
     +
     +    I don't know if add any message here will be helpful for users,
     +    so I choose to just skip binary content, since it already append filename previously.
     +
          Signed-off-by: dorgon.chang <dorgonman@hotmail.com>
      
       ## git-p4.py ##
     @@ git-p4.py: def get_diff_description(self, editedFiles, filesToAdd, symlinks):
      +                    for line in f.readlines():
      +                        newdiff += "+" + line
      +                except UnicodeDecodeError:
     -+                    pass # Fond non-text data
     ++                    pass # Found non-text data and skip, since diff description should only include text
                       f.close()
       
               return (diff + newdiff).replace('\r\n', '\n')
 git-p4.py | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
Show changes to git-p4.py +5 −2
diff --git a/git-p4.py b/git-p4.py
index 4433ca53de7e..dc1f46351845 100755
--- a/git-p4.py
+++ b/git-p4.py
@@ -1977,8 +1977,11 @@ def get_diff_description(self, editedFiles, filesToAdd, symlinks):
                 newdiff += "+%s\n" % os.readlink(newFile)
             else:
                 f = open(newFile, "r")
-                for line in f.readlines():
-                    newdiff += "+" + line
+                try:
+                    for line in f.readlines():
+                        newdiff += "+" + line
+                except UnicodeDecodeError:
+                    pass # Found non-text data and skip, since diff description should only include text
                 f.close()
 
         return (diff + newdiff).replace('\r\n', '\n')

base-commit: d4a392452e292ff924e79ec8458611c0f679d6d4
-- 
gitgitgadget

← back to recent threads