threads / patch / 39813

patchgit_open_noatime: return with errno=0 on success

Subject: [PATCH] git_open_noatime: return with errno=0 on success

## tl;dr

7 messages between Jul 8, 2015 and Aug 5, 2015. Diffs are folded; open one to read it.

replies: 6people: 4as markdown or json

Clemens Buchacher· Jul 8, 2015, 12:38 UTC · lore

In read_sha1_file_extended we die if read_object fails with a fatal error. We detect a fatal error if errno is non-zero and is not ENOENT. If the object could not be read because it does not exist, this is not considered a fatal error and we want to return NULL.

Somewhere down the line, read_object calls git_open_noatime to open a pack index file, for example. We first try open with O_NOATIME. If O_NOATIME fails with EPERM, we retry without O_NOATIME. When the second open succeeds, errno is however still set to EPERM from the first attemt. When we finally determine that the object does not exist, read_object returns NULL and read_sha1_file_extended dies with a fatal error:

    fatal: failed to read object <sha1>: Operation not permitted

Fix this by resetting errno to zero before we call open again.

Cc: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Clemens Buchacher <clemens.buchacher@intel.com>
Helped-by: Martin Schröder <martin.h.schroeder@intel.com>
---
 sha1_file.c | 1 +
 1 file changed, 1 insertion(+)

Show changes to sha1_file.c +1 −0

diff --git a/sha1_file.c b/sha1_file.c
index 77cd81d..62b7ad6 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1453,6 +1453,7 @@ int git_open_noatime(const char *name)
 	static int sha1_file_open_flag = O_NOATIME;
 
 	for (;;) {
+		errno = 0;
 		int fd = open(name, O_RDONLY | sha1_file_open_flag);
 		if (fd >= 0)
 			return fd;

-- 
1.9.4

Eric Sunshine· Jul 8, 2015, 18:51 UTC · re: Clemens Buchacher · lore

Re: [PATCH] git_open_noatime: return with errno=0 on success

On Wed, Jul 8, 2015 at 8:38 AM, Clemens Buchacher <clemens.buchacher@intel.com> wrote:

Show 10 quoted lines

> In read_sha1_file_extended we die if read_object fails with a fatal
> error. We detect a fatal error if errno is non-zero and is not
> ENOENT. If the object could not be read because it does not exist,
> this is not considered a fatal error and we want to return NULL.
>
> Somewhere down the line, read_object calls git_open_noatime to open
> a pack index file, for example. We first try open with O_NOATIME.
> If O_NOATIME fails with EPERM, we retry without O_NOATIME. When the
> second open succeeds, errno is however still set to EPERM from the
> first attemt. When we finally determine that the object does not

s/attemt/attempt/

Show 10 quoted lines

> exist, read_object returns NULL and read_sha1_file_extended dies
> with a fatal error:
>
>     fatal: failed to read object <sha1>: Operation not permitted
>
> Fix this by resetting errno to zero before we call open again.
>
> Cc: Linus Torvalds <torvalds@linux-foundation.org>
> Signed-off-by: Clemens Buchacher <clemens.buchacher@intel.com>
> Helped-by: Martin Schröder <martin.h.schroeder@intel.com>

Clemens Buchacher· Aug 4, 2015, 08:24 UTC · re: Eric Sunshine · lore

Somewhere down the line, read_object calls git_open_noatime to open a pack index file, for example. We first try open with O_NOATIME. If O_NOATIME fails with EPERM, we retry without O_NOATIME. When the second open succeeds, errno is however still set to EPERM from the first attempt. When we finally determine that the object does not exist, read_object returns NULL and read_sha1_file_extended dies with a fatal error:

    fatal: failed to read object <sha1>: Operation not permitted

Fix this by resetting errno to zero before we call open again.

Cc: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Clemens Buchacher <clemens.buchacher@intel.com>
---

This is a re-submission without changes except for a typo fix in the comments (thanks Eric). The original submission received no other comments, but I think it is a clear improvement and I hope it was just missed the first time.

Best regards, Clemens

 sha1_file.c | 1 +
 1 file changed, 1 insertion(+)

Show changes to sha1_file.c +1 −0

diff --git a/sha1_file.c b/sha1_file.c
index 77cd81d..62b7ad6 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1453,6 +1453,7 @@ int git_open_noatime(const char *name)
 	static int sha1_file_open_flag = O_NOATIME;
 
 	for (;;) {
+		errno = 0;
 		int fd = open(name, O_RDONLY | sha1_file_open_flag);
 		if (fd >= 0)
 			return fd;

-- 
1.9.4

Junio C Hamano· Aug 4, 2015, 21:03 UTC · re: Clemens Buchacher · lore

Re: [PATCH] git_open_noatime: return with errno=0 on success

Clemens Buchacher <clemens.buchacher@intel.com> writes:

Show 10 quoted lines

> diff --git a/sha1_file.c b/sha1_file.c
> index 77cd81d..62b7ad6 100644
> --- a/sha1_file.c
> +++ b/sha1_file.c
> @@ -1453,6 +1453,7 @@ int git_open_noatime(const char *name)
>  	static int sha1_file_open_flag = O_NOATIME;
>  
>  	for (;;) {
> +		errno = 0;
>  		int fd = open(name, O_RDONLY | sha1_file_open_flag);

Please avoid decl-after-stmt, which this codebase does not accept.

>  		if (fd >= 0)
>  			return fd;

More importantly, is this the right place to clear errno?

I would agree it is a good idea to clear it after seeing the first open fail due to lack of O_NOATIME before trying open for the second time, iow, more like this?

 sha1_file.c | 1 +
 1 file changed, 1 insertion(+)

Show changes to sha1_file.c +1 −0

diff --git a/sha1_file.c b/sha1_file.c
index 1cee438..bf2f229 100644
--- a/sha1_file.c
+++ b/sha1_file.c
@@ -1467,6 +1467,7 @@ int git_open_noatime(const char *name)
 
 		/* Might the failure be due to O_NOATIME? */
 		if (errno != ENOENT && sha1_file_open_flag) {
+			errno = 0;
 			sha1_file_open_flag = 0;
 			continue;
 		}

Linus Torvalds· Aug 5, 2015, 08:59 UTC · re: Junio C Hamano · lore

Re: [PATCH] git_open_noatime: return with errno=0 on success

On Tue, Aug 4, 2015 at 11:03 PM, Junio C Hamano <gitster@pobox.com> wrote:

>
> I would agree it is a good idea to clear it after seeing the first
> open fail due to lack of O_NOATIME before trying open for the second
> time, iow, more like this?

So I don't think this is _wrong_ per se, but I think the deeper issue is that somebody cares about 'errno' here in the first place.

A stale 'errno' generally shouldn't matter, because we either

 (a) return success (and nobody should look at errno)

or

 (b) return an error later, without setting errno for that _later_ error.

and I think either of those two situations are the real bug, and this "clear stale errno" is just a workaround.

But as mentioned, I don't think clearign errno is wrong, so I'm not objecting to the patch. I just suspect there's something else goign on too..

              Linus

Clemens Buchacher· Aug 5, 2015, 14:36 UTC · re: Linus Torvalds · lore

Re: [PATCH] git_open_noatime: return with errno=0 on success

On Wed, Aug 05, 2015 at 10:59:09AM +0200, Linus Torvalds wrote:

Show 5 quoted lines

> On Tue, Aug 4, 2015 at 11:03 PM, Junio C Hamano <gitster@pobox.com> wrote:
> >
> > I would agree it is a good idea to clear it after seeing the first
> > open fail due to lack of O_NOATIME before trying open for the second
> > time, iow, more like this?

Looks good to me.

Show 13 quoted lines

> So I don't think this is _wrong_ per se, but I think the deeper issue
> is that somebody cares about 'errno' here in the first place.
> 
> A stale 'errno' generally shouldn't matter, because we either
> 
>  (a) return success (and nobody should look at errno)
> 
> or
> 
>  (b) return an error later, without setting errno for that _later_ error.
> 
> and I think either of those two situations are the real bug, and this
> "clear stale errno" is just a workaround.

I agree. But I do not see how to get there easily.

We are trying to read an object. We first try to read from a pack. We may encounter broken pack files, missing index files, unreadable files, but those errors are not necessarily fatal since we may still be able to read the object from the next pack file or from a sha1 file.

If finally we do not find the object anywhere, in read_sha1_file_extended we try our best to die with an appropriate error message, for example by looking at errno, and otherwise we just return NULL. Most callers seem to die explicitly or they dereference the null pointer.

I think we should instead output error messages closer to the source, like for example in map_sha1_file, but continue anyway. In particular we should immediately report failures due to EPERM or unexpected ENOENT. In the end we may return NULL without another message, but at least the user should have some hints about what went wrong along the way.

Junio C Hamano· Aug 5, 2015, 16:27 UTC · re: Clemens Buchacher · lore

Re: [PATCH] git_open_noatime: return with errno=0 on success

Clemens Buchacher <clemens.buchacher@intel.com> writes:

Show 28 quoted lines

> On Wed, Aug 05, 2015 at 10:59:09AM +0200, Linus Torvalds wrote:
> ...
>> A stale 'errno' generally shouldn't matter, because we either
>> 
>>  (a) return success (and nobody should look at errno)
>> 
>> or
>> 
>>  (b) return an error later, without setting errno for that _later_ error.
>> 
>> and I think either of those two situations are the real bug, and this
>> "clear stale errno" is just a workaround.
>
> I agree. But I do not see how to get there easily.
>
> We are trying to read an object. We first try to read from a pack. We
> may encounter broken pack files, missing index files, unreadable files,
> but those errors are not necessarily fatal since we may still be able to
> read the object from the next pack file or from a sha1 file.
>
> If finally we do not find the object anywhere, in
> read_sha1_file_extended we try our best to die with an appropriate error
> message, for example by looking at errno, and otherwise we just return
> NULL. Most callers seem to die explicitly or they dereference the null
> pointer.
>
> I think we should instead output error messages closer to the source,
> like for example in map_sha1_file, but continue anyway.

Hmm, if we find one data source unreadable but an alternative usable, do we really want that error message? What should it say? "error: cannot read from pack"? Such a message, unless we later give "info: but we managed to read it from elsewhere" and make sure these two messages are clearly associated with each other, would make things unnecessarily alarming, wouldn't it?

Perhaps we should not rely so heavily on 'errno', but explicitly pass around error code (or enough information to formulate an intelligent message at the end) in the callchain instead.

Then the earlier part can notice EPERM on a pack, for example, and return to the caller, and after consulting an alternate data source (e.g. loose object file), the caller can then choose to say "we managed to read the data, but FYI, you may want to check the permission bits of this pack", or choose to stay silent.