Re: Does "git push" open a pack for read before closing it?

From: brian m. carlson <sandals@crustytoothpaste.net>
Date: Dec 22, 2018, 23:12 UTC
Message-ID: <20181222231215.GC26554@genre.crustytoothpaste.net>
In-Reply-To: <5878633d-cdbd-ce05-30de-3417386c720e@happybeing.com>

On Fri, Dec 21, 2018 at 12:46:35PM +0000, git-mailinglist@happybeing.com wrote:

Show 27 quoted lines

> Here's the output from the last command above:
> 
> Enumerating objects: 373, done.
> Counting objects: 100% (373/373), done.
> Delta compression using up to 8 threads
> Compressing objects: 100% (371/371), done.
> Writing objects: 100% (373/373), 187.96 KiB | 33.00 KiB/s, done.
> Total 373 (delta 254), reused 0 (delta 0)
> remote: fatal: unable to open
> /home/mrh/SAFE/_public/tests/data1/blah/./objects/incoming-73lbb6/pack/tmp_pack_pL28kQ:
> Remote I/O error
> error: remote unpack failed: index-pack abnormal exit
> To /home/mrh/SAFE/_public/tests/data1/blah
>  ! [remote rejected] master -> master (unpacker error)
> error: failed to push some refs to '/home/mrh/SAFE/_public/tests/data1/blah'
> 
> Inspecting the logs from my FUSE implementation I see that there's a
> problem related to this file on the mounted storage:
> 
>  /_public/tests/data1/blah/objects/incoming-73lbb6/pack/tmp_pack_pL28kQ
> 
> Prior to the error the file is written to multiple times by git - all
> good (about 200kB in all). Then, before the file is closed I see an
> attempt to open it for read, which fails. The failure is because I don't
> support read on a file that is open for write yet, and I'm not sure if
> that is sensible or what git might be expecting to do given the file has
> not even been flushed to disk at this point.

What I expect is happening is that Git receives the objects and writes them to a temporary file (which you see in "objects/incoming") and then they're passed to either git unpack-objects or git index-pack, which then attempts to read it.

> So I'd like to know if this is expected behaviour by git (or where to
> look to find out), and if it is expected, then what might git expect to
> do if the file were opened successfully?

This behavior is expected. POSIX says that a read that can be proved to have occurred after a write must contain the new data, so it's possible that a separate process may choose to read the file and index it, knowing that the index process was started after all the writes.

This is definitely an important invariant to preserve if your FUSE file system is going to be used on a Unix system. In other words, consistency (in the CAP sense) is required.

> N.B. After the failure, the file is closed and then deleted!

Right, if this had succeeded, we would have renamed it into place (or unpacked it and deleted it), but since it failed, we clean up after ourselves so as not to leave large temporary files around.

-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204