Does "git push" open a pack for read before closing it?

[Major ignorance alert]

I'm writing software to implement a FUSE mount for a decentralised file system and during testing with git I see some strange behaviour which I'd like to investigate. It might be a bug in my code, or even the FUSE lib I'm using, or it might be intended behaviour by git.

So one thing I'd like to do is check if this is expected in git.

SYSTEM
OS: Ubuntu 18.10
git version 2.19.1
Decentralised storage mounted at ~/SAFE

What I'm doing I'm testing my FUSE implementation for SAFE Network while exploring the use of git with decentralised storage, so not necessarily in a sensible arrangement (comments on that also welcome).

I have a folder at ~/SAFE/_public/tests/data1/ and want to create a bare repo there to use as a remote from my local drive for an existing git repo at ~/src/safe/sjs.git

Anyway, I do the following sequence of commands which are all fine up until the last one which eventually fails:

  cd ~/SAFE/_public/tests/data1
  git init --bare blah
  cd ~/src/safe/sjs.git
  git remote remove origin
  git remote add origin ~/SAFE/_public/tests/data1/blah
  git push origin master

Here's the output from the last command above:

Enumerating objects: 373, done.
Counting objects: 100% (373/373), done.
Delta compression using up to 8 threads
Compressing objects: 100% (371/371), done.
Writing objects: 100% (373/373), 187.96 KiB | 33.00 KiB/s, done.
Total 373 (delta 254), reused 0 (delta 0)
remote: fatal: unable to open
/home/mrh/SAFE/_public/tests/data1/blah/./objects/incoming-73lbb6/pack/tmp_pack_pL28kQ:
Remote I/O error
error: remote unpack failed: index-pack abnormal exit
To /home/mrh/SAFE/_public/tests/data1/blah
 ! [remote rejected] master -> master (unpacker error)
error: failed to push some refs to '/home/mrh/SAFE/_public/tests/data1/blah'

Inspecting the logs from my FUSE implementation I see that there's a problem related to this file on the mounted storage:

 /_public/tests/data1/blah/objects/incoming-73lbb6/pack/tmp_pack_pL28kQ

Prior to the error the file is written to multiple times by git - all good (about 200kB in all). Then, before the file is closed I see an attempt to open it for read, which fails. The failure is because I don't support read on a file that is open for write yet, and I'm not sure if that is sensible or what git might be expecting to do given the file has not even been flushed to disk at this point.

So I'd like to know if this is expected behaviour by git (or where to look to find out), and if it is expected, then what might git expect to do if the file were opened successfully?

N.B. After the failure, the file is closed and then deleted!

Also note that it is possible the behaviour I'm seeing is not really git but another issue, such as a bug in the sync/async aspect of my code.

Thanks

Mark

On Fri, Dec 21, 2018 at 12:46:35PM +0000, git-mailinglist@happybeing.com wrote:

What I expect is happening is that Git receives the objects and writes them to a temporary file (which you see in "objects/incoming") and then they're passed to either git unpack-objects or git index-pack, which then attempts to read it.

This behavior is expected. POSIX says that a read that can be proved to have occurred after a write must contain the new data, so it's possible that a separate process may choose to read the file and index it, knowing that the index process was started after all the writes.

This is definitely an important invariant to preserve if your FUSE file system is going to be used on a Unix system. In other words, consistency (in the CAP sense) is required.

Right, if this had succeeded, we would have renamed it into place (or unpacked it and deleted it), but since it failed, we clean up after ourselves so as not to leave large temporary files around.

On 22/12/2018 23:12, brian m. carlson wrote: Thanks Brian, you helped me make some progress. I'm stuck again trying to understand git behaviour though and wondering if there are better ways of me seeing into git (source, debug o/p etc) than posting here.

As a reminder, I'm doing the following to create a bare repository on my FUSE mounted decentralised storage:

  cd ~/SAFE/_public/tests/data1
  git init --bare blah
  cd ~/src/safe/sjs.git
  git remote remove origin
  git remote add origin ~/SAFE/_public/tests/data1/blah
  git push origin master

The bugs are in my implementation of FUSE on the SAFE storage.

I get additional output from git using the following (but it doesn't
help me):
 set -x; GIT_TRACE=2 GIT_CURL_VERBOSE=2 GIT_TRACE_PERFORMANCE=2 \
 GIT_TRACE_PACK_ACCESS=2 GIT_TRACE_PACKET=2 GIT_TRACE_PACKFILE=2 \
 GIT_TRACE_SETUP=2 GIT_TRACE_SHALLOW=2 git push origin master -v -v \
 2>&1 |tee ~/git-trace.log; set +x

Anyway, to add a little to your observations...

The git console output seems to confirm it is 'git index-pack' that encounters the error, which is currently:

  Enumerating objects: 373, done.
  Counting objects: 100% (373/373), done.
  Delta compression using up to 8 threads
  Compressing objects: 100% (371/371), done.
  Writing objects: 100% (373/373), 192.43 KiB | 54.00 KiB/s, done.
  Total 373 (delta 255), reused 0 (delta 0)
  remote: fatal: premature end of pack file, 36 bytes missing
  remote: fatal: premature end of pack file, 65 bytes missing
  error: remote unpack failed: index-pack abnormal exit
  To /home/mrh/SAFE/_public/tests/data1/blah
   ! [remote rejected] master -> master (unpacker error)
  error: failed to push some refs to
'/home/mrh/SAFE/_public/tests/data/blah'

So I conclude I'm either not writing the file properly, or not reading it back properly. I can continue looking into that of course, but looking at the file requests I'm curious about what git is doing and how to learn more about it as it looks odd.

I have quite a few questions, but will focus on just the point at which it bails out. In summary, what I see is:

- The pack file is created and written with multiple calls, ending up
about 200k long.

- While still open for write, it is opened *four* times, so git has five
handles active on it. One write and four read.

- At this point I see the following FUSE read operation:

  read('/_public/tests/data1/blah/objects/incoming-quFPHB
        /pack/tmp_pack_E4ea92', 58, buf, 4096, 16384)

  58 is the file handle, 4096 the length of buf, and 16384 the position

- Presumably this is where git encounters a problem because it then
closes everything and cleans up the incoming directory.

It seems odd to me that it is starting to read the pack file at position 16384 rather than at 0 (or at 12 after the header). I can surmise it might open it four times to speed access, but would expect to see it read the beginning of the file (or at position 12) before trying to interpret the content and bailing out.

So I'm wondering what git is doing there. Any comments on this, or a pointer to the relevant git code so I can look myself would be great.

Thanks,

Mak