git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[Help][Bug] Git pull fails while pulling large commits on FUSE based filesystem

From
PNPalaniappan Nagarajan <palaniappan.feb22@gmail.com>
Date
Feb 1, 2023, 22:56 UTC
Message-ID
<CAPsey7=9y=j3s1gmO5sFbZXfD_X9osEJ8i-x=SdRM0e27Pbtpg@mail.gmail.com>

Hi Community, We have a peculiar environment: We use git repository on a directory mounted using s3-fuse(https://github.com/s3fs-fuse/s3fs-fuse). The directory is backed up to the Amazon S3 bucket. Whenever we try to run `git pull` that could pull in *large commits*, we are facing the following error.

$ git pull origin main remote: Enumerating objects: 102, done. remote: Counting objects: 100% (102/102), done. remote: Compressing objects: 100% (101/101), done. remote: Total 102 (delta 1), reused 102 (delta 1), pack-reused 0 Receiving objects: 100% (102/102), 72.24 MiB | 15.48 MiB/s, done. Resolving deltas: 100% (1/1), completed with 1 local object. fatal: Failed to checksum '.git/objects/pack/tmp_pack_VvTvzo': No such file or directory fatal: fetch-pack: invalid index-pack output

I can confirm that the file '.git/objects/pack/tmp_pack_XXXXX' is present once the command completes. In the S3 fuse logs, I see that the file is read before the file is created.

The issue happens only during `git pull` and `git clone` with the large commits works without any issues.

Can anyone help with any pointers/ workaround/ potential solution to fix this issue? Is there a way to override the location of temporary files "tmp_pack_XXXXX" files so that it can be created outside the mount directory? Setting the object directory outside the mount directory using GIT_OBJECT_DIRECTORY is not an option as we want to backup git history.

P.S. My experiment that could throw some light on the issue.
1. Created a Git repository with 1000 files of 1 MB.
(https://github.com/cyn0/2kfiles-large)
2. Git clone on the mounted directory works without any issue.
3. Added 5 files of each 1MB size to the repository.
(https://github.com/cyn0/2kfiles-large/commit/42aa5d83e9035460ffa9c3b2f4494c0828f6b1c7)
4. From the mounted directory, "git pull origin main" works without any size.
5. Now, add 100 files of each 1MB to the repository.
(https://github.com/cyn0/2kfiles-large/commit/7304886116d9bdec8a8c2da020776995a6624bd2)
6. From the mounted directory, "git pull origin main" fails with the
above error.
Thanks in advance for your help!

Regards, Palaniappan

Message 1 of 1 in “[Help][Bug] Git pull fails while pulling large commits on FUSE based filesystem”
  1. Palaniappan NagarajanFeb 1, 2023

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.