{"thread":{"id":"60941","subject":"Git commit causes data download in partial clone","startedAt":"2024-02-17T20:38:36Z","lastAt":"2024-02-22T19:34:16Z","messageCount":3,"participants":["charmocc","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"488851","messageId":"OK9E_kNDYqB1tDn6YJhtTgkdDDrcr2LhZEGRdmqismu6KyTki-M22CpCAxXHZCn45SZICjnPNYxvw02BnWeJic3mx47-zeI0HDhzdgoJpG0=@proton.me","threadId":"60941","inReplyTo":null,"subject":"Git commit causes data download in partial clone","fromName":"charmocc","fromEmail":"charmocc@proton.me","sentAt":"2024-02-17T20:38:08Z","receivedAt":"2024-02-17T20:38:36Z","isPatch":false,"sender":{"key":"charmocc@proton.me","avatar":null},"body":"Hi everyone,\n\nI was recently exploring git partial clone feature because I wanted to\ncontribute to repository which has a lot of binary files. My intent was to only\nadd new files without modifying any existing ones and to download as few data\nas possible in the process. Here are the steps I followed:\n\n$ git clone --no-checkout --filter=blob:none https://github.com/libretro-thumbnails/Nintendo_-_Nintendo_Entertainment_System.git nes\n$ cd nes\n$ echo foo > bar\n$ git add bar\n$ git commit bar # causes git fetch behind the scene and download of a lot of objects!\n\nNow for reasons I don't understand the last command cause download of a lot of\nobjects from remote (blobs) which is what I was trying to avoid. By enabling\ntracing options I can see that it runs fetch operation in the background:\n\ngit -c fetch.negotiationAlgorithm=noop fetch origin --no-tags --no-write-fetch-head --recurse-submodules=no --filter=blob:none --stdin\n\nUnfortunately trace doesn't show what refspec is being used for the fetch \noperation. The regular \"git fetch\" doesn't cause any additional object \ndownloads.\n\nIs there something I'm doing wrong here or maybe what I'm trying to achieve \n(contribute to repository without downloading all of its files) is simply \nnot possible?\n\ngit version 2.34.1 (Ubuntu 22.04)\n\nThanks for your time\ncharmocc\n"},{"id":"488952","messageId":"20240220024310.GD2713741@coredump.intra.peff.net","threadId":"60941","inReplyTo":"OK9E_kNDYqB1tDn6YJhtTgkdDDrcr2LhZEGRdmqismu6KyTki-M22CpCAxXHZCn45SZICjnPNYxvw02BnWeJic3mx47-zeI0HDhzdgoJpG0=@proton.me","subject":"Re: Git commit causes data download in partial clone","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-02-20T02:43:10Z","receivedAt":"2024-02-20T02:43:12Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Feb 17, 2024 at 08:38:08PM +0000, charmocc wrote:\n\n> I was recently exploring git partial clone feature because I wanted to\n> contribute to repository which has a lot of binary files. My intent was to only\n> add new files without modifying any existing ones and to download as few data\n> as possible in the process. Here are the steps I followed:\n> \n> $ git clone --no-checkout --filter=blob:none https://github.com/libretro-thumbnails/Nintendo_-_Nintendo_Entertainment_System.git nes\n> $ cd nes\n> $ echo foo > bar\n> $ git add bar\n> $ git commit bar # causes git fetch behind the scene and download of a lot of objects!\n> \n> Now for reasons I don't understand the last command cause download of a lot of\n> objects from remote (blobs) which is what I was trying to avoid. By enabling\n> tracing options I can see that it runs fetch operation in the background:\n\nI think what is happening is something like:\n\n  1. You clone with --no-checkout, so you do not fetch any of the blobs.\n     But you also have an empty index, with no entries at all.\n\n  2. Running \"git commit\" is going to need all of those entries in the\n     index (to compute the hash of the new tree). So it will read it\n     from the tree of the current HEAD.\n\n  3. When we load entries into the index, the usual next thing to do is\n     to check them out. So rather than fetch them one by one as we do\n     the actual checkout, the index-reading code collects all of the\n     entries we don't have and then does a single fetch for them. This\n     is prefetch_cache_entries() in read-cache.c.\n\nNow obviously in your example, the \"usual\" thing is not happening; we do\nnot intend to write those entries into the working tree, so fetching\nthem is pointless.\n\nThere may be some room for improvement here. E.g., teaching the\nindex-reading code a flag that says \"don't bother prefetching\", and use\nit in this call chain. I'm not sure if there would be other gotchas,\nthough.\n\nBut here are a few alternatives that you can try without making any code\nchanges:\n\n  a. Your --no-checkout skips the checkout, but it does not tell Git\n     that you are fundamentally uninterested in those other paths. To do\n     that, you can try the sparse-checkout mechanism. I'm not super\n     familiar with the feature myself, but doing:\n\n       git clone --sparse --filter=blob:none $url nes\n\n     ends up with an empty checkout to which you can add things (the\n     trick is that we do have all of those index entries, but they are\n     marked as \"not interesting\").\n\n     Do note that --sparse checks out the contents of the top-level tree\n     by default. That's OK for your repo (all of the files are in the\n     Named_Titles directory), but it might not be true for some other\n     repos (it may also not work if your intent is to put another entry\n     into Named_Titles, though it looks like you might just need to say\n     \"git add --sparse\").\n\n  b. Skip the index entirely and just construct your own tree/commit.\n     E.g., doing:\n\n       blob=$(git hash-object -w some-file)\n       tree=$({\n                git ls-tree HEAD &&\n\t\tprintf \"100644 blob $blob\\t%s\" some-file\n\t      } | git mktree --missing)\n       commit=$(echo my commit message | git commit-tree -p HEAD $tree)\n       git update-ref HEAD $commit\n\n    It gets a little trickier if your want to add to a sub-directory\n    (you have to recursively generate each tree).\n\nIn both cases you might also want to clone with \"--depth 1\", so you do\nnot bother grabbing old commits and trees, either.\n\n> git version 2.34.1 (Ubuntu 22.04)\n\nThe sparse-checkout feature is new-ish and has been actively worked on\nin the past few years. What I showed above works with the latest release\nof Git, but you may or may not need to upgrade (I didn't dig into the\ndetails).\n\n-Peff\n"},{"id":"489184","messageId":"MIda3qF70GEKtbezTxMyZo7RrlyBmvzN8w1T28cnCro4dmTqwjhQjBe2A-PYbgS6jkkev3pzd4oxqmbOfucfifVhTRQsynmjCpbQmvPzm8k=@proton.me","threadId":"60941","inReplyTo":"20240220024310.GD2713741@coredump.intra.peff.net","subject":"Re: Git commit causes data download in partial clone","fromName":"charmocc","fromEmail":"charmocc@proton.me","sentAt":"2024-02-22T19:34:01Z","receivedAt":"2024-02-22T19:34:16Z","isPatch":false,"sender":{"key":"charmocc@proton.me","avatar":null},"body":"Jeff thank you for taking a look at this.\n\nOn Tuesday, February 20th, 2024 at 3:43 AM, Jeff King <peff@peff.net> \nwrote:\n\n> a. Your --no-checkout skips the checkout, but it does not tell Git\n> that you are fundamentally uninterested in those other paths. To do\n> that, you can try the sparse-checkout mechanism. I'm not super\n> familiar with the feature myself, but doing:\n> \n> git clone --sparse --filter=blob:none $url nes\n\nThat's it! Now git knows exactly what I want from it. I am now able \nto add and commit new files and git status no longer lists any deleted \npaths. Also I can search for files I actually need with \"git ls-files\" \nand add them to the sparse checkout incrementally with \"git \nsparse-checkout add\". I haven't explored this feature before because \nbecause I assumed it is limited to selecting entire directory \nsubtrees not individual files which would not help with the \nmentioned repo.\n\n> Do note that --sparse checks out the contents of the top-level tree\n> by default. That's OK for your repo (all of the files are in the\n> Named_Titles directory), but it might not be true for some other\n> repos (it may also not work if your intent is to put another entry\n> into Named_Titles, though it looks like you might just need to say\n> \"git add --sparse\").\n\nI think this could be avoided by cloning with --no-checkout, \ncreating index with \"git reset\" and setting up empty sparse \ncheckout afterwards with \"git sparse-checkout set /\".\n\nAnyway it seems like with these tools I'm finally ready to tackle any \nhuge git repo they throw at me in the future :)\n\n\n"}]}