{"thread":{"id":"55825","subject":"Partial Clone, and a strange slow rev-list call on fetch","startedAt":"2021-06-02T04:56:40Z","lastAt":"2021-06-05T06:36:26Z","messageCount":5,"participants":["Tao Klerks","Derrick Stolee"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"426196","messageId":"CAPMMpogCz4o3ZGYHnux_6w+uFyxV-FR0R1hFNeg1COiv0qd_0g@mail.gmail.com","threadId":"55825","inReplyTo":null,"subject":"Partial Clone, and a strange slow rev-list call on fetch","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2021-06-02T04:56:25Z","receivedAt":"2021-06-02T04:56:40Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"Hi folks,\n\nI'm learning to use Partial Clone, and finding a behavior that I don't\nknow how to interpret or investigate:\n\nUnder some circumstances, doing a plain \"git fetch <remote>\" on a\nfiltered repo results in a very long (6-30 min?) wait, during which I\ncan see the following command being executed in the background:\n\n/usr/libexec/git-core/git rev-list --objects --stdin\n--exclude-promisor-objects --not --all --quiet --alternate-refs\n\nSo far, I have noted this happening under two distinct circumstances:\n* Anytime I try to fetch on a filtered repo with a git 2.23 client -\nshorter pause\n* When I try to fetch with a recent (2.31) client in a repo where one\nlarge packfile has no *.promisor file (but the others do, and the\nremote I am fetching from has promisor=true) - looong pause\n\nCan anyone explain what this rev-list call intends, and/or any hints\nas to how I could see what the stdin content being fed to it from the\nparent process actually is?\n\nFor background, I ended up in the \"missing promisor file\" situation by\ntrying to be (too?) clever about the blobs present in my clone: I\ncloned unfiltered shallow to a certain depth with certain refspecs,\nthen added the promisor and filter config, and finally fetched with\n\"--unshallow\". This produced exactly the blob-population state I\nintended, but meant the original first packfile had no \".promisor\"\nfile.\n\nCreating an empty promisor file for that packfile *appears* to fix the\nissue, and hasn't produced any weird side-effects that I've noted, and\nfrom the \"removing partial clone filtering\" description from gitlab at\nhttps://docs.gitlab.com/ee/topics/git/partial_clone.html#remove-partial-clone-filtering,\nappears to be a reasonable thing to do (the implication there is that\na promisor packfile with no missing objects hs exactly the same\nstructure as a non-promisor packfile), but of course I would welcome\nany validation or correction to that assumption.\n\nThanks for any info,\nTao Klerks\n"},{"id":"426220","messageId":"71e60d80-44c1-225d-3cf4-26740de2ac6d@gmail.com","threadId":"55825","inReplyTo":"CAPMMpogCz4o3ZGYHnux_6w+uFyxV-FR0R1hFNeg1COiv0qd_0g@mail.gmail.com","subject":"Re: Partial Clone, and a strange slow rev-list call on fetch","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-06-02T11:18:46Z","receivedAt":"2021-06-02T11:18:50Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/2/21 12:56 AM, Tao Klerks wrote:\n> Hi folks,\n> \n> I'm learning to use Partial Clone, and finding a behavior that I don't\n> know how to interpret or investigate:\n> \n> Under some circumstances, doing a plain \"git fetch <remote>\" on a\n> filtered repo results in a very long (6-30 min?) wait, during which I\n> can see the following command being executed in the background:\n> \n> /usr/libexec/git-core/git rev-list --objects --stdin\n> --exclude-promisor-objects --not --all --quiet --alternate-refs\n> \n> So far, I have noted this happening under two distinct circumstances:\n> * Anytime I try to fetch on a filtered repo with a git 2.23 client -\n> shorter pause\n> * When I try to fetch with a recent (2.31) client in a repo where one\n> large packfile has no *.promisor file (but the others do, and the\n> remote I am fetching from has promisor=true) - looong pause\n\nThis makes me think that there was a bug fix for this situation\nbut the fix requires doing extra work. To help track this down,\ncould you re-run the scenario with GIT_TRACE2_PERF=1 which will\ngive the full Git process stack as we reach that rev-list call.\n\n> Can anyone explain what this rev-list call intends, and/or any hints\n> as to how I could see what the stdin content being fed to it from the\n> parent process actually is?\n> \n> For background, I ended up in the \"missing promisor file\" situation by\n> trying to be (too?) clever about the blobs present in my clone: I\n> cloned unfiltered shallow to a certain depth with certain refspecs,\n> then added the promisor and filter config, and finally fetched with\n> \"--unshallow\". This produced exactly the blob-population state I\n> intended, but meant the original first packfile had no \".promisor\"\n> file.\n\nThis is the critical point: you first cloned without a filter,\nand then converted the remote to a promisor remote without\nmarking the pack-files you received from that remote as promisor\npack-files. That means that Git needs to do some work to discover\nwhich objects are reachable from promisor packs or not, and that\nextra work is slowing you down.\n\nPartial clone is designed to work where every remote is a\npromisor remote, and always has been so. Any deviation from that\nnorm is venturing into uncharted territory and will have friction\nlike this. Another similar issue comes when you have multiple\nremotes and one of them is a promisor remote and another is not.\n\nThe general advice right now is to use partial clone only if you\nwill use it for all remotes across the entire existence of the\nrepo.\n\nPart of the difficulty here is that once you download that first\npack-file from the remote, Git has no way of knowing that the\npack came from that source or was created in another way. We\nhave no way to be sure that we can \"upgrade\" the remote in an\nautomated process.\n\nThis does make me wonder what happens when Git repacks objects\ncreated locally and then starts fetching from a promisor remote.\n\nThere are some challenges here, for sure. Most likely also some\npotential gains, but it is unlikely to create a seamless\nexperience for what you are trying to do.\n\nThanks,\n-Stolee\n"},{"id":"426360","messageId":"CAPMMpogSepMk07HVtwO8qKaHx2SHmYVB+vnrnWwVWBNo8U5hAQ@mail.gmail.com","threadId":"55825","inReplyTo":"71e60d80-44c1-225d-3cf4-26740de2ac6d@gmail.com","subject":"Re: Partial Clone, and a strange slow rev-list call on fetch","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2021-06-03T21:10:58Z","receivedAt":"2021-06-03T21:11:12Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"On Wed, Jun 2, 2021 at 1:18 PM Derrick Stolee <stolee@gmail.com> wrote:\n\n> could you re-run the scenario with GIT_TRACE2_PERF=1 which will\n> give the full Git process stack as we reach that rev-list call.\n\nSorry about the delay, I've been trying to reproduce... reproduceably :)\n\nI now have a whole file of examples and observations, attached (I\nassume text attachments are allowed on this mailing list?), which\nshould be reproducible for anyone as I was able to use the linux\nkernel repo to illustrate all cases.\n\nMy observations probably use incorrect terminology and/or illustrate a\nlack of understanding of the underlying intended behaviors, but here\nare the most surprising (not-intentional-seeming) ones:\n * the \"rev-list\" calls are normal behavior for non-promisor remotes;\nthey normally complete very fast (with perhaps very little stdin\ninput??)\n * they happen in \"fetch.c\", right after the \"remote_refs\" labelled\ncode and within the \"consume_refs\" labelled code\n * for promisor remotes, these rev-list calls are normally\n(intentionally?) skipped. They are run only is there is a non-promisor\npackfile which contains a commit that is at the \"tip\" of one of the\n(promisor) remote's refs\n * when these rev-list calls happen, they are incredibly, strangely\nexpensive (a full \"git rev-list --all --objects\" completes *much*\nfaster)\n * even weirder, the cost of these calls appears proportional to the\n*promisor packfile sizes* - if you specify a more permissive filter at\nclone-time, and therefore have more objects actually there (not\nmissing/skipped) in your promisor packfiles, then it takes up taking\neven (much) longer\n * maybe weirdest - all of this goes away when the non-promisor\npackfile gets \"hidden\" behind a more recent promisor packfile;\nwhatever this check is supposed to do, I don't believe it is\nsound/correct.\n\n> This is the critical point: you first cloned without a filter,\n> and then converted the remote to a promisor remote without\n> marking the pack-files you received from that remote as promisor\n> pack-files. That means that Git needs to do some work to discover\n> which objects are reachable from promisor packs or not, and that\n> extra work is slowing you down.\n\nAs noted above, I don't think that work is, in the version I was\ntesting at least (2.31.1), correct. That said, I may well be\nmisunderstanding its intent.\n\n> This does make me wonder what happens when Git repacks objects\n> created locally and then starts fetching from a promisor remote.\n\nI can confirm that *if* the locally-created non-promisor packfile\ncontains a commit that is the \"tip\" of a (promisor) remote branch,\nthen this will trigger the strange/pathological fetch performance\nissue. As soon as you add a \"promisor\" marker for the packfile, or as\nsoon as someone else pushes new commits to the branch that then get\nfetched as new promisor packfiles in your repo, you're golden, the\nfetch process stops doing its \"panic - non-promisor packfile found\"\nbehavior... Even though a non-promisor packfile *is still in scope!*\n(just not at the tip of a ref for that remote)\n\nRelated to this, we have the other notable observation that \"git\nrepack\" (without asking to do anything with promisors) ends up doing\nthe same kind of work as the \"rev-list\" calls noted above, also\nproportional to the size and/or number of objects present in promisor\npackfiles; which is a little frustrating when the simplest cure to\n*other* performance issue, above, is to make all packfile promissor\nones.\n\n> There are some challenges here, for sure. Most likely also some\n> potential gains, but it is unlikely to create a seamless\n> experience for what you are trying to do.\n\nThere's one other crazy finding I should note explicitly here, which\nis that force-pushing a branch can, for reasons I cannot explain,\ncause you to redownload all the repo's commits & trees again (during\nthe forced push, as a just-in-time fetch). And again (if the branch\nchanges and you force-push again). And again for as long as you're\noverwriting others' changes. For the linux kernel repo without blobs,\nthat's 1.09GB a pop, of purely duplicated promisor packfiles\ncontaining the whole (no-blob-filtered) repo. Interestingly, the\nbehavior is the same regardless of your configured filter - so the\n\"blob=none\" behavior must be hardcoded or implied somehow in the\ncodepath that produces this weird outcome.\n\nThat said, so far I think everything I'm finding is manageable:\n* To avoid \"non-promisor packfile\" issues in the initial dual-stage\nclone, we just need to add extra promisor marker files when we\n\"upgrade\" the remote to a promisor remote.\n* Git repack performance will (presumably?) only affect background GC jobs\n* Any repack-originated non-promissor packfiles can be cleaned up\n(made promissor packfiles) by our tooling\n* Any weird re-downloads of the commits+trees during force-pushes\nshould be OK, they would \"only\" cost 400MB in our repo\n\nIf these are issues that someone is interested in looking into, I'd be\nvery happy to work with them, but my understanding of the codebase\n(and even the language) is... poor.\n\nThanks,\nTao\n\n\n\n# \n# This file is a log of investigations into non-obvious behaviors of \"partial clone\" repositories,\n# using a (writable, throwaway) fork of the linux kernel as test repository.\n# \n# The OUTCOMES described here are for git 2.31.1 on fedora linux 33 as of 2021-06-03\n# \n\n\n# -----\n# SETUP\n# -----\n\n# Enable perf tracing, including child process details\nexport GIT_TRACE2_PERF=1\n\n# Set up a suitable read & writable linux kernel remote URL\nexport REMOTE_URL=\"https://github.com/TaoK/linux.git\"\n\n\n\n# -------\n# OBSERVATIONS: \n#  - repos without promisor packfiles run a \"git rev-list\" process as part of \"fetch.c\", right after the \"remote_refs\" and within the \"consume_refs\" labelled code areas\n#  - these processes are generally very fast\n#  - these processes *don't run* if the packfiles containing the \"tips\"/refs shared/fetched from the remote are all promisor packfiles\n#  - these processes *run differently* if *any* of the packfiles containing the \"tips\"/refs shared/fetched from the remote are not promisor packfiles, but the remote is a promisor\n#    - Specifically, the same child processes do run, but an extra \"--exclude-promisor-objects\" parameter is added.\n#  - these child processes, running with a promisor remote, run very fast if promisor packfiles are small\n#  - these same child processes run very *slow* if promisor packfiles are large\n#  - that slowness is not *proportional* to packfile size, but it is related; this relationship appears to be non-linear\n#  - that slowness is substantially increased, for example, by using a \"loose\" filter, and ending up with many or all blobs *not missing* from the promisor packfile\n#  - that slowness seems to depend exclusively on the size/content of the promisor packfiles; non-promisor packfiles \"trigger\" the behavior but do not appear to impact its speed\n# -------\n\n\n# Regular single-branch quite-shallow clone without checkout, followed by a regular fetch\n# -> in the fetch, the \"remote_refs\" and \"consume_refs\" labelled areas spawn \"git rev-list --objects --stdin --not --all --quiet --alternate-refs\" processes\n# -> these processes complete instantaneously in this context\n# -> an equivalent process (same args) is invoked during clone, and labels progress as \"Checking connectivity\"\nTESTFOLDER=\"linuxtest_$RANDOM\" &&\n  git clone $REMOTE_URL $TESTFOLDER --no-checkout --shallow-since=2018-01-01 &&\n  git -C $TESTFOLDER fetch\n\n\n# Fully blob-filtered single-branch quite-shallow clone without checkout, followed by a regular fetch\n# -> \"remote_refs\" and \"consume_refs\" don't call \"rev-list\" child processes at all\nTESTFOLDER=\"linuxtest_$RANDOM\" &&\n  git clone $REMOTE_URL $TESTFOLDER --no-checkout --shallow-since=2018-01-01 --filter=blob:none &&\n  git -C $TESTFOLDER fetch\n\n# Regular single-branch quite-shallow clone, with a blob-filtered \"tip\"\n#  (another repo is created just to move the tip for the promisor packfile to be created)\n# -> \"remote_refs\" and \"consume_refs\" don't call \"rev-list\" child processes at all\nTESTFOLDER=\"linuxtest_$RANDOM\" &&\n  TIP_MOVER_FOLDER=\"linuxtest_$RANDOM\" &&\n  git clone $REMOTE_URL $TESTFOLDER --no-checkout --shallow-since=2018-01-01 &&\n  git clone $REMOTE_URL $TIP_MOVER_FOLDER --depth=1 && \n  echo \"Something $RANDOM\" > \"$TIP_MOVER_FOLDER/test_file_$RANDOM\" &&\n  git -C $TIP_MOVER_FOLDER add -A &&\n  git -C $TIP_MOVER_FOLDER commit --no-gpg-sign -m \"test file commit\" &&\n  git -C $TIP_MOVER_FOLDER push origin HEAD &&\n  git -C $TESTFOLDER config remote.origin.promisor true &&\n  git -C $TESTFOLDER config remote.origin.partialclonefilter blob:none &&\n  git -C $TESTFOLDER fetch &&\n  git -C $TESTFOLDER fetch\n\n# Regular single-branch quite-shallow clone, upgraded to a promisor remote\n# -> \"remote_refs\" and \"consume_refs\" *do* call \"rev-list\" child processes, with an extra parameter\n# -> these rev-list child processes complete very fast (presumably their speed relates to the volume of promisor packfiles in some way)\nTESTFOLDER=\"linuxtest_$RANDOM\" &&\n  git clone $REMOTE_URL $TESTFOLDER --no-checkout --shallow-since=2018-01-01 &&\n  git -C $TESTFOLDER config remote.origin.promisor true &&\n  git -C $TESTFOLDER config remote.origin.partialclonefilter blob:none &&\n  git -C $TESTFOLDER fetch\n\n# Regular single-branch quite-shallow clone, with *short* blob-filtered \"roots\" added later\n#  (the deepening is a little chaotic, with lots of blobs being retrieved, presumably from the different shallow roots)\n# -> the presence of small promisor packfiles (NOT at the tip) makes no difference; non-promisor packfiles at the tip trigger the rev-list call, it completes fast\nTESTFOLDER=\"linuxtest_$RANDOM\" &&\n  git clone $REMOTE_URL $TESTFOLDER --no-checkout --shallow-since=2018-01-01 &&\n  git -C $TESTFOLDER config remote.origin.promisor true &&\n  git -C $TESTFOLDER config remote.origin.partialclonefilter blob:none &&\n  git -C $TESTFOLDER fetch --shallow-since=2017-12-30 &&\n  git -C $TESTFOLDER fetch\n\n\n# Regular single-branch quite-shallow clone, with both a blob-filtered \"tip\" and a non-promisor-packfile \"tip\"\n#  (another repo is created just to move the tip for the promisor packfile to be created)\n# -> the presence of small promisor packfiles (at the tip) makes no difference; non-promisor packfiles at the tip trigger the rev-list call, it completes fast\nTESTFOLDER=\"linuxtest_$RANDOM\" &&\n  TIP_MOVER_FOLDER=\"linuxtest_$RANDOM\" &&\n  ORIGINAL_TIP_BRANCH=\"testbranch_$RANDOM\" &&\n  git clone $REMOTE_URL $TESTFOLDER --no-checkout --shallow-since=2018-01-01 &&\n  git clone $REMOTE_URL $TIP_MOVER_FOLDER --depth=1 && \n  echo \"Something $RANDOM\" > \"$TIP_MOVER_FOLDER/test_file_$RANDOM\" &&\n  git -C $TIP_MOVER_FOLDER add -A &&\n  git -C $TIP_MOVER_FOLDER commit --no-gpg-sign -m \"test file commit\" &&\n  git -C $TIP_MOVER_FOLDER push origin HEAD &&\n  git -C $TESTFOLDER config --add remote.origin.fetch \"+refs/heads/$ORIGINAL_TIP_BRANCH:refs/remotes/origin/$ORIGINAL_TIP_BRANCH\" &&\n  git -C $TESTFOLDER push origin \"HEAD:refs/heads/$ORIGINAL_TIP_BRANCH\" &&\n  git -C $TESTFOLDER config remote.origin.promisor true &&\n  git -C $TESTFOLDER config remote.origin.partialclonefilter blob:none &&\n  git -C $TESTFOLDER fetch &&\n  git -C $TESTFOLDER fetch\n\n\n# Regular single-branch very-shallow clone, with full blob-filtered history added later\n# -> the presence of *large* promisor packfiles (with non-promisor packfiles at the tip) means the rev-list call runs, and runs long...\n# (size: 205MB + 1.09GB, clone & unshallow: 60s + 314s, fetch: 324s made up of 2X 162s rev-list)\nTESTFOLDER=\"linuxtest_$RANDOM\" &&\n  git clone $REMOTE_URL $TESTFOLDER --no-checkout --depth=1 &&\n  git -C $TESTFOLDER config remote.origin.promisor true &&\n  git -C $TESTFOLDER config remote.origin.partialclonefilter blob:none &&\n  git -C $TESTFOLDER fetch --unshallow &&\n  git -C $TESTFOLDER fetch\n\n\n# Regular single-branch very-shallow clone, with full barely-filtered history added later\n# -> the presence of *huge* promisor packfiles (with non-promisor packfiles at the tip) means the rev-list call runs even longer\n# (size: 205MB + 2.73GB, clone & unshallow: 60s + 1420s, fetch: 1900s made up of 2X 940s rev-list)\nTESTFOLDER=\"linuxtest_$RANDOM\" &&\n  git clone $REMOTE_URL $TESTFOLDER --no-checkout --depth=1 &&\n  git -C $TESTFOLDER config remote.origin.promisor true &&\n  git -C $TESTFOLDER config remote.origin.partialclonefilter blob:limit=1m &&\n  git -C $TESTFOLDER fetch --unshallow &&\n  git -C $TESTFOLDER fetch\n\n\n# Regular single-branch \"deep\" clone *of a very old ref*, tag \"v2.6.13\", and then full blob-filtered history *since then* added later\n# -> the presence of *large* promisor packfiles (with non-promisor packfiles at the root) has exactly the same impact as with non-promisors at tip\n# (size: 76MB + 1.09GB, clone & fetch: 9s + 265s, fetch: 320s made up of 2X 160s rev-list)\nTESTFOLDER=\"linuxtest_$RANDOM\" &&\n  git clone $REMOTE_URL $TESTFOLDER --no-checkout -b v2.6.13 --single-branch &&\n  git -C $TESTFOLDER config --add remote.origin.fetch \"+refs/heads/master:refs/remotes/origin/master\" &&\n  git -C $TESTFOLDER config remote.origin.promisor true &&\n  git -C $TESTFOLDER config remote.origin.partialclonefilter blob:none &&\n  git -C $TESTFOLDER fetch &&\n  git -C $TESTFOLDER fetch\n\n\n\n\n# -------\n# OBSERVATIONS: \n#  - running git repack on a fully-promisored repo can/will yield non-promisor packfiles (if there are loose objects from initially-local commits)\n#  - if/when such a locally-packed non-promisor packfile contains any \"tip\" commits for the promisor remote, then:\n#    - fetch will be slow (depending on the size of promissor packfiles)\n#    - this will continue until you manually mark the pack file as \".promisor\" OR you get a later tip for the affected branch(es) from a promisor remote\n# -------\n\n#\n# (clone: 180s, add: 70s warmup, fetch: 1s, repack: 173s, fetch: 340s from 2X rev-list at 170s)\n# -> after repack, fetch is *very* slow until one of two corrective circumstances arise:\n#  1. you make the stray packfile a promisor packfile, or \n#  2. you fetch another change (to that branch) from the promisor remote / the tip of the branch \"naturally\" becomes a promisor again\nTESTFOLDER=\"linuxtest_$RANDOM\" &&\n  git clone $REMOTE_URL $TESTFOLDER --single-branch --filter=blob:none &&\n  echo \"Something $RANDOM\" > \"$TESTFOLDER/test_file_$RANDOM\" &&\n  git -C $TESTFOLDER add -A &&\n  git -C $TESTFOLDER commit --no-gpg-sign -m \"test file commit\" &&\n  git -C $TESTFOLDER push &&\n  git -C $TESTFOLDER fetch\n  git -C $TESTFOLDER repack &&\n  git -C $TESTFOLDER fetch\nFIXUPFOLDER=\"linuxtest_$RANDOM\" &&\n  git clone $REMOTE_URL $FIXUPFOLDER --depth=1 &&\n  echo \"Something $RANDOM\" > \"$FIXUPFOLDER/test_file_$RANDOM\" &&\n  git -C $FIXUPFOLDER add -A &&\n  git -C $FIXUPFOLDER commit --no-gpg-sign -m \"test file commit\" &&\n  git -C $FIXUPFOLDER push\ngit -C $TESTFOLDER fetch\n\n\n\n\n\n# -------\n# OBSERVATIONS: \n#  - Under some specific circumstances, force-pushing a branch from a partial clone causes this repo \n#       to *re-fetch the repo's commits & trees* into another new promisor packfile. \n#    - This can be repeated any number of times, yielding effectively identical (large) duplicate packfiles\n#    - The preconditions appear to be that the remote and the local repo each have mutually unknown commits at the tip...\n#  - The behavior is to re-download the commits & trees only, even if the filter settings on the repo are \"lax\" (eg \"limit=1M\")\n# -------\n\n\n# Get a fully filtered full clone, commit to the branch from elsewhere, and force the originally cloned branch state back\n# -> during the forced push we randomly, strangely, re-retrieve the whole filtered clone data - commits + trees\nTESTFOLDER=\"linuxtest_$RANDOM\" &&\n  INTERFERINGFOLDER=\"linuxtest_$RANDOM\" &&\n  git clone $REMOTE_URL $INTERFERINGFOLDER --depth=1 &&\n  git clone $REMOTE_URL $TESTFOLDER --filter=blob:none &&\n  git -C $INTERFERINGFOLDER commit --allow-empty --no-gpg-sign -m \"test file commit\" &&\n  git -C $INTERFERINGFOLDER push &&\n  git -C $TESTFOLDER push -f\n\n# Confirm - there are two 1.09-GB packfiles\nll \"$TESTFOLDER/.git/objects/pack\"\n\n# This can be repeated any number of times, each time creating yet another 1.09GB promisor packfile...\ngit -C $INTERFERINGFOLDER commit --allow-empty --no-gpg-sign -m \"test file commit\" &&\n  git -C $INTERFERINGFOLDER push -f &&\n  git -C $TESTFOLDER push -f \n\n# Confirm - there are three 1.09-GB packfiles\nll \"$TESTFOLDER/.git/objects/pack\"\n\n# This can be repeated any number of times, each time creating yet another 1.09GB promisor packfile...\ngit -C $INTERFERINGFOLDER commit --allow-empty --no-gpg-sign -m \"test file commit\" &&\n  git -C $INTERFERINGFOLDER push -f &&\n  git -C $TESTFOLDER push -f \n\n# Confirm - there are four 1.09-GB packfiles\nll \"$TESTFOLDER/.git/objects/pack\"\n\n\n\n# Again with a \"loosely\" filtered clone\n# -> Even though the original clone size is 3GB, the later \"duplicated\" download is once again 1.09 GB - commits + trees only.\nTESTFOLDER=\"linuxtest_$RANDOM\" &&\n  INTERFERINGFOLDER=\"linuxtest_$RANDOM\" &&\n  git clone $REMOTE_URL $INTERFERINGFOLDER --depth=1 &&\n  git clone $REMOTE_URL $TESTFOLDER --filter=blob:limit=1M &&\n  git -C $INTERFERINGFOLDER commit --allow-empty --no-gpg-sign -m \"test file commit\" &&\n  git -C $INTERFERINGFOLDER push &&\n  git -C $TESTFOLDER push -f\n\n\n\n\n\n# ----\n# OBSERVATIONS: \n#  - git repack is slow on filtered repos (repos with promisor packfiles)\n#  - git repack's speed is *directly related to the size of the promisor packfiles*\n#     (but given that repos with a mix of promisor and non-promisor packfiles misbehave in other ways, this conclusion is of limited value)\n# ----\n\n# Baseline - regular full clone\n# (size = 3.11GB, clone = 235s, repack = 10s)\nTESTFOLDER=\"linuxtest_$RANDOM\" &&\n  git clone $REMOTE_URL $TESTFOLDER &&\n  git -C $TESTFOLDER repack\n\n# Baseline - very-shallow single-branch clone\n# (size = 203MB, clone = 74s, repack = 0s)\nTESTFOLDER=\"linuxtest_$RANDOM\" &&\n  git clone $REMOTE_URL $TESTFOLDER --depth=1 &&\n  git -C $TESTFOLDER repack\n\n# Demo - fully blob-filtered full clone, normal checkout\n# (size = 1.09GB + 203MB, clone = 90s + 100s, repack = 180s)\nTESTFOLDER=\"linuxtest_$RANDOM\" &&\n  git clone $REMOTE_URL $TESTFOLDER --filter=blob:none &&\n  git -C $TESTFOLDER repack\n\n# Demo - fully blob-filtered full clone, no checkout\n# (size = 1.09GB, clone = 70s, repack = 160s)\nTESTFOLDER=\"linuxtest_$RANDOM\" &&\n  git clone $REMOTE_URL $TESTFOLDER --filter=blob:none --no-checkout &&\n  git -C $TESTFOLDER repack\n\n# Demo - fully blob-filtered very-shallow single-branch clone, normal checkout\n# (size = 203MB, clone = 150s, repack = 8s)\nTESTFOLDER=\"linuxtest_$RANDOM\" &&\n  git clone $REMOTE_URL $TESTFOLDER --filter=blob:none --depth=1 &&\n  git -C $TESTFOLDER repack\n\n# Demo - fully blob-filtered very-shallow single-branch clone, no checkout\n# (size = 2MB, clone = 2s, repack = 0s)\nTESTFOLDER=\"linuxtest_$RANDOM\" &&\n  git clone $REMOTE_URL $TESTFOLDER --filter=blob:none --no-checkout --depth=1 &&\n  git -C $TESTFOLDER repack\n\n# Demo - barely blob-filtered single-branch clone, no checkout\n# (size = 3.1GB, clone = 200s, repack = 810s)\nTESTFOLDER=\"linuxtest_$RANDOM\" &&\n  git clone $REMOTE_URL $TESTFOLDER --filter=blob:limit=10m --single-branch --no-checkout &&\n  git -C $TESTFOLDER repack\n\n# Demo - medium-deep shallow single-branch clone (no checkout), followed by fully filtered unshallow\n# (promisor packfile \"in the past\" of the single tip ref, with most history in the \"tip\" non-promisor packfile)\n# (size = 2.63GB + 200MB, clone = 337s + 108s, repack = 28s)\n#  -> Repack time is directly related to promisor packfile size/content/scope when further back in history\nTESTFOLDER=\"linuxtest_$RANDOM\" &&\n  git clone $REMOTE_URL $TESTFOLDER --no-checkout --shallow-since=2010-01-01 --single-branch && \n  git -C $TESTFOLDER config remote.origin.promisor true &&\n  git -C $TESTFOLDER config remote.origin.partialclonefilter blob:none &&\n  git -C $TESTFOLDER fetch --unshallow &&\n  git -C $TESTFOLDER repack\n\n# Demo - reasonably shallow single-branch clone (no checkout), followed by extra commit appearing at tip on the remote, \n#  followed by filtered fetch of that new tip\n# (promisor packfile \"at the tip\", with most of the history in a non-promisor packfile \"behind the tip\")\n# (size = 981MB + , clone = 2s, repack = 3s)\n# (TIP_MOVER clone time ignored)\n#  -> Repack time is directly related to promisor packfile size/content/scope when at tip\nTESTFOLDER=\"linuxtest_$RANDOM\" &&\n  TIP_MOVER_FOLDER=\"linuxtest_$RANDOM\" &&\n  git clone $REMOTE_URL $TESTFOLDER --no-checkout --shallow-since=2018-01-01 --single-branch && \n  git clone $REMOTE_URL $TIP_MOVER_FOLDER --depth=1 && \n  echo \"Something $RANDOM\" > \"$TIP_MOVER_FOLDER/test_file_$RANDOM\" &&\n  git -C $TIP_MOVER_FOLDER add -A &&\n  git -C $TIP_MOVER_FOLDER commit --no-gpg-sign -m \"test file commit\" &&\n  git -C $TIP_MOVER_FOLDER push origin HEAD &&\n  git -C $TESTFOLDER config remote.origin.promisor true &&\n  git -C $TESTFOLDER config remote.origin.partialclonefilter blob:none &&\n  git -C $TESTFOLDER fetch &&\n  git -C $TESTFOLDER repack\n\n\n\n# -------\n# CLEANUP\n# -------\n\n# Delete all the weird repos created.\nrm -rf linuxtest_*\n\n\n"},{"id":"426416","messageId":"5ce5286c-9390-7cb5-a039-8e26be990b8d@gmail.com","threadId":"55825","inReplyTo":"CAPMMpogSepMk07HVtwO8qKaHx2SHmYVB+vnrnWwVWBNo8U5hAQ@mail.gmail.com","subject":"Re: Partial Clone, and a strange slow rev-list call on fetch","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-06-04T13:21:49Z","receivedAt":"2021-06-04T13:23:07Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/3/2021 5:10 PM, Tao Klerks wrote:\n> On Wed, Jun 2, 2021 at 1:18 PM Derrick Stolee <stolee@gmail.com> wrote:\n> \n>> could you re-run the scenario with GIT_TRACE2_PERF=1 which will\n>> give the full Git process stack as we reach that rev-list call.\n> \n> Sorry about the delay, I've been trying to reproduce... reproduceably :)\n> \n> I now have a whole file of examples and observations, attached (I\n> assume text attachments are allowed on this mailing list?), which\n> should be reproducible for anyone as I was able to use the linux\n> kernel repo to illustrate all cases.\n\nI appreciate that you took so much time to investigate here. You\nhave convinced me that there are deeper things going on than just\nthe \"unshallow, but with filters this time\" situation.\n\nI have created an internal issue for my team to investigate this\nwhen we have capacity for it. I don't think it will happen this\nmonth, so if anyone else has the time now then don't wait for us.\n\nThanks,\n-Stolee\n"},{"id":"426471","messageId":"CAPMMpojV6McG-mc1xwuC0CEY5_E7=PBGQF-ijhb-EP4T=6Qu7w@mail.gmail.com","threadId":"55825","inReplyTo":"5ce5286c-9390-7cb5-a039-8e26be990b8d@gmail.com","subject":"Re: Partial Clone, and a strange slow rev-list call on fetch","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2021-06-05T06:35:02Z","receivedAt":"2021-06-05T06:36:26Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"On Fri, Jun 4, 2021 at 3:21 PM Derrick Stolee <stolee@gmail.com> wrote:\n> You\n> have convinced me that there are deeper things going on than just\n> the \"unshallow, but with filters this time\" situation.\n>\n> I have created an internal issue for my team to investigate this\n> when we have capacity for it.\n\nAwesome, thanks!\n\nOne more comment on this: As I was trawling through release notes\nlooking for the right minimum git client version to target/support, I\ndiscovered that the whole \"fetch shallow, then unshallow filtered\"\nflow, which I might have thought I came up with on my own, was\nintended to be supported with a change Xin Li made about a year ago,\nand landed in 2.28.0:\nhttps://github.com/git/git/commit/01bbbbd9daaa277a95ae46e5a32f6fba026610ac.\n\nAs far as I can tell this implementation/behavior does *not* attempt\nto convert existing non-promisor packfiles to promisor packfiles, so\neither this behavior should be adjusted to do so, or we should\nconsider this flow (and the presence of non-promisor packfiles at the\n\"tip\") supported...?\n"}]}