{"thread":{"id":"55815","subject":"Removing Partial Clone / Filtered Clone on a repo","startedAt":"2021-06-01T10:24:42Z","lastAt":"2021-06-02T05:04:46Z","messageCount":6,"participants":["Tao Klerks","Derrick Stolee"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"426108","messageId":"CAPMMpoim38J3=4pd0_fM2h=DN_PrEE_Osg2duU5Ur8WUZ5S1Pg@mail.gmail.com","threadId":"55815","inReplyTo":null,"subject":"Removing Partial Clone / Filtered Clone on a repo","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2021-06-01T10:24:23Z","receivedAt":"2021-06-01T10:24:42Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"Hi folks,\n\nI'm trying to deepen my understanding of the Partial Clone\nfunctionality for a possible deployment at scale (with a large-ish\n13GB project where we are using date-based shallow clones for the time\nbeing), and one thing that I can't get my head around yet is how you\n\"unfilter\" an existing filtered clone.\n\nThe gitlab intro document\n(https://docs.gitlab.com/ee/topics/git/partial_clone.html#remove-partial-clone-filtering)\nsuggests that you need to get the full list of missing blobs, and pass\nthat into a fetch...:\n\ngit fetch origin $(git rev-list --objects --all --missing=print | grep\n-oP '^\\?\\K\\w+')\n\nIn my project's case, that would be millions of blob IDs! I tested\nthis with a path-based filter to rev-list, to see what getting 30,000\nblobs might look like, and it took a looong while... I don't\nunderstand much about the negotiation process, but I have to assume\nthere is a fixed per-blob cost in this scenario which is *much* higher\nthan in a \"regular\" fetch or clone.\n\nObviously one answer is to throw away the repo and start again with a\nclean unfiltered clone... But between repo-local config, project\nsettings in IDEs / external tools, and unpushed local branches, this\nis an awkward thing to ask people to do.\n\nI initially thought it might be possible to add an extra remote\n(without filter / promisor settings), mess with the negotiation\nsettings to make the new remote not know anything about what's local,\nand then get a full set of refs and their blobs from that remote...\nbut I must have misunderstood how the negotation-tip stuff works\nbecause I can't get that to do anything (it always \"sees\" my existing\nrefs and I just get the new remote's refs \"for free\" without object\ntransfer).\n\nThe official doc at https://git-scm.com/docs/partial-clone makes no\nmention of plans or goals (or non-goals) related to this \"unfiltering\"\n- is it something that we should expect a story to emerge around?\n\nThanks,\nTao Klerks\n"},{"id":"426110","messageId":"032cabb2-652a-1d88-2e12-601b40a4020c@gmail.com","threadId":"55815","inReplyTo":"CAPMMpoim38J3=4pd0_fM2h=DN_PrEE_Osg2duU5Ur8WUZ5S1Pg@mail.gmail.com","subject":"Re: Removing Partial Clone / Filtered Clone on a repo","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-06-01T10:39:28Z","receivedAt":"2021-06-01T10:39:38Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/1/21 6:24 AM, Tao Klerks wrote:\n> Hi folks,\n> \n> I'm trying to deepen my understanding of the Partial Clone\n> functionality for a possible deployment at scale (with a large-ish\n> 13GB project where we are using date-based shallow clones for the time\n> being), and one thing that I can't get my head around yet is how you\n> \"unfilter\" an existing filtered clone.\n> \n> The gitlab intro document\n> (https://docs.gitlab.com/ee/topics/git/partial_clone.html#remove-partial-clone-filtering)\n> suggests that you need to get the full list of missing blobs, and pass\n> that into a fetch...:\n> \n> git fetch origin $(git rev-list --objects --all --missing=print | grep\n> -oP '^\\?\\K\\w+')\n\nI think the short answer is to split your \"git rev-list\" call\ninto batches by limiting the count. Perhaps pipe that command\nto a file and then split it into batches of \"reasonable\" size.\n\nYour definition of \"reasonable\" may vary, so try a few numbers.\n\n> The official doc at https://git-scm.com/docs/partial-clone makes no\n> mention of plans or goals (or non-goals) related to this \"unfiltering\"\n> - is it something that we should expect a story to emerge around?\n\nThe design is not intended for this kind of \"unfiltering\". The\nfeature is built for repositories where doing so would be too\nexpensive (both network time and disk space) to be valuable.\n\nAlso, asking for the objects one-by-one like this is very\ninefficient on the server side. A fresh clone can make use of\nexisting delta compression in a way that this type of request\ncannot (at least, not easily). You _would_ be better off making\na fresh clone and then adding that pack-file to your\n.git/objects/pack directory of the repository you want.\n\nCould you describe more about your scenario and why you want to\nget all objects?\n\nThanks,\n-Stolee\n"},{"id":"426122","messageId":"CAPMMpohOuXX-0YOjV46jFZFvx7mQdj0p7s8SDR4SQxj5hEhCgg@mail.gmail.com","threadId":"55815","inReplyTo":"032cabb2-652a-1d88-2e12-601b40a4020c@gmail.com","subject":"Re: Removing Partial Clone / Filtered Clone on a repo","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2021-06-01T13:16:51Z","receivedAt":"2021-06-01T13:17:06Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"On Tue, Jun 1, 2021 at 12:39 PM Derrick Stolee <stolee@gmail.com> wrote:\n\n> Could you describe more about your scenario and why you want to\n> get all objects?\n\nA 13GB (with 1.2GB shallow head) repo is in that in-between spot where\nyou want to be able to get something useful to the user as fast as\npossible (read: in less than the 4 hours it would take to download the\nwhole thing over a mediocre VPN, with corresponding risk of errors\npartway), but where a user might later (eg overnight) want to get the\nrest of the repo, to avoid history inconsistency issues.\n\nIn our current mode of operation (Shallow clones to 15 months' depth\nby default), the initial clone can complete in well under an hour, but\nthe problem with the resulting clone is that normal git tooling will\nsee the shallow grafted commit as the \"initial commit\" of all older\nfiles, and that causes no end of confusion on the part of users, eg on\n\"git blame\". This is the main reason why we would like to consider\nmoving to full-history but filtered-blob clones.\n\n(there are other reasons around manageability, eg the git server's\nbehavior around --shallow-since when some branches in refspec scope\nare older than that date; it sends them with all their history,\neffectively downloading the whole repo; similarly if a refspec is\nexpanded and the next fetch is run without explicit --shallow-since,\nand finds new branches not already shallow-grafted, it will download\nthose in their entirely because the shallow-since date is not\npersisted beyond the shallow grafts themselves).\n\nWith a (full-history all-trees no-blobs-except-HEAD) filtered clone,\nthe initial download can be quite a bit smaller than the shallow clone\nscenario above (eg 1.5GB vs 2.2GB), and most of the disadvantages of\nshallow clones are addressed: the just-in-time fetching can typically\nwork quite naturally, there are no \"lies\" in the history, nor are\nthere scenarios where you suddenly fetch an extra 10GB of history\nwithout wanting/expecting to.\n\nWith the filtered clone there are still little edge-cases that might\nmotivate a user to \"bite the bullet\" and unfilter their clone,\nhowever: The most obvious one I've found so far is \"git blame\" - it\nloops fetch requests serially until it bottoms out, which on an older\npoorly-factored file (hundreds or thousands of commits, each touching\ndifferent bits of a file) will effectively never complete, at\n10s/fetch. And depending on the UI tooling the user is using, they may\nhave almost no visibility into why this \"git blame\" (or \"annotate\", or\nwhatever the given UI calls it) seems to hang forever.\n\nYou can work around this \"git blame\" issue for *most* situations, in\nthe case of our repo, by using a different initial filter spec, eg\n\"--filter=blob:limit=200k\", which only costs you an extra 1GB or so...\nBut then you still have outliers - and in fact, the most \"blameable\"\nfiles will tend to be the larger ones... :)\n\nMy working theory is that we should explain all the following to users:\n* Your initial download is a nice compromise between functionality and\ndownload delay\n* You have almost all the useful history, and you have it within less\nthan an hour\n* If you try to use \"git blame\" (or some other as-yet-undiscovered\nscenarios) on a larger file, it may hang. In that case cancel, run a\nmagic command we provide which fetches all the blobs in that specific\nfile's history, and try again. (the magic command is a path-filtered\nrev-list looking for missing objects, passed into fetch)\n* If you ever get tired of the rare weird hangs, you have the option\nof running *some process* that \"unfilters\" the repo, paying down that\ninitial compromise (and taking up a bit more HD space), eg overnight\n\nThis explanation is a little awkward, but less awkward than the\nprevious \"'git blame' lies to you - it blames completely the wrong\nperson for the bulk of the history for the bulk of the files;\nunshallow overnight if this bothers you\", which is the current story\nwith shallow clone.\n\nOf course, if unfiltering a large repo is impractical (and if it will\nremain so), then we will probably need to err on the side of\ngenerosity in the original clone - eg 1M instead of 200k as the blob\nfilter, 3GB vs 2.5GB as the initial download - and remove the last\nline of the explanation! If unfiltering, or refiltering, were\npractical, then we would probably err on the size of\nless-blobs-by-default to optimize first download.\n\nOver time, as we refactor the project itself to reduce the incidence\nof megafiles, I would expect to be able to drop the\nstandard/recommended blob-size-limit too.\n\nSorry about the wall-of-text, hopefully I've answered the question!\n\nThanks,\nTao\n"},{"id":"426124","messageId":"0b57cba9-3ab3-dfdf-5589-a0016eaea634@gmail.com","threadId":"55815","inReplyTo":"CAPMMpohOuXX-0YOjV46jFZFvx7mQdj0p7s8SDR4SQxj5hEhCgg@mail.gmail.com","subject":"Re: Removing Partial Clone / Filtered Clone on a repo","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-06-01T13:40:40Z","receivedAt":"2021-06-01T13:40:45Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/1/2021 9:16 AM, Tao Klerks wrote:\n> On Tue, Jun 1, 2021 at 12:39 PM Derrick Stolee <stolee@gmail.com> wrote:\n> \n>> Could you describe more about your scenario and why you want to\n>> get all objects?\n> \n> A 13GB (with 1.2GB shallow head) repo is in that in-between spot where\n> you want to be able to get something useful to the user as fast as\n> possible (read: in less than the 4 hours it would take to download the\n> whole thing over a mediocre VPN, with corresponding risk of errors\n> partway), but where a user might later (eg overnight) want to get the\n> rest of the repo, to avoid history inconsistency issues.\n\nAs you describe below, the inconsistency is in terms of performance,\nnot correctness. I thought it was worth a clarification.\n\n...\n> With the filtered clone there are still little edge-cases that might\n> motivate a user to \"bite the bullet\" and unfilter their clone,\n> however: The most obvious one I've found so far is \"git blame\" - it\n> loops fetch requests serially until it bottoms out, which on an older\n> poorly-factored file (hundreds or thousands of commits, each touching\n> different bits of a file) will effectively never complete, at\n> 10s/fetch. And depending on the UI tooling the user is using, they may\n> have almost no visibility into why this \"git blame\" (or \"annotate\", or\n> whatever the given UI calls it) seems to hang forever.\n\nI'm aware that the first 'git blame' on a file is a bit slow in the\npartial clone case. It's been on my list for improvement whenever I\nget the \"spare\" time to do it. However, if someone else wants to work\non it I will briefly outline the approach I was going to investigate:\n\n  During the history walk for 'git blame', it might be helpful to\n  collect a batch of blobs to download in a single round trip. This\n  requires refactoring the search to walk the commit history and\n  collect a list of (commit id, blob id) pairs as if we were doing\n  a simplified history walk. We can then ask for the list of blob id's\n  in a single request and then perform the line-by-line blaming logic\n  on that list. [If we ever hit a point where we would do a rename\n  check, pause the walk and request all blobs so far and flush the\n  line-by-line diff before continuing.]\n\nThis basic idea is likely difficult to implement, but would likely\ndramatically improve the first 'git blame' in a blobless clone. A\nsimilar approach could maybe be used by the line-log logic\n(git log -L).\n\n> You can work around this \"git blame\" issue for *most* situations, in\n> the case of our repo, by using a different initial filter spec, eg\n> \"--filter=blob:limit=200k\", which only costs you an extra 1GB or so...\n> But then you still have outliers - and in fact, the most \"blameable\"\n> files will tend to be the larger ones... :)\n\nI'm interested in this claim that 'the most \"blameable\" files will\ntend to be the larger ones.' I typically expect blame to be used on\nhuman-readable text files, and my initial reaction is that larger\nfiles are harder to use with 'git blame'.\n\nHowever, your 200k limit isn't so large that we can't expect _some_\nfiles to reach that size. Looking at the root of git.git I see a\nfew files above 100k and files like diff.c reaching very close to\n200k (uncompressed). I tend to find that the files in git.git are\nsmaller than the typical large project.\n \n> My working theory is that we should explain all the following to users:\n> * Your initial download is a nice compromise between functionality and\n> download delay\n> * You have almost all the useful history, and you have it within less\n> than an hour\n> * If you try to use \"git blame\" (or some other as-yet-undiscovered\n> scenarios) on a larger file, it may hang. In that case cancel, run a\n> magic command we provide which fetches all the blobs in that specific\n> file's history, and try again. (the magic command is a path-filtered\n> rev-list looking for missing objects, passed into fetch)\n> * If you ever get tired of the rare weird hangs, you have the option\n> of running *some process* that \"unfilters\" the repo, paying down that\n> initial compromise (and taking up a bit more HD space), eg overnight\n\nPartial clone is all about tradeoffs: you get faster clones that\ndownload missing objects as needed. The user behavior dictates how\nmany objects are needed, so the user has the capability to adjust\nthat need. The fewer objects needed locally, the faster the repo\nwill be.\n\nYour concern about slow commands is noted, but also blindly\ndownloading every file in history will slow the repo due to the\nfull size of the objects on disk.\n\nI think there is merit to your back-filling history idea. There\nare likely benefits to the \"download everything missing\" concept,\nbut also it would be good to design such a feature to have other\ncustom knobs, such as:\n\n* Get only \"recent\" history, perhaps with a \"--since=<date>\"\n  kind of flag. This would walk commits only to a certain date,\n  then find all missing blobs reachable from their root trees.\n\n* Get only a \"cone\" of history. This could work especially well\n  with sparse-checkout, but other pathspecs could be used to\n  limit the walk. \n\n...\n> Of course, if unfiltering a large repo is impractical (and if it will\n> remain so), then we will probably need to err on the side of\n> generosity in the original clone - eg 1M instead of 200k as the blob\n> filter, 3GB vs 2.5GB as the initial download - and remove the last\n> line of the explanation! If unfiltering, or refiltering, were\n> practical, then we would probably err on the size of\n> less-blobs-by-default to optimize first download.\n\nI'm glad that you have self-discovered a workaround to handle\nthese cases. If we had a refiltering feature, then you could even\nstart with a blobless clone to have an extremely fast initial\nclone, followed by a background job that downloads the remaining\nobjects.\n\n> Over time, as we refactor the project itself to reduce the incidence\n> of megafiles, I would expect to be able to drop the\n> standard/recommended blob-size-limit too.\n\nMy experience working with large repos and partial clone is\nsimilar: the new pain points introduced by these features make\nusers aware of \"repository smells\" in their organization and\nthey tend to self-correct by refactoring the repository. This\nis a never-ending process as repos grow, especially with many\ncontributors.\n\nThank you for sharing your experience!\n\n-Stolee\n"},{"id":"426151","messageId":"CAPMMpoi_XSLJXtxGD74r-uoao+Z6Xcg8xkXuZCYix4=tBWD8rw@mail.gmail.com","threadId":"55815","inReplyTo":"0b57cba9-3ab3-dfdf-5589-a0016eaea634@gmail.com","subject":"Re: Removing Partial Clone / Filtered Clone on a repo","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2021-06-01T16:54:20Z","receivedAt":"2021-06-01T16:54:35Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"On Tue, Jun 1, 2021 at 3:40 PM Derrick Stolee <stolee@gmail.com> wrote:\n> > you want to be able to get something useful to the user as fast as\n> > possible [...] but where a user might later (eg overnight) want to get the\n> > rest of the repo, to avoid history inconsistency issues.\n>\n> As you describe below, the inconsistency is in terms of performance,\n> not correctness. I thought it was worth a clarification.\n\nSorry I was not clear here - I did not mean formal correctness nor\nperformance, when referring to the incentive to get the rest of the\nrepo - I was referring to the fact that a medium-shallow clone (eg 15\nmonths of a 20-year project) provides an inconsistent perspective on\nthe code history:\n * On the one hand, most of the time you have everything you need, and\nwhen you bump up against *available* history limits from a file or\nbranch history view, it's reasonably clear that's what's happening (in\nsome UI tools this is more explicit than in others).\n * On the other hand, when you happen to look at something older, it\nis easy for the history to seem to \"lie\", showing changes made in a\nfile by a person that really *didn't* make those changes. Their commit\njust happened to be selected as the shallow graft, and so seems to\nhave \"added\" all the files in the project. This reasonably\nintelligible when looking at file history, but extremely non-obvious\nwhen looking at git blame (in a medium-shallow clone).\n\n> I'm aware that the first 'git blame' on a file is a bit slow in the\n> partial clone case.\n\nWithout wanting to harp on about it, it can easily be pathologically\nslow, eg in my case a random well-trafficked file has 300 in-scope\ncommits, at 10 seconds per independent blob fetch - and so ends up\ntaking an hour to git blame (the first time for such a file, as you\nnoted).\n\n> It's been on my list for improvement whenever I\n> get the \"spare\" time to do it. However, if someone else wants to work\n> on it I will briefly outline the approach I was going to investigate:\n\nOne reason I wasn't asking about / angling for this, particularly, is\nthat I expect there will be other tools doing their own versions of\nthis. I haven't tested \"tig\" on this, for example, but I suspect it\ndoesn't do a plain git blame, given what I've seen of its instantly\nshowing the file contents and  \"gradually\" filling in the authorship\ndata. I for one rarely use plain git blame, I don't know much about\nthe usage patterns of other users. Most of \"my\" users will be using\nIntellij IDEA, which seems to have a surprisingly solid/scalable git\nintegration (but I have not yet tested this case there yet)\n\nThere also other related reasons to go for a \"get most of the relevant\nblobs across history\" approach, specifically around tooling: there are\nlots of tools & integrations that use git libraries (or even homebrew\nimplementations) rather than the git binaries / IPC, and many of those\ntend to lag *far* behind in support for things like shallow clone,\npartial clone, mailmap, core.splitindex, replace refs, etc etc. My\ncurrent beef is with Sublime Merge, which is snappy as one could wish\nfor, really lovely to use within its scope, but doesn't have any idea\nwhat a promisor is, and simply says \"nah, no content here\" when you\nlook at a missing blob. (for the moment)\n\n> > the most \"blameable\"\n> > files will tend to be the larger ones... :)\n>\n> I'm interested in this claim that 'the most \"blameable\" files will\n> tend to be the larger ones.' I typically expect blame to be used on\n> human-readable text files, and my initial reaction is that larger\n> files are harder to use with 'git blame'.\n\nAbsolutely, I meant \"the larger text/code files\", not including other\nstuff that tends to accumulate in the higher filesize brackets. I\nmeant that I, for one, in this project at least, often find myself\nusing git blame (or equivalent) to \"spelunk\" into who touched a\nspecific line, in cases where looking at the plain history is useless\nbecause there have been many hundreds or thousands of changes - and in\nmy limited experience, files with that many reasons to change tend to\nbe large.\n\n> Your concern about slow commands is noted, but also blindly\n> downloading every file in history will slow the repo due to the\n> full size of the objects on disk.\n\nI have in the past claimed that \"larger repo\" (specifically, a deeper\nclone that gets many larger blobs) is slower, but haven't actually\nfound any significant evidence to back my claim. Obviously something\nlike \"git gc\" will be slower, but is there anything in the practical\nday-to-day that cares whether the commit depth is 10,000 commits or\n200,000 commits for a given branch, or whether you only have the blobs\nat the \"tip\" of the branch/project, or all the blobs in history?\n(besides GC, specifically)\n\n> it would be good to design such a feature to have other\n> custom knobs, such as:\n> * Get only \"recent\" history, perhaps with a \"--since=<date>\"\n>   kind of flag. This would walk commits only to a certain date,\n>   then find all missing blobs reachable from their root trees.\n\nAs long as you know at initial clone time that this is what you want,\ncombining shallow clone with sparse clone already enables this today\n(shallow clone, set up filter, unshallow, and potentially remove\nfilter). You can even do more complicated things like unshallowing\nwith different increasingly-aggressive filters in multiple\nsteps/fetches over different time periods. The main challenge that I\nperceive at the moment is that you're effectively locked into \"one\nshot\". As soon as you've retrieved the commits with blobs missing,\n\"filling them in\" at scale seems to be orders of magnitude more\nexpensive than an equivalent clone would have been.\n\n> If we had a refiltering feature, then you could even\n> start with a blobless clone to have an extremely fast initial\n> clone, followed by a background job that downloads the remaining\n> objects.\n\nYes please!\n\n\nI think one thing that I'm not clearly understanding yet in this\nconversation, is whether the tax on explicit and specialized blob list\nfetching could be made much lower. As far as I can tell, in a blobless\nclone with full trees we have most of the data one could want, to\ndecide what blobs to request - paths, filetypes, and commit dates.\nThis leaves three pain points that I am aware of:\n* Filesizes are not (afaik) available in a blobless clone. This sounds\nlike a pretty deep limitation, which I'll gloss over.\n* Blob paths are available in trees, but not trivially exposed by git\nrev-list - could a new \"--missing\" option value make sense? Or does it\nmake just as much sense to expect the caller/scripter to iterate\nls-tree outputs? (I assume doing so would be much slower, but have not\ntested)\n* Something about the \"git fetch <remote> blob-hash ...\" pattern seems\nto scale very poorly - is that something that might see change in\nfuture, or is it a fundamental issue?\n\n\nThanks again for the detailed feedback!\nTao\n"},{"id":"426197","messageId":"CAPMMpoiReALTrn5R2VK6FK+15XgOJd-8r2rZsmQZXbH0R1kq+A@mail.gmail.com","threadId":"55815","inReplyTo":"CAPMMpoi_XSLJXtxGD74r-uoao+Z6Xcg8xkXuZCYix4=tBWD8rw@mail.gmail.com","subject":"Re: Removing Partial Clone / Filtered Clone on a repo","fromName":"Tao Klerks","fromEmail":"tao@klerks.biz","sentAt":"2021-06-02T05:04:30Z","receivedAt":"2021-06-02T05:04:46Z","isPatch":false,"sender":{"key":"tao@klerks.biz","avatar":"https://avatars.githubusercontent.com/u/531704?v=4"},"body":"I understand replying to myself is bad form, but I need to add a\ncorrection/clarification to a statement I made below:\n\nOn Tue, Jun 1, 2021 at 6:54 PM Tao Klerks <tao@klerks.biz> wrote:> >\nit would be good to design such a feature to have other> > custom\nknobs, such as:\n> > * Get only \"recent\" history, perhaps with a \"--since=<date>\"\n> >   kind of flag. This would walk commits only to a certain date,\n> >   then find all missing blobs reachable from their root trees.\n>\n> As long as you know at initial clone time that this is what you want,\n> combining shallow clone with sparse clone already enables this today\n> (shallow clone, set up filter, unshallow, and potentially remove\n> filter). You can even do more complicated things like unshallowing\n> with different increasingly-aggressive filters in multiple\n> steps/fetches over different time periods. The main challenge that I\n> perceive at the moment is that you're effectively locked into \"one\n> shot\". As soon as you've retrieved the commits with blobs missing,\n> \"filling them in\" at scale seems to be orders of magnitude more\n> expensive than an equivalent clone would have been.\n\nAs I just noted in another thread, there seems to be one extra step\nneeded to pull this off: you need to add a *.promisor file for the\ninitial shallow clone's packfile, because otherwise (at least with the\n2.31 client that I am using) later \"git fetch\" calls take forever\ndoing something with rev-list that I don't understand, presumably due\nto the relationship between promisor packfiles and non-promisor\npackfiles...\n"}]}