{"thread":{"id":"32735","subject":"[PATCH 0/2] optimizing pack access on \"read only\" fetch repos","startedAt":"2013-01-26T22:40:11Z","lastAt":"2017-07-12T20:43:14Z","messageCount":21,"participants":["Jeff King","Jonathan Nieder","Junio C Hamano","Shawn Pearce","Duy Nguyen","Martin Fick","Ævar Arnfjörð Bjarmason"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"207895","messageId":"20130126224011.GA20675@sigill.intra.peff.net","threadId":"32735","inReplyTo":null,"subject":"[PATCH 0/2] optimizing pack access on \"read only\" fetch repos","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-01-26T22:40:11Z","receivedAt":"2013-01-26T22:40:11Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"This is a repost from here:\n\n http://thread.gmane.org/gmane.comp.version-control.git/211176\n\nwhich got no response initially. Basically the issue is that read-only\nrepos (e.g., a CI server) whose workflow is something like:\n\n git fetch $some_branch &&\n git checkout -f $some_branch &&\n make test\n\nwill never run git-gc, and will accumulate a bunch of small packs and\nloose objects, leading to poor performance.\n\nPatch 1 runs \"gc --auto\" on fetch, which I think is sane to do.\n\nPatch 2 optimizes our pack dir re-scanning for fetch-pack (which, unlike\nthe rest of git, should expect to be missing lots of objects, since we\nare deciding what to fetch).\n\nI think 1 is a no-brainer. If your repo is packed, patch 2 matters less,\nbut it still seems like a sensible optimization to me.\n\n [1/2]: fetch: run gc --auto after fetching\n [2/2]: fetch-pack: avoid repeatedly re-scanning pack directory\n\n-Peff\n"},{"id":"207896","messageId":"20130126224038.GA20849@sigill.intra.peff.net","threadId":"32735","inReplyTo":"20130126224011.GA20675@sigill.intra.peff.net","subject":"[PATCH 1/2] fetch: run gc --auto after fetching","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-01-26T22:40:38Z","receivedAt":"2013-01-26T22:40:38Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"We generally try to run \"gc --auto\" after any commands that\nmight introduce a large number of new objects. An obvious\nplace to do so is after running \"fetch\", which may introduce\nnew loose objects or packs (depending on the size of the\nfetch).\n\nWhile an active developer repository will probably\neventually trigger a \"gc --auto\" on another action (e.g.,\ngit-rebase), there are two good reasons why it is nicer to\ndo it at fetch time:\n\n 1. Read-only repositories which track an upstream (e.g., a\n continuous integration server which fetches and builds,\n but never makes new commits) will accrue loose objects\n and small packs, but never coalesce them into a more\n efficient larger pack.\n\n 2. Fetching is often already perceived to be slow to the\n user, since they have to wait on the network. It's much\n more pleasant to include a potentially slow auto-gc as\n part of the already-long network fetch than in the\n middle of productive work with git-rebase or similar.\n\nSigned-off-by: Jeff King \n---\n builtin/fetch.c | 5 +++++\n 1 file changed, 5 insertions(+)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 4b5a898..1ddbf0d 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -959,6 +959,9 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tstruct string_list list = STRING_LIST_INIT_NODUP;\n \tstruct remote *remote;\n \tint result = 0;\n+\tstatic const char *argv_gc_auto[] = {\n+\t\t\"gc\", \"--auto\", NULL,\n+\t};\n \n \tpacket_trace_identity(\"fetch\");\n \n@@ -1026,5 +1029,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tlist.strdup_strings = 1;\n \tstring_list_clear(&list, 0);\n \n+\trun_command_v_opt(argv_gc_auto, RUN_GIT_CMD);\n+\n \treturn result;\n }\n-- \n1.8.0.2.16.g72e2fc9\n"},{"id":"207897","messageId":"20130126224043.GB20849@sigill.intra.peff.net","threadId":"32735","inReplyTo":"20130126224011.GA20675@sigill.intra.peff.net","subject":"[PATCH 2/2] fetch-pack: avoid repeatedly re-scanning pack directory","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-01-26T22:40:43Z","receivedAt":"2013-01-26T22:40:43Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"When we look up a sha1 object for reading, we first check\npackfiles, and then loose objects. If we still haven't found\nit, we re-scan the list of packfiles in `objects/pack`. This\nfinal step ensures that we can co-exist with a simultaneous\nrepack process which creates a new pack and then prunes the\nold object.\n\nThis extra re-scan usually does not have a performance\nimpact for two reasons:\n\n 1. If an object is missing, then typically the re-scan\n will find a new pack, then no more misses will occur.\n Or if it truly is missing, then our next step is\n usually to die().\n\n 2. Re-scanning is cheap enough that we do not even notice.\n\nHowever, these do not always hold. The assumption in (1) is\nthat the caller is expecting to find the object. This is\nusually the case, but the call to `parse_object` in\n`everything_local` does not follow this pattern. It is\nlooking to see whether we have objects that the remote side\nis advertising, not something we expect to have. Therefore\nif we are fetching from a remote which has many refs\npointing to objects we do not have, we may end up\nre-scanning the pack directory many times.\n\nEven with this extra re-scanning, the impact is often not\nnoticeable due to (2); we just readdir() the packs directory\nand skip any packs that are already loaded. However, if\nthere are a large number of packs, then even enumerating the\ndirectory directory can be expensive (especially if we do it\nrepeatedly). Having this many packs is a good sign the user\nshould run `git gc`, but it would still be nice to avoid\nhaving to scan the directory at all.\n\nSigned-off-by: Jeff King \n---\n fetch-pack.c | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex f0acdf7..6d8926a 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -594,6 +594,9 @@ static int everything_local(struct fetch_pack_args *args,\n \tfor (ref = *refs; ref; ref = ref->next) {\n \t\tstruct object *o;\n \n+\t\tif (!has_sha1_file(ref->old_sha1))\n+\t\t\tcontinue;\n+\n \t\to = parse_object(ref->old_sha1);\n \t\tif (!o)\n \t\t\tcontinue;\n-- \n1.8.0.2.16.g72e2fc9\n"},{"id":"207902","messageId":"20130127015110.GB3224@elie.Belkin","threadId":"32735","inReplyTo":"20130126224038.GA20849@sigill.intra.peff.net","subject":"Re: [PATCH 1/2] fetch: run gc --auto after fetching","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2013-01-27T01:51:10Z","receivedAt":"2013-01-27T01:51:10Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jeff King wrote:\n\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -959,6 +959,9 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> \tstruct string_list list = STRING_LIST_INIT_NODUP;\n> \tstruct remote *remote;\n> \tint result = 0;\n> +\tstatic const char *argv_gc_auto[] = {\n> +\t\t\"gc\", \"--auto\", NULL,\n> +\t};\n> \n> \tpacket_trace_identity(\"fetch\");\n> \n> @@ -1026,5 +1029,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n> \tlist.strdup_strings = 1;\n> \tstring_list_clear(&list, 0);\n> \n> +\trun_command_v_opt(argv_gc_auto, RUN_GIT_CMD);\n> +\n> \treturn result;\n\nGood idea, and the execution is obviously correct. Thanks.\n"},{"id":"207937","messageId":"7vlibfxhit.fsf@alter.siamese.dyndns.org","threadId":"32735","inReplyTo":"20130126224011.GA20675@sigill.intra.peff.net","subject":"Re: [PATCH 0/2] optimizing pack access on \"read only\" fetch repos","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-27T06:32:42Z","receivedAt":"2013-01-27T06:32:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> This is a repost from here:\n>\n> http://thread.gmane.org/gmane.comp.version-control.git/211176\n>\n> which got no response initially. Basically the issue is that read-only\n> repos (e.g., a CI server) whose workflow is something like:\n>\n> git fetch $some_branch &&\n> git checkout -f $some_branch &&\n> make test\n>\n> will never run git-gc, and will accumulate a bunch of small packs and\n> loose objects, leading to poor performance.\n>\n> Patch 1 runs \"gc --auto\" on fetch, which I think is sane to do.\n>\n> Patch 2 optimizes our pack dir re-scanning for fetch-pack (which, unlike\n> the rest of git, should expect to be missing lots of objects, since we\n> are deciding what to fetch).\n>\n> I think 1 is a no-brainer. If your repo is packed, patch 2 matters less,\n> but it still seems like a sensible optimization to me.\n>\n> [1/2]: fetch: run gc --auto after fetching\n> [2/2]: fetch-pack: avoid repeatedly re-scanning pack directory\n>\n> -Peff\n\nBoth makes sense to me.\n\nI also wonder if we would be helped by another \"repack\" mode that\ncoalesces small packs into a single one with minimum overhead, and\nrun that often from \"gc --auto\", so that we do not end up having to\nhave 50 packfiles.\n\nWhen we have 2 or more small and young packs, we could:\n\n - iterate over idx files for these packs to enumerate the objects\n to be packed, replacing read_object_list_from_stdin() step;\n\n - always choose to copy the data we have in these existing packs,\n instead of doing a full prepare_pack(); and\n\n - use the order the objects appear in the original packs, bypassing\n compute_write_order().\n\nThe procedure cannot be a straight byte-for-byte copy, because some\nobjects may appear in multiple packs, and extra copies of the same\nobject have to be excised from the result. OFS_DELTA offsets need\nto be adjusted for objects that appear later in the output and for\nobjects that were deltified against such an object that recorded its\nbase with OFS_DELTA format.\n\nBut other than such OFS_DELTA adjustments, it feels that such an\n\"only coalesce multiple packs into one\" mode should be fairly quick.\n"},{"id":"207944","messageId":"20130127102753.GB4228@elie.Belkin","threadId":"32735","inReplyTo":"20130126224043.GB20849@sigill.intra.peff.net","subject":"Re: [PATCH 2/2] fetch-pack: avoid repeatedly re-scanning pack directory","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2013-01-27T10:27:53Z","receivedAt":"2013-01-27T10:27:53Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nJeff King wrote:\n\n> When we look up a sha1 object for reading, we first check\n> packfiles, and then loose objects. If we still haven't found\n> it, we re-scan the list of packfiles in `objects/pack`. This\n> final step ensures that we can co-exist with a simultaneous\n> repack process which creates a new pack and then prunes the\n> old object.\n\nI like the context above and what follows it, but I think you forgot\nto mention what the patch actually does. :)\n\nI guess it is:\n\n\tHowever, in the first scan over refs in fetch-pack.c::everything_local,\n\tthis double-check of packfiles is not necessary since we are only\n\ttrying to get a rough estimate of the last time we fetched from this\n\tremote repository in order to find good candidate common commits ---\n\ta missed object would only result in a slightly slower fetch.\n\n\tAvoid that slow second scan in the common case by guarding the object\n\tlookup with has_sha1_file().\n\nSounds like it would not affect most fetches except by making them\na lot faster in the many-refs case, so for what it's worth I like it.\n\nI had not read this codepath before. I'm left with a few questions:\n\n * Why is 49bb805e (\"Do not ask for objects known to be complete\",\n 2005-10-19) trying to do? Are we hurting that in any way?\n\n For the sake of an example, suppose in my stalled project I\n maintain 20 topic branches against an unmoving mainline I do not\n advertise and you regularly fetch from me. The cutoff is the\n *newest* commit date of any of my topic branches you already have.\n By declaring you have that topic branch you avoid a complicated\n negotiation to discover that we both have the mainline. Is that\n the goal?\n\n * Is has_sha1_file() generally succeptible to the race against repack\n you mentioned? How is that normally dealt with?\n\n * Can a slow operation get confused if an object is incorporated into\n a pack and then expelled again by two repacks in sequence?\n\nThanks,\nJonathan\n"},{"id":"207977","messageId":"7vvcaiv14s.fsf@alter.siamese.dyndns.org","threadId":"32735","inReplyTo":"20130127102753.GB4228@elie.Belkin","subject":"Re: [PATCH 2/2] fetch-pack: avoid repeatedly re-scanning pack directory","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-27T20:09:39Z","receivedAt":"2013-01-27T20:09:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder writes:\n\n> Jeff King wrote:\n>\n>> When we look up a sha1 object for reading, we first check\n>> packfiles, and then loose objects. If we still haven't found\n>> it, we re-scan the list of packfiles in `objects/pack`. This\n>> final step ensures that we can co-exist with a simultaneous\n>> repack process which creates a new pack and then prunes the\n>> old object.\n>\n> I like the context above and what follows it, but I think you forgot\n> to mention what the patch actually does. :)\n>\n> I guess it is:\n>\n> \tHowever, in the first scan over refs in fetch-pack.c::everything_local,\n> \tthis double-check of packfiles is not necessary since we are only\n> \ttrying to get a rough estimate of the last time we fetched from this\n> \tremote repository in order to find good candidate common commits ---\n> \ta missed object would only result in a slightly slower fetch.\n\nIt is not about a rough estimate nor common commits, though. The\n\"everything local\" check in question is interested in only one\nthing: are we _clearly_ up to date without fetching anything from\nthem?\n\nLoosening the check may miss the rare case where we race against a\nsimultaneous repack and will cause us to go to the network when we\ndo not have to, and it becomes a trade off between the common unracy\ncase going faster by allowing the \"Are we clearly up to date\" check\nto cheat, at the expense of rare racy cases suffering unnecessary\nobject transfer overhead.\n\n> \tAvoid that slow second scan in the common case by guarding the object\n> \tlookup with has_sha1_file().\n\nThis conclusion is correct.\n\n> I had not read this codepath before. I'm left with a few questions:\n>\n> * Why is 49bb805e (\"Do not ask for objects known to be complete\",\n> 2005-10-19) trying to do? Are we hurting that in any way?\n\nAn earlier fetch may have acquired all the necessary objects but may\nnot have updated our refs for some reason (e.g. fast-forward check\nmay have fired). In such a case, we may already have a history that\nis good (i.e. not missing paths down to the common history) in our\nrepository that is not connected to any of our refs, and we can\nupdate our refs (or write to FETCH_HEAD) without asking the remote\nend to do any common ancestor computation or object transfer.\n\nThat was the primary thing the patch wanted to do.\n\nAs a side-effect, we know more objects than just the objects at the\ntips of our refs are complete and that may help the later common\nhistory discovery step, but obviously we do not want to dig the\nhistory down to root. The cutoff value is merely a heuristics\nchosen without any deep thought.\n\n> * Is has_sha1_file() generally succeptible to the race against repack\n> you mentioned? How is that normally dealt with?\n\nBy failing to find, so that the user will restart. When the caller\nreally wants to use the object, parse_objects() => read_sha1_file()\n=> read_object() is used and we will see the retry.\n\n> * Can a slow operation get confused if an object is incorporated into\n> a pack and then expelled again by two repacks in sequence?\n\nIf it checks \"the object should be there\" first, wait for a long\ntime, and then tries to find that object's data, the later access\nwill go to the parse_objects() callpath and I think it should do the\nright thing. If that slow opearation stops inside read_object(), it\ncould find it unable to map the loose object file and then unable to\nfind it in the pack, either. Is that what you are worried about?\n"},{"id":"208004","messageId":"20130127232033.GA8206@elie.Belkin","threadId":"32735","inReplyTo":"7vvcaiv14s.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 2/2] fetch-pack: avoid repeatedly re-scanning pack directory","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2013-01-27T23:20:33Z","receivedAt":"2013-01-27T23:20:33Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Junio C Hamano wrote:\n\n> It is not about a rough estimate nor common commits, though. The\n> \"everything local\" check in question is interested in only one\n> thing: are we _clearly_ up to date without fetching anything from\n> them?\n[...]\n> Jonathan Nieder writes:\n\n>> * Why is 49bb805e (\"Do not ask for objects known to be complete\",\n>> 2005-10-19) trying to do? Are we hurting that in any way?\n>\n> An earlier fetch may have acquired all the necessary objects but may\n> not have updated our refs for some reason (e.g. fast-forward check\n> may have fired). In such a case, we may already have a history that\n> is good (i.e. not missing paths down to the common history) in our\n> repository that is not connected to any of our refs, and we can\n> update our refs (or write to FETCH_HEAD) without asking the remote\n> end to do any common ancestor computation or object transfer.\n>\n> That was the primary thing the patch wanted to do.\n\nInteresting.\n\nAfter I fetch objects for branches a, b, and c which all have different\ncommit times in a fetch that fails due to the fast-forward check, I\nhave the following objects:\n\n\ta\tcommit date = 25 January 2013\n\tb\tcommit date = 26 January 2013\n\tc\tcommit date = 27 January 2013\n\nWhen I try to fetch again (forcibly this time), git notices that the\nobjects are available locally and sets \"cutoff\" to 27 January 2013 as\na hint about the last time I fetched.\n\nmark_recent_complete_commits() is called, and since these objects are\nnot reachable from any local refs, none are visited in the walk, and\n49bb805e does not affect the outcome of the fetch positively or\nnegatively.\n\nOn the other hand, if I fetched a, b, and c to local branches and then\nbuilt on top of them, 49bb805e ensures that a second fetch of the same\nrefs will know right away not to request objects for c. So it brings\nsome of the benefits of 2759cbc7 (git-fetch-pack: avoid unnecessary\nzero packing, 2005-10-18) when the receiving side has either (A)\nrenamed refs or (B) built new history on top of them.\n\nCorrect?\n\nIt is only in case (B) that the cutoff matters. If we miscalculate\nthe cutoff, that means either (i) cutoff == 0, and we would lose the\nbenefit of the walk that finds complete commits, or (ii) cutoff is a\nlittle earlier, and the walk to find complete commits would take a\nlittle longer. Neither effects the result of the fetch, and neither\nis likely to be a significant enough performance difference to offset\nthe benefit of Jeff's patch.\n\nThanks for explaining.\nJonathan\n"},{"id":"208159","messageId":"CAJo=hJtDn_6qsQ7uj2Pi=vnjNXR7WPN1VH1Mb8YdneD_bHUK8w@mail.gmail.com","threadId":"32735","inReplyTo":"7vlibfxhit.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 0/2] optimizing pack access on \"read only\" fetch repos","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2013-01-29T08:06:33Z","receivedAt":"2013-01-29T08:06:33Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Sat, Jan 26, 2013 at 10:32 PM, Junio C Hamano wrote:\n> Jeff King writes:\n>\n>> This is a repost from here:\n>>\n>> http://thread.gmane.org/gmane.comp.version-control.git/211176\n>>\n>> which got no response initially. Basically the issue is that read-only\n>> repos (e.g., a CI server) whose workflow is something like:\n>>\n>> git fetch $some_branch &&\n>> git checkout -f $some_branch &&\n>> make test\n>>\n>> will never run git-gc, and will accumulate a bunch of small packs and\n>> loose objects, leading to poor performance.\n...\n> I also wonder if we would be helped by another \"repack\" mode that\n> coalesces small packs into a single one with minimum overhead, and\n> run that often from \"gc --auto\", so that we do not end up having to\n> have 50 packfiles.\n\nYes. This does help....\n\n> When we have 2 or more small and young packs, we could:\n>\n> - iterate over idx files for these packs to enumerate the objects\n> to be packed, replacing read_object_list_from_stdin() step;\n>\n> - always choose to copy the data we have in these existing packs,\n> instead of doing a full prepare_pack(); and\n>\n> - use the order the objects appear in the original packs, bypassing\n> compute_write_order().\n\nHmm, sounds familiar. Seems like its what we do in JGit for Android. :-)\n"},{"id":"208163","messageId":"20130129082939.GB6396@sigill.intra.peff.net","threadId":"32735","inReplyTo":"7vlibfxhit.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 0/2] optimizing pack access on \"read only\" fetch repos","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-01-29T08:29:40Z","receivedAt":"2013-01-29T08:29:40Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Sat, Jan 26, 2013 at 10:32:42PM -0800, Junio C Hamano wrote:\n\n> Both makes sense to me.\n> \n> I also wonder if we would be helped by another \"repack\" mode that\n> coalesces small packs into a single one with minimum overhead, and\n> run that often from \"gc --auto\", so that we do not end up having to\n> have 50 packfiles.\n> \n> When we have 2 or more small and young packs, we could:\n> \n> - iterate over idx files for these packs to enumerate the objects\n> to be packed, replacing read_object_list_from_stdin() step;\n> \n> - always choose to copy the data we have in these existing packs,\n> instead of doing a full prepare_pack(); and\n> \n> - use the order the objects appear in the original packs, bypassing\n> compute_write_order().\n\nI'm not sure. If I understand you correctly, it would basically just be\nconcatenating packs without trying to do delta compression between the\nobjects which are ending up in the same pack. So it would save us from\nhaving to do (up to) 50 binary searches to find an object in a pack, but\nwould not actually save us much space.\n\nI would be interested to see the timing on how quick it is compared to a\nreal repack, as the I/O that happens during a repack is non-trivial\n(although if you are leaving aside the big \"main\" pack, then it is\nprobably not bad).\n\nBut how do these somewhat mediocre concatenated packs get turned into\nreal packs? Pack-objects does not consider deltas between objects in the\nsame pack. And when would you decide to make a real pack? How do you\nknow you have 50 young and small packs, and not 50 mediocre coalesced\npacks?\n\n-Peff\n"},{"id":"208176","messageId":"CACsJy8AHEyQ5-qz9auB3vT_Esmos2t-cRhtvf3zbVJDMFRET_A@mail.gmail.com","threadId":"32735","inReplyTo":"7vlibfxhit.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 0/2] optimizing pack access on \"read only\" fetch repos","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2013-01-29T11:01:38Z","receivedAt":"2013-01-29T11:01:38Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Jan 27, 2013 at 1:32 PM, Junio C Hamano wrote:\n> I also wonder if we would be helped by another \"repack\" mode that\n> coalesces small packs into a single one with minimum overhead, and\n> run that often from \"gc --auto\", so that we do not end up having to\n> have 50 packfiles.\n>\n> When we have 2 or more small and young packs, we could:\n>\n> - iterate over idx files for these packs to enumerate the objects\n> to be packed, replacing read_object_list_from_stdin() step;\n>\n> - always choose to copy the data we have in these existing packs,\n> instead of doing a full prepare_pack(); and\n>\n> - use the order the objects appear in the original packs, bypassing\n> compute_write_order().\n\nIsn't it easier and cheaper to create the \"master index\", something\nlike bup does?\n-- \nDuy\n"},{"id":"208189","messageId":"2c88f15d-405b-4dd1-8546-94f54f0ac565@email.android.com","threadId":"32735","inReplyTo":"20130129082939.GB6396@sigill.intra.peff.net","subject":"Re: [PATCH 0/2] optimizing pack access on \"read only\" fetch repos","fromName":"Martin Fick","fromEmail":"mfick@codeaurora.org","sentAt":"2013-01-29T15:25:39Z","receivedAt":"2013-01-29T15:25:39Z","isPatch":true,"sender":{"key":"mfick@codeaurora.org","avatar":null},"body":"\n\nJeff King wrote:\n\n>On Sat, Jan 26, 2013 at 10:32:42PM -0800, Junio C Hamano wrote:\n>\n>> Both makes sense to me.\n>> \n>> I also wonder if we would be helped by another \"repack\" mode that\n>> coalesces small packs into a single one with minimum overhead, and\n>> run that often from \"gc --auto\", so that we do not end up having to\n>> have 50 packfiles.\n>> \n>> When we have 2 or more small and young packs, we could:\n>> \n>> - iterate over idx files for these packs to enumerate the objects\n>> to be packed, replacing read_object_list_from_stdin() step;\n>> \n>> - always choose to copy the data we have in these existing packs,\n>> instead of doing a full prepare_pack(); and\n>> \n>> - use the order the objects appear in the original packs, bypassing\n>> compute_write_order().\n>\n>I'm not sure. If I understand you correctly, it would basically just be\n>concatenating packs without trying to do delta compression between the\n>objects which are ending up in the same pack. So it would save us from\n>having to do (up to) 50 binary searches to find an object in a pack,\n>but\n>would not actually save us much space.\n>\n>I would be interested to see the timing on how quick it is compared to\n>a\n>real repack, as the I/O that happens during a repack is non-trivial\n>(although if you are leaving aside the big \"main\" pack, then it is\n>probably not bad).\n>\n>But how do these somewhat mediocre concatenated packs get turned into\n>real packs? Pack-objects does not consider deltas between objects in\n>the\n>same pack. And when would you decide to make a real pack? How do you\n>know you have 50 young and small packs, and not 50 mediocre coalesced\n>packs?\n\n\nIf we are reconsidering repacking strategies, I would like to propose an approach that might be a more general improvement to repacking which would help in more situations. \n\nYou could roll together any packs which are close in size, say within 50% of each other. With this strategy you will end up with files which are spread out by size exponentially. I implementated this strategy on top of the current gc script using keep files, it works fairly well:\n\nhttps://gerrit-review.googlesource.com/#/c/35215/3/contrib/git-exproll.sh\n\nThis saves some time, but mostly it saves I/O when repacking regularly. I suspect that if this strategy were used in core git that further optimizations could be made to also reduce the repack time, but I don't know enough about repacking to know? We run it nightly on our servers, both write and read only mirrors. We us are a ratio of 5 currently to drastically reduce large repack file rollovers,\n\n-Martin\n"},{"id":"208191","messageId":"7vwquwrng6.fsf@alter.siamese.dyndns.org","threadId":"32735","inReplyTo":"20130129082939.GB6396@sigill.intra.peff.net","subject":"Re: [PATCH 0/2] optimizing pack access on \"read only\" fetch repos","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-29T15:58:01Z","receivedAt":"2013-01-29T15:58:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n>> I also wonder if we would be helped by another \"repack\" mode that\n>> coalesces small packs into a single one with minimum overhead, and\n>> run that often from \"gc --auto\", so that we do not end up having to\n>> have 50 packfiles.\n>> ...\n>\n> I'm not sure. If I understand you correctly, it would basically just be\n> concatenating packs without trying to do delta compression between the\n> objects which are ending up in the same pack. So it would save us from\n> having to do (up to) 50 binary searches to find an object in a pack, but\n> would not actually save us much space.\n\nThe point is not about space. Disk is cheap, and it is not making\nit any worse than what happens to your target audience, that is a\nfetch-only repository with only \"gc --auto\" in it, where nobody\npasses \"-f\" to \"repack\" to cause recomputation of delta.\n\nWhat I was trying to seek was a way to reduce the runtime penalty we\npay every time we run git in such a repository.\n\n - Object look-up cost will become log2(50*n) from 50*log2(n), which\n is about 50/log2(50) improvement;\n\n - System resource cost we incur by having to keep 50 file\n descriptors open and maintaining 50 mmap windows will reduce by\n 50 fold.\n\n - Anything else I missed?\n\n> I would be interested to see the timing on how quick it is compared to a\n> real repack,...\n\nYes, that is what I meant by \"wonder if we would be helped by\" ;-)\n\n> But how do these somewhat mediocre concatenated packs get turned into\n> real packs?\n\nHow do they get processed in a fetch-only repositories that\nsometimes run \"gc --auto\" today? By runninng \"repack -a -d -f\"\noccasionally, perhaps?\n\nAt some point, you would need to run a repack that involves a real\nobject-graph traversal that feeds you the paths for objects to\nobtain a reasonably compressed pack. We can inspect existing packs\nand guess a rough traversal order for commits, but for trees and\nblobs, you cannot unify existing delta chains from multiple packs\neffectively with data in the pack alone.\n"},{"id":"208222","messageId":"20130129211932.GA17377@sigill.intra.peff.net","threadId":"32735","inReplyTo":"7vwquwrng6.fsf@alter.siamese.dyndns.org","subject":"Re: [PATCH 0/2] optimizing pack access on \"read only\" fetch repos","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-01-29T21:19:33Z","receivedAt":"2013-01-29T21:19:33Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Jan 29, 2013 at 07:58:01AM -0800, Junio C Hamano wrote:\n\n> The point is not about space. Disk is cheap, and it is not making\n> it any worse than what happens to your target audience, that is a\n> fetch-only repository with only \"gc --auto\" in it, where nobody\n> passes \"-f\" to \"repack\" to cause recomputation of delta.\n> \n> What I was trying to seek was a way to reduce the runtime penalty we\n> pay every time we run git in such a repository.\n> \n> - Object look-up cost will become log2(50*n) from 50*log2(n), which\n> is about 50/log2(50) improvement;\n\nYes and no. Our heuristic is to look at the last-used pack for an\nobject. So assuming we have locality of requests, we should quite often\nget \"lucky\" and find the object in the first log2 search. Even if we\ndon't assume locality, a situation with one large pack and a few small\npacks will have the large one as \"last used\" more often than the others,\nand it will also have the looked-for object more often than the others\n\nSo I can see how it is something we could potentially optimize, but I\ncould also see it being surprisingly not a big deal. I'd be very\ninterested to see real measurements, even of something as simple as a\n\"master index\" which can reference multiple packfiles.\n\n> - System resource cost we incur by having to keep 50 file\n> descriptors open and maintaining 50 mmap windows will reduce by\n> 50 fold.\n\nI wonder how measurable that is (and if it matters on Linux versus less\nefficient platforms).\n\n> > I would be interested to see the timing on how quick it is compared to a\n> > real repack,...\n> \n> Yes, that is what I meant by \"wonder if we would be helped by\" ;-)\n\nThere is only one way to find out... :)\n\nMaybe I am blessed with nice machines, but I have mostly found the\nrepack process not to be that big a deal these days (especially with\nthreaded delta compression).\n\n> > But how do these somewhat mediocre concatenated packs get turned into\n> > real packs?\n> \n> How do they get processed in a fetch-only repositories that\n> sometimes run \"gc --auto\" today? By runninng \"repack -a -d -f\"\n> occasionally, perhaps?\n\nDo we run \"repack -adf\" regularly? The usual \"git gc\" procedure will not\nuse \"-f\", and without that, we will not even consider making deltas\nbetween objects that were formerly in different packs (but now are in\nthe same pack).\n\nSo you are avoiding doing medium-effort packs (\"repack -ad\") in favor of\ndoing potentially quick packs, but occasionally doing a big-effort pack\n(\"repack -adf\"). It may be reasonable advice to \"repack -adf\"\noccasionally, but I suspect most people are not doing it regularly (if\nonly because \"git gc\" does not do it by default).\n\n-Peff\n"},{"id":"208224","messageId":"7vmwvrocb9.fsf@alter.siamese.dyndns.org","threadId":"32735","inReplyTo":"20130129211932.GA17377@sigill.intra.peff.net","subject":"Re: [PATCH 0/2] optimizing pack access on \"read only\" fetch repos","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2013-01-29T22:26:50Z","receivedAt":"2013-01-29T22:26:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n>> > But how do these somewhat mediocre concatenated packs get turned into\n>> > real packs?\n>> \n>> How do they get processed in a fetch-only repositories that\n>> sometimes run \"gc --auto\" today? By runninng \"repack -a -d -f\"\n>> occasionally, perhaps?\n>\n> Do we run \"repack -adf\" regularly? The usual \"git gc\" procedure will not\n> use \"-f\", and without that, we will not even consider making deltas\n> between objects that were formerly in different packs (but now are in\n> the same pack).\n\nCorrect. It is not a new problem, and while I think it would need\nsome solution, the \"coalesce 50 small young packs into one\" is not\nan attempt to address it.\n\n> So you are avoiding doing medium-effort packs (\"repack -ad\") in favor of\n> doing potentially quick packs, but occasionally doing a big-effort pack\n> (\"repack -adf\").\n\nSo I think \"but occasionally\" part is not correct. In either way,\nthe packs use suboptimal delta, and you have to eventually pack with\n\"-f\", whether you coalesce 50 packs into 1 often or keep paying the\ncost of having 50 packs longer.\n\nThe trade-off is purely between \"one potentially quick coalescing\nper fetch in fetch-only repository\" vs \"any use of the data in the\nfetch-only repository (what do they do? build? serving gitweb\nlocally?) has to deal with 25 packs on average, and we still need to\npay medium repack cost every 50 fetches\".\n"},{"id":"208363","messageId":"CAJo=hJuGw8x=VrjWhvZhzakuhWrCWr2FRuEsNt5gQNC=6PPuVw@mail.gmail.com","threadId":"32735","inReplyTo":"20130129211932.GA17377@sigill.intra.peff.net","subject":"Re: [PATCH 0/2] optimizing pack access on \"read only\" fetch repos","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2013-01-31T16:47:37Z","receivedAt":"2013-01-31T16:47:37Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Tue, Jan 29, 2013 at 1:19 PM, Jeff King wrote:\n> On Tue, Jan 29, 2013 at 07:58:01AM -0800, Junio C Hamano wrote:\n>\n>> The point is not about space. Disk is cheap, and it is not making\n>> it any worse than what happens to your target audience, that is a\n>> fetch-only repository with only \"gc --auto\" in it, where nobody\n>> passes \"-f\" to \"repack\" to cause recomputation of delta.\n>>\n>> What I was trying to seek was a way to reduce the runtime penalty we\n>> pay every time we run git in such a repository.\n>>\n>> - Object look-up cost will become log2(50*n) from 50*log2(n), which\n>> is about 50/log2(50) improvement;\n>\n> Yes and no. Our heuristic is to look at the last-used pack for an\n> object. So assuming we have locality of requests, we should quite often\n> get \"lucky\" and find the object in the first log2 search. Even if we\n> don't assume locality, a situation with one large pack and a few small\n> packs will have the large one as \"last used\" more often than the others,\n> and it will also have the looked-for object more often than the others\n\nOpening all of those files does impact performance. It depends on how\nslow your open(2) syscall is. I know on Mac OS X that its not the\nfastest function we get from the C library. Performing ~40 opens to\nlook through the most recent pack files and finally find the \"real\"\npack that contains that tag you asked `git show` for isn't that quick.\n\nSome of us also use Git on filesystems that are network based, and\nslow compared to local disk Linux ext2/3/4 with gobs of free RAM.\n\n> So I can see how it is something we could potentially optimize, but I\n> could also see it being surprisingly not a big deal. I'd be very\n> interested to see real measurements, even of something as simple as a\n> \"master index\" which can reference multiple packfiles.\n\nI actually tried this many many years ago. There are threads in the\narchive about it. Its slower. We ruled it out.\n\n>> - System resource cost we incur by having to keep 50 file\n>> descriptors open and maintaining 50 mmap windows will reduce by\n>> 50 fold.\n>\n> I wonder how measurable that is (and if it matters on Linux versus less\n> efficient platforms).\n\nIt does matter. We know it has a negative impact on JGit even on Linux\nfor example. You don't want 300 packs in a repository. 50 might be\ntolerable. 300 is not.\n"},{"id":"208459","messageId":"20130201091431.GC30644@sigill.intra.peff.net","threadId":"32735","inReplyTo":"CAJo=hJuGw8x=VrjWhvZhzakuhWrCWr2FRuEsNt5gQNC=6PPuVw@mail.gmail.com","subject":"Re: [PATCH 0/2] optimizing pack access on \"read only\" fetch repos","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2013-02-01T09:14:31Z","receivedAt":"2013-02-01T09:14:31Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jan 31, 2013 at 08:47:37AM -0800, Shawn O. Pearce wrote:\n\n> >> - System resource cost we incur by having to keep 50 file\n> >> descriptors open and maintaining 50 mmap windows will reduce by\n> >> 50 fold.\n> >\n> > I wonder how measurable that is (and if it matters on Linux versus less\n> > efficient platforms).\n> \n> It does matter. We know it has a negative impact on JGit even on Linux\n> for example. You don't want 300 packs in a repository. 50 might be\n> tolerable. 300 is not.\n\nI'd love to see numbers if you have them. It's not that I don't believe\nit is slower, but knowing _how much_ is important when thinking about\nwhat kind of performance increase we are looking to get (which in turn\nimpacts how much effort to put into the repacking).\n\n-Peff\n"},{"id":"208522","messageId":"CAJo=hJvPz_EYFGzb8LLdW77T2MhcOqnUBtroLMWKkEWWKSTchQ@mail.gmail.com","threadId":"32735","inReplyTo":"20130201091431.GC30644@sigill.intra.peff.net","subject":"Re: [PATCH 0/2] optimizing pack access on \"read only\" fetch repos","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2013-02-02T10:07:49Z","receivedAt":"2013-02-02T10:07:49Z","isPatch":true,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"On Fri, Feb 1, 2013 at 1:14 AM, Jeff King wrote:\n> On Thu, Jan 31, 2013 at 08:47:37AM -0800, Shawn O. Pearce wrote:\n>\n>> >> - System resource cost we incur by having to keep 50 file\n>> >> descriptors open and maintaining 50 mmap windows will reduce by\n>> >> 50 fold.\n>> >\n>> > I wonder how measurable that is (and if it matters on Linux versus less\n>> > efficient platforms).\n>>\n>> It does matter. We know it has a negative impact on JGit even on Linux\n>> for example. You don't want 300 packs in a repository. 50 might be\n>> tolerable. 300 is not.\n>\n> I'd love to see numbers if you have them. It's not that I don't believe\n> it is slower, but knowing _how much_ is important when thinking about\n> what kind of performance increase we are looking to get (which in turn\n> impacts how much effort to put into the repacking).\n\nNever done a formal experiment. Just working from memory where 4 years\nand 3 desks ago I got called because one of our Git servers was\nstruggling to keep up with user git fetch commands. Turns out the\nrepository had O(200) pack files. git gc that normally took only about\n5 minutes took a hellva lot longer, like an hour or more.\n\nThe problem happened because the server was saving every push to a\npack and never exploding incoming packs to loose objects. This meant\nthe thin packs from the wire got delta bases appended to them.\npack-objects was looking at O(50) different alternatives for most\nobjects when trying to decide which one it should copy into the output\npack... for either a fetch/clone client, or the gc I was trying to run\nto fix the repository.\n"},{"id":"324292","messageId":"20170712200054.mxcabiyttijpbkbb@sigill.intra.peff.net","threadId":"32735","inReplyTo":"87bmopzbqx.fsf@gmail.com","subject":"Re: git gc --auto aquires *.lock files that make a subsequent git-fetch error out","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-07-12T20:00:54Z","receivedAt":"2017-07-12T20:01:01Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 12, 2017 at 09:38:46PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> In 131b8fcbfb (\"fetch: run gc --auto after fetching\", 2013-01-26) first\n> released with v1.8.2 Jeff changed git-fetch to run \"git gc --auto\"\n> afterwards.\n> \n> This means that if you run two git fetches in a row the second one may\n> fail because it can't acquire the *.lock files on the remote branches you\n> have & which the next git-fetch needs to update.\n\nIs it really \"in a row\" that's a problem? The second fetch should not\nbegin until the first one is done, including until its auto-gc exits.\nAnd even with background gc, we do the ref-locking operations first, due\nto 62aad1849 (gc --auto: do not lock refs in the background,\n2014-05-25).\n\n> I happen to run into this on a git.git which has a lot of remotes (most\n> people on-list whose remotes I know about) and fetch them in parallel:\n> \n> $ git config alias.pfetch\n> !parallel 'git fetch {}' ::: $(git remote)\n\nAh, so it's not in a row. It's parallel. Then yes, you may run into\nproblems with the gc locks conflicting with real operations. This isn't\nreally unique to fetch. Any simultaneous operation can run into problems\n(e.g., on a busy server repo you may see conflicts between pack-refs and\nregular pushes).\n\n> And so would 'git fetch --all':\n> \n> $ GIT_TRACE=1 git fetch --all 2>&1|grep --line-buffered built-in|grep -v rev-list\n> 19:31:26.273577 git.c:328 trace: built-in: git 'fetch' '--all'\n> 19:31:26.278869 git.c:328 trace: built-in: git 'fetch' '--append' 'origin'\n> 19:31:27.993312 git.c:328 trace: built-in: git 'gc' '--auto'\n> 19:31:27.995855 git.c:328 trace: built-in: git 'fetch' '--append' 'avar'\n> 19:31:29.656925 git.c:328 trace: built-in: git 'gc' '--auto'\n> \n> I think those two cases are bugs (but ones which I don't have the\n> inclination to chase myself beyond sending this E-Mail). We should be\n> running the 'git gc --auto' at the very end of the entire program, not\n> after fetching every single remote.\n> \n> Passing some env variable (similar to the config we pass via the env) to\n> subprograms to make them avoid \"git gc --auto\" so the main process can\n> do it would probably be the most simple solution.\n\nYes, I agree that's poor. Ideally there would be a command-line option\nto tell the sub-fetches not to run auto-gc. It could be done with:\n\n git -c gc.auto=0 fetch --append ...\n\nOr we could even take the \"--append\" as a hint not to run auto-gc.\n\n> The more general case (such as with my parallel invocation) is harder to\n> solve.\n\nYes, I don't think it can solved. The most general case is two totally\nunrelated processes which know nothing about each other.\n\n> Maybe \"git gc --auto\" should have a heuristic so it checks whether\n> there's been recent activity on the repo, and waits until there's been\n> say 60 seconds of no activity, or alternatively if it's waited 600\n> seconds and hasn't run gc yet.\n\nThat sounds complicated.\n\n> Ideally a \"real\" invocation like git-fetch would have a way to simply\n> steal any *.lock a background \"git gc --auto\" creates, aborting the gc\n> but allowing the \"real\" invocation to proceed. But that sounds even\n> trickier to implement, and might without an extra heuristic on top\n> postpone gc indefinitely.\n\nThe locks are generally due to ref-packing and reflog expiration. I\nthink in the long run, it would be nice to move to a ref store that\ndidn't need packing, and that could do reflog expiration more\natomically.\n\nI think the way \"reflog expire\" is done holds the locks for a lot longer\nthan is strictly necessary, too (it actually computes reachability for\n--expire-unreachable on the fly while holding some locks).\n\n-Peff\n"},{"id":"324293","messageId":"87a849z9cu.fsf@gmail.com","threadId":"32735","inReplyTo":"20170712200054.mxcabiyttijpbkbb@sigill.intra.peff.net","subject":"Re: git gc --auto aquires *.lock files that make a subsequent git-fetch error out","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2017-07-12T20:30:25Z","receivedAt":"2017-07-12T20:30:53Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Jul 12 2017, Jeff King jotted:\n\n> On Wed, Jul 12, 2017 at 09:38:46PM +0200, Ævar Arnfjörð Bjarmason wrote:\n>\n>> In 131b8fcbfb (\"fetch: run gc --auto after fetching\", 2013-01-26) first\n>> released with v1.8.2 Jeff changed git-fetch to run \"git gc --auto\"\n>> afterwards.\n>>\n>> This means that if you run two git fetches in a row the second one may\n>> fail because it can't acquire the *.lock files on the remote branches you\n>> have & which the next git-fetch needs to update.\n>\n> Is it really \"in a row\" that's a problem? The second fetch should not\n> begin until the first one is done, including until its auto-gc exits.\n> And even with background gc, we do the ref-locking operations first, due\n> to 62aad1849 (gc --auto: do not lock refs in the background,\n> 2014-05-25).\n>\n>> I happen to run into this on a git.git which has a lot of remotes (most\n>> people on-list whose remotes I know about) and fetch them in parallel:\n>>\n>> $ git config alias.pfetch\n>> !parallel 'git fetch {}' ::: $(git remote)\n>\n> Ah, so it's not in a row. It's parallel. Then yes, you may run into\n> problems with the gc locks conflicting with real operations. This isn't\n> really unique to fetch. Any simultaneous operation can run into problems\n> (e.g., on a busy server repo you may see conflicts between pack-refs and\n> regular pushes).\n\nThis is what I thought at first, and I've only encountered the issue in\nthis parallel mode (mainly because it's tedious to reproduce). But I\nthink the traces below show that it would happen with \"git fetch --all\"\n& \"git remote update\" as well, so the parallel invocations didn't\nmatter.\n\nI.e. I'd just update my first remote, then git-gc would start in the\nbackground and lock refs for my other remotes, which I'd then fail to\nupdate.\n\n>> And so would 'git fetch --all':\n>>\n>> $ GIT_TRACE=1 git fetch --all 2>&1|grep --line-buffered built-in|grep -v rev-list\n>> 19:31:26.273577 git.c:328 trace: built-in: git 'fetch' '--all'\n>> 19:31:26.278869 git.c:328 trace: built-in: git 'fetch' '--append' 'origin'\n>> 19:31:27.993312 git.c:328 trace: built-in: git 'gc' '--auto'\n>> 19:31:27.995855 git.c:328 trace: built-in: git 'fetch' '--append' 'avar'\n>> 19:31:29.656925 git.c:328 trace: built-in: git 'gc' '--auto'\n>>\n>> I think those two cases are bugs (but ones which I don't have the\n>> inclination to chase myself beyond sending this E-Mail). We should be\n>> running the 'git gc --auto' at the very end of the entire program, not\n>> after fetching every single remote.\n>>\n>> Passing some env variable (similar to the config we pass via the env) to\n>> subprograms to make them avoid \"git gc --auto\" so the main process can\n>> do it would probably be the most simple solution.\n>\n> Yes, I agree that's poor. Ideally there would be a command-line option\n> to tell the sub-fetches not to run auto-gc. It could be done with:\n>\n> git -c gc.auto=0 fetch --append ...\n>\n> Or we could even take the \"--append\" as a hint not to run auto-gc.\n>\n>> The more general case (such as with my parallel invocation) is harder to\n>> solve.\n>\n> Yes, I don't think it can solved. The most general case is two totally\n> unrelated processes which know nothing about each other.\n>\n>> Maybe \"git gc --auto\" should have a heuristic so it checks whether\n>> there's been recent activity on the repo, and waits until there's been\n>> say 60 seconds of no activity, or alternatively if it's waited 600\n>> seconds and hasn't run gc yet.\n>\n> That sounds complicated.\n>\n>> Ideally a \"real\" invocation like git-fetch would have a way to simply\n>> steal any *.lock a background \"git gc --auto\" creates, aborting the gc\n>> but allowing the \"real\" invocation to proceed. But that sounds even\n>> trickier to implement, and might without an extra heuristic on top\n>> postpone gc indefinitely.\n>\n> The locks are generally due to ref-packing and reflog expiration. I\n> think in the long run, it would be nice to move to a ref store that\n> didn't need packing, and that could do reflog expiration more\n> atomically.\n>\n> I think the way \"reflog expire\" is done holds the locks for a lot longer\n> than is strictly necessary, too (it actually computes reachability for\n> --expire-unreachable on the fly while holding some locks).\n>\n> -Peff\n"},{"id":"324296","messageId":"20170712204308.2emzv5y3rndflmpg@sigill.intra.peff.net","threadId":"32735","inReplyTo":"87a849z9cu.fsf@gmail.com","subject":"Re: git gc --auto aquires *.lock files that make a subsequent git-fetch error out","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-07-12T20:43:08Z","receivedAt":"2017-07-12T20:43:14Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Jul 12, 2017 at 10:30:25PM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> > Is it really \"in a row\" that's a problem? The second fetch should not\n> > begin until the first one is done, including until its auto-gc exits.\n> > And even with background gc, we do the ref-locking operations first, due\n> > to 62aad1849 (gc --auto: do not lock refs in the background,\n> > 2014-05-25).\n> >\n> >> I happen to run into this on a git.git which has a lot of remotes (most\n> >> people on-list whose remotes I know about) and fetch them in parallel:\n> >>\n> >> $ git config alias.pfetch\n> >> !parallel 'git fetch {}' ::: $(git remote)\n> >\n> > Ah, so it's not in a row. It's parallel. Then yes, you may run into\n> > problems with the gc locks conflicting with real operations. This isn't\n> > really unique to fetch. Any simultaneous operation can run into problems\n> > (e.g., on a busy server repo you may see conflicts between pack-refs and\n> > regular pushes).\n> \n> This is what I thought at first, and I've only encountered the issue in\n> this parallel mode (mainly because it's tedious to reproduce). But I\n> think the traces below show that it would happen with \"git fetch --all\"\n> & \"git remote update\" as well, so the parallel invocations didn't\n> matter.\n> \n> I.e. I'd just update my first remote, then git-gc would start in the\n> background and lock refs for my other remotes, which I'd then fail to\n> update.\n\nNo, it should be OK because of the commit I mentioned at the top of the\nquoted section. Each one runs sequentially.\n\n-Peff\n"}]}