{"thread":{"id":"36130","subject":"Borrowing objects from nearby repositories","startedAt":"2014-03-12T03:37:49Z","lastAt":"2014-03-28T17:02:31Z","messageCount":10,"participants":["Andrew Keller","Phil Hord","Ævar Arnfjörð Bjarmason","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"236557","messageId":"BFF5FBC7-8F53-4958-8D56-90EADD3AD626@kellerfarm.com","threadId":"36130","inReplyTo":null,"subject":"Borrowing objects from nearby repositories","fromName":"Andrew Keller","fromEmail":"andrew@kellerfarm.com","sentAt":"2014-03-12T03:37:49Z","receivedAt":"2014-03-12T03:37:49Z","isPatch":false,"sender":{"key":"andrew@kellerfarm.com","avatar":"https://avatars.githubusercontent.com/u/304204?v=4"},"body":"Hi all,\n\nI am considering developing a new feature, and I'd like to poll the group for opinions.\n\nBackground: A couple years ago, I wrote a set of scripts that speed up cloning of frequently used repositories. The scripts utilize a bare Git repository located at a known location, and automate providing a --reference parameter to `git clone` and `git submodule update`. Recently, some coworkers of mine expressed an interest in using the scripts, so I published the current version of my scripts, called `git repocache`, described at the bottom of .\n\nSlowly, it has occurred to me that this feature, or something similar to it, may be worth adding to Git, so I've been thinking about the best approach. Here's my best idea so far:\n\n1) Introduce '--borrow' to `git-fetch`. This would behave similarly to '--reference', except that it operates on a temporary basis, and does not assume that the reference repository will exist after the operation completes, so any used objects are copied into the local objects database. In theory, this mechanism would be distinct from '--reference', so if both are used, some objects would be copied, and some objects would be accessible via a reference repository referenced by the alternates file.\n\n2) Teach `git fetch` to read 'repocache.path' (or a better-named configuration), and use it to automatically activate borrowing.\n\n3) For consistency, `git clone`, `git pull`, and `git submodule update` should probably all learn '--borrow', and forward it to `git fetch`.\n\n4) In some scenarios, it may be necessary to temporarily not automatically borrow, so `git fetch`, and everything that calls it may need an argument to do that.\n\nIntended outcome: With 'repocache.path' set, and the cached repository properly updated, one could run `git clone `, and the operation would complete much faster than it does now due to less load on the network.\n\nThings I haven't figured out yet:\n\n* What's the best approach to copying the needed objects? It's probably inefficient to copy individual objects out of pack files one at a time, but it could be wasteful to copy entire pack files just because you need one object. Hard-linking could help, but that won't always be available. One of my previous ideas was to add a '--auto-repack' option to `git-clone`, which solves this problem better, but introduces some other front-end usability problems.\n* To maintain optimal effectiveness, users would have to regularly run a fetch in the cache repository. Not all users know how to set up a scheduled task on their computer, so this might become a maintenance problem for the user. This kind of problem I think brings into question the viability of the underlying design here, assuming that the ultimate goal is to clone faster, with very little or no change in the use of git.\n\n\nThoughts?\n\nThanks,\nAndrew Keller\n"},{"id":"237419","messageId":"CABURp0rKz9s7aPx_6ucTQ5C8NpPZMcJL3jaB_v_rQaBE+sFt1Q@mail.gmail.com","threadId":"36130","inReplyTo":"BFF5FBC7-8F53-4958-8D56-90EADD3AD626@kellerfarm.com","subject":"Re: Borrowing objects from nearby repositories","fromName":"Phil Hord","fromEmail":"phil.hord@gmail.com","sentAt":"2014-03-23T18:04:12Z","receivedAt":"2014-03-23T18:04:12Z","isPatch":false,"sender":{"key":"phil.hord@gmail.com","avatar":"https://avatars.githubusercontent.com/u/123908?v=4"},"body":"On Tue, Mar 11, 2014 at 11:37 PM, Andrew Keller wrote:\n> I am considering developing a new feature, and I'd like to poll the group for opinions.\n>\n> Background: A couple years ago, I wrote a set of scripts that speed up cloning of frequently used repositories. The scripts utilize a bare Git repository located at a known location, and automate providing a --reference parameter to `git clone` and `git submodule update`. Recently, some coworkers of mine expressed an interest in using the scripts, so I published the current version of my scripts, called `git repocache`, described at the bottom of .\n>\n> Slowly, it has occurred to me that this feature, or something similar to it, may be worth adding to Git, so I've been thinking about the best approach. Here's my best idea so far:\n>\n> 1) Introduce '--borrow' to `git-fetch`. This would behave similarly to '--reference', except that it operates on a temporary basis, and does not assume that the reference repository will exist after the operation completes, so any used objects are copied into the local objects database. In theory, this mechanism would be distinct from '--reference', so if both are used, some objects would be copied, and some objects would be accessible via a reference repository referenced by the alternates file.\n\nInteresting. I do something similar on my CI Server to reduce\nworkload on Gerrit. Having a built-in to support submodules would be\nnice. Currently my script does this:\n\nMIRROR=/path/to/local/mirror\nNEW=ssh://gerrit-server\ngit clone ${MIRROR}/project && cd project\n\n#-- Init/update submodules from our local mirror if possible\ngit submodule update --recursive --init\n\n#-- Switch to the remote server URL\ngit config remote.origin.url $(git config remote.origin.url|sed -e\n\"s|^${MIRROR}|${NEW}|\")\ngit submodule sync #--recursive ; recursive not supported :-[\n\n#-- Checkout remote updates\ngit pull --ff-only --recurse-submodules origin ${BRANCH}\ngit submodule update --recursive --init\n\n\nIs that about the same as you are aiming for?\n\n\n> 2) Teach `git fetch` to read 'repocache.path' (or a better-named configuration), and use it to automatically activate borrowing.\n\nSeems like this could be trouble if a local repo is coincidentally\nnamed the same as some unrelated repo you want to clone. But I can\nsee the value.\n\nWhat about something similar to url.insteadOf? Maybe\n'url.${SERVER}.autoBorrow = ${MIRROR}', with replacement semantics\nsimilar to insteadOf.\n\n> 3) For consistency, `git clone`, `git pull`, and `git submodule update` should probably all learn '--borrow', and forward it to `git fetch`.\n>\n> 4) In some scenarios, it may be necessary to temporarily not automatically borrow, so `git fetch`, and everything that calls it may need an argument to do that.\n\n--no-borrow\n\nPhil\n"},{"id":"237494","messageId":"CACBZZX5teZuqtNkPT4PdXJn=g34cOhRH2oNehROT8kJ_M2cgfg@mail.gmail.com","threadId":"36130","inReplyTo":"BFF5FBC7-8F53-4958-8D56-90EADD3AD626@kellerfarm.com","subject":"Re: Borrowing objects from nearby repositories","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2014-03-24T21:21:03Z","receivedAt":"2014-03-24T21:21:03Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Wed, Mar 12, 2014 at 4:37 AM, Andrew Keller wrote:\n> Hi all,\n>\n> I am considering developing a new feature, and I'd like to poll the group for opinions.\n>\n> Background: A couple years ago, I wrote a set of scripts that speed up cloning of frequently used repositories. The scripts utilize a bare Git repository located at a known location, and automate providing a --reference parameter to `git clone` and `git submodule update`. Recently, some coworkers of mine expressed an interest in using the scripts, so I published the current version of my scripts, called `git repocache`, described at the bottom of .\n>\n> Slowly, it has occurred to me that this feature, or something similar to it, may be worth adding to Git, so I've been thinking about the best approach. Here's my best idea so far:\n>\n> 1) Introduce '--borrow' to `git-fetch`. This would behave similarly to '--reference', except that it operates on a temporary basis, and does not assume that the reference repository will exist after the operation completes, so any used objects are copied into the local objects database. In theory, this mechanism would be distinct from '--reference', so if both are used, some objects would be copied, and some objects would be accessible via a reference repository referenced by the alternates file.\n\nIsn't this the same as git clone --reference --no-hardlinks ?\n\nAlso without --no-hardlinks we're not assuming that the other repo\ndoesn't go away (you could rm-rf it), just that the files won't be\n*modified*, which Git won't do, but you could manually do with other\ntools, so the default is to hardlink.\n\n> 2) Teach `git fetch` to read 'repocache.path' (or a better-named configuration), and use it to automatically activate borrowing.\n\nSo a default path for --reference --no-hardlinks ?\n\n> 3) For consistency, `git clone`, `git pull`, and `git submodule update` should probably all learn '--borrow', and forward it to `git fetch`.\n>\n> 4) In some scenarios, it may be necessary to temporarily not automatically borrow, so `git fetch`, and everything that calls it may need an argument to do that.\n>\n> Intended outcome: With 'repocache.path' set, and the cached repository properly updated, one could run `git clone `, and the operation would complete much faster than it does now due to less load on the network.\n>\n> Things I haven't figured out yet:\n>\n> * What's the best approach to copying the needed objects? It's probably inefficient to copy individual objects out of pack files one at a time, but it could be wasteful to copy entire pack files just because you need one object. Hard-linking could help, but that won't always be available. One of my previous ideas was to add a '--auto-repack' option to `git-clone`, which solves this problem better, but introduces some other front-end usability problems.\n> * To maintain optimal effectiveness, users would have to regularly run a fetch in the cache repository. Not all users know how to set up a scheduled task on their computer, so this might become a maintenance problem for the user. This kind of problem I think brings into question the viability of the underlying design here, assuming that the ultimate goal is to clone faster, with very little or no change in the use of git.\n>\n>\n> Thoughts?\n>\n> Thanks,\n> Andrew Keller\n>\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at http://vger.kernel.org/majordomo-info.html\n"},{"id":"237651","messageId":"9A24D2D1-DD59-41DC-8237-2B5829695753@kellerfarm.com","threadId":"36130","inReplyTo":"CACBZZX5teZuqtNkPT4PdXJn=g34cOhRH2oNehROT8kJ_M2cgfg@mail.gmail.com","subject":"Re: Borrowing objects from nearby repositories","fromName":"Andrew Keller","fromEmail":"andrew@kellerfarm.com","sentAt":"2014-03-25T13:13:56Z","receivedAt":"2014-03-25T13:13:56Z","isPatch":false,"sender":{"key":"andrew@kellerfarm.com","avatar":"https://avatars.githubusercontent.com/u/304204?v=4"},"body":"On Mar 24, 2014, at 5:21 PM, Ævar Arnfjörð Bjarmason wrote:\n> On Wed, Mar 12, 2014 at 4:37 AM, Andrew Keller wrote:\n>> Hi all,\n>> \n>> I am considering developing a new feature, and I'd like to poll the group for opinions.\n>> \n>> Background: A couple years ago, I wrote a set of scripts that speed up cloning of frequently used repositories. The scripts utilize a bare Git repository located at a known location, and automate providing a --reference parameter to `git clone` and `git submodule update`. Recently, some coworkers of mine expressed an interest in using the scripts, so I published the current version of my scripts, called `git repocache`, described at the bottom of .\n>> \n>> Slowly, it has occurred to me that this feature, or something similar to it, may be worth adding to Git, so I've been thinking about the best approach. Here's my best idea so far:\n>> \n>> 1) Introduce '--borrow' to `git-fetch`. This would behave similarly to '--reference', except that it operates on a temporary basis, and does not assume that the reference repository will exist after the operation completes, so any used objects are copied into the local objects database. In theory, this mechanism would be distinct from '--reference', so if both are used, some objects would be copied, and some objects would be accessible via a reference repository referenced by the alternates file.\n> \n> Isn't this the same as git clone --reference --no-hardlinks ?\n\n'--reference` adds an entry to 'info/alternates' inside the objects folder. When an object is looked up, any objects folder listed in 'objects/info/alternates' is considered to be an extension of the local objects folder. So, when, for example, fetch runs, when it goes to decide whether or not it already has a blob locally, it may decide \"yes\", and not download the blob at all, because it already exists in one of the reference repositories. If I clone one of my 80 GB repositories over SSH using a reference repository, the resulting clone is only about 175 KB, because it's assuming the reference repository will exist going forward, so it doesn't actually own any objects itself at all.\n\nThe '--no-hardlinks' option is only applicable when hard linking is available in the first place - i.e., when cloning from one local folder to another on the same filesystem (assuming the filesystem supports hard links).\n\nThanks,\n - Andrew\n"},{"id":"237665","messageId":"xmqqtxammctc.fsf@gitster.dls.corp.google.com","threadId":"36130","inReplyTo":"CACBZZX5teZuqtNkPT4PdXJn=g34cOhRH2oNehROT8kJ_M2cgfg@mail.gmail.com","subject":"Re: Borrowing objects from nearby repositories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-03-25T17:02:23Z","receivedAt":"2014-03-25T17:02:23Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason writes:\n\n>> 1) Introduce '--borrow' to `git-fetch`. This would behave similarly\n> to '--reference', except that it operates on a temporary basis, and\n> does not assume that the reference repository will exist after the\n> operation completes, so any used objects are copied into the local\n> objects database. In theory, this mechanism would be distinct from\n> --reference', so if both are used, some objects would be copied, and\n> some objects would be accessible via a reference repository referenced\n> by the alternates file.\n>\n> Isn't this the same as git clone --reference --no-hardlinks\n> ?\n>\n> Also without --no-hardlinks we're not assuming that the other repo\n> doesn't go away (you could rm-rf it), just that the files won't be\n> *modified*, which Git won't do, but you could manually do with other\n> tools, so the default is to hardlink.\n\nI think that the standard practice with the existing toolset is to\nclone with reference and then repack. That is:\n\n $ git clone --reference git://over/there mine\n $ cd mine\n $ git repack -a -d\n\nAnd then you can try this:\n\n $ mv .git/objects/info/alternates .git/objects/info/alternates.disabled\n $ git fsck\n\nto make sure that you are no longer borrowing anything from the\nborrowee. Once you are satisfied, you can remove the saved-away\nalternates.disabled file.\n"},{"id":"237777","messageId":"xmqqvbv1kjoc.fsf@gitster.dls.corp.google.com","threadId":"36130","inReplyTo":"xmqqtxammctc.fsf@gitster.dls.corp.google.com","subject":"Re: Borrowing objects from nearby repositories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-03-25T22:17:07Z","receivedAt":"2014-03-25T22:17:07Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Ævar Arnfjörð Bjarmason writes:\n>\n>>> 1) Introduce '--borrow' to `git-fetch`. This would behave similarly\n>> to '--reference', except that it operates on a temporary basis, and\n>> does not assume that the reference repository will exist after the\n>> operation completes, so any used objects are copied into the local\n>> objects database. In theory, this mechanism would be distinct from\n>> --reference', so if both are used, some objects would be copied, and\n>> some objects would be accessible via a reference repository referenced\n>> by the alternates file.\n>>\n>> Isn't this the same as git clone --reference --no-hardlinks\n>> ?\n>>\n>> Also without --no-hardlinks we're not assuming that the other repo\n>> doesn't go away (you could rm-rf it), just that the files won't be\n>> *modified*, which Git won't do, but you could manually do with other\n>> tools, so the default is to hardlink.\n>\n> I think that the standard practice with the existing toolset is to\n> clone with reference and then repack. That is:\n>\n> $ git clone --reference git://over/there mine\n> $ cd mine\n> $ git repack -a -d\n>\n> And then you can try this:\n>\n> $ mv .git/objects/info/alternates .git/objects/info/alternates.disabled\n> $ git fsck\n>\n> to make sure that you are no longer borrowing anything from the\n> borrowee. Once you are satisfied, you can remove the saved-away\n> alternates.disabled file.\n\nOh, I forgot to say that I am not opposed if somebody wants to teach\n\"git clone\" a new option to copy its objects from two places,\n(hopefully) the majority from near-by reference repository and the\nremainder over the network, without permanently relying on the\nformer via the alternates mechanism. The implementation of such a\nfeature could even literally be \"clone with reference first and then\nrepack\" at least initially but even in the final version.\n"},{"id":"237799","messageId":"3533946C-DE97-4214-9B55-F5B788DDD952@kellerfarm.com","threadId":"36130","inReplyTo":"xmqqvbv1kjoc.fsf@gitster.dls.corp.google.com","subject":"Re: Borrowing objects from nearby repositories","fromName":"Andrew Keller","fromEmail":"andrew@kellerfarm.com","sentAt":"2014-03-26T13:36:09Z","receivedAt":"2014-03-26T13:36:09Z","isPatch":false,"sender":{"key":"andrew@kellerfarm.com","avatar":"https://avatars.githubusercontent.com/u/304204?v=4"},"body":"On Mar 25, 2014, at 6:17 PM, Junio C Hamano wrote:\n\n> Junio C Hamano writes:\n> \n>> Ævar Arnfjörð Bjarmason writes:\n>> \n>>>> 1) Introduce '--borrow' to `git-fetch`. This would behave similarly\n>>> to '--reference', except that it operates on a temporary basis, and\n>>> does not assume that the reference repository will exist after the\n>>> operation completes, so any used objects are copied into the local\n>>> objects database. In theory, this mechanism would be distinct from\n>>> --reference', so if both are used, some objects would be copied, and\n>>> some objects would be accessible via a reference repository referenced\n>>> by the alternates file.\n>>> \n>>> Isn't this the same as git clone --reference --no-hardlinks\n>>> ?\n>>> \n>>> Also without --no-hardlinks we're not assuming that the other repo\n>>> doesn't go away (you could rm-rf it), just that the files won't be\n>>> *modified*, which Git won't do, but you could manually do with other\n>>> tools, so the default is to hardlink.\n>> \n>> I think that the standard practice with the existing toolset is to\n>> clone with reference and then repack. That is:\n>> \n>> $ git clone --reference git://over/there mine\n>> $ cd mine\n>> $ git repack -a -d\n>> \n>> And then you can try this:\n>> \n>> $ mv .git/objects/info/alternates .git/objects/info/alternates.disabled\n>> $ git fsck\n>> \n>> to make sure that you are no longer borrowing anything from the\n>> borrowee. Once you are satisfied, you can remove the saved-away\n>> alternates.disabled file.\n> \n> Oh, I forgot to say that I am not opposed if somebody wants to teach\n> \"git clone\" a new option to copy its objects from two places,\n> (hopefully) the majority from near-by reference repository and the\n> remainder over the network, without permanently relying on the\n> former via the alternates mechanism. The implementation of such a\n> feature could even literally be \"clone with reference first and then\n> repack\" at least initially but even in the final version.\n\nThat was actually one of my first ideas - adding some sort of '--auto-repack' option to git-clone. It's a relatively small change, and would work. However, keeping in mind my end goal of automating the feature to the point where you could run simply 'git clone ', an '--auto-repack' option is more difficult to undo. You would need a new parameter to disable the automatic adding of reference repositories, and a new parameter to undo '--auto-repack', and you'd have to remember to actually undo both of those settings.\n\nIn contrast, if the new feature was '--borrow', and the evolution of the feature was a global configuration 'fetch.autoBorrow', then to turn it off temporarily, one only needs a single new parameter '--no-auto-borrow'. I think this is a cleaner approach than the former, although much more work.\n\nThanks,\n - Andrew Keller\n"},{"id":"237819","messageId":"xmqqbnwskgwd.fsf@gitster.dls.corp.google.com","threadId":"36130","inReplyTo":"3533946C-DE97-4214-9B55-F5B788DDD952@kellerfarm.com","subject":"Re: Borrowing objects from nearby repositories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-03-26T17:29:22Z","receivedAt":"2014-03-26T17:29:22Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Andrew Keller writes:\n\n> On Mar 25, 2014, at 6:17 PM, Junio C Hamano wrote:\n> ...\n>>> I think that the standard practice with the existing toolset is to\n>>> clone with reference and then repack. That is:\n>>> \n>>> $ git clone --reference git://over/there mine\n>>> $ cd mine\n>>> $ git repack -a -d\n>>> \n>>> And then you can try this:\n>>> \n>>> $ mv .git/objects/info/alternates .git/objects/info/alternates.disabled\n>>> $ git fsck\n>>> \n>>> to make sure that you are no longer borrowing anything from the\n>>> borrowee. Once you are satisfied, you can remove the saved-away\n>>> alternates.disabled file.\n>> \n>> Oh, I forgot to say that I am not opposed if somebody wants to teach\n>> \"git clone\" a new option to copy its objects from two places,\n>> (hopefully) the majority from near-by reference repository and the\n>> remainder over the network, without permanently relying on the\n>> former via the alternates mechanism. The implementation of such a\n>> feature could even literally be \"clone with reference first and then\n>> repack\" at least initially but even in the final version.\n\n[Administrivia: please wrap your lines to a reasonable length]\n\n> That was actually one of my first ideas - adding some sort of\n> '--auto-repack' option to git-clone. It's a relatively small\n> change, and would work. However, keeping in mind my end goal of\n> automating the feature to the point where you could run simply\n> 'git clone ', an '--auto-repack' option is more difficult to\n> undo. You would need a new parameter to disable the automatic\n> adding of reference repositories, and a new parameter to undo\n> '--auto-repack', and you'd have to remember to actually undo both\n> of those settings.\n>\n> In contrast, if the new feature was '--borrow', and the evolution\n> of the feature was a global configuration 'fetch.autoBorrow', then\n> to turn it off temporarily, one only needs a single new parameter\n> '--no-auto-borrow'. I think this is a cleaner approach than the\n> former, although much more work.\n\nI think you may have misread me. With the \"new option\", I was\nhinting that the \"clone --reference && repack && rm alternates\"\nwill be an acceptable internal implementation of the \"--borrow\"\noption that was mentioned in the thread. I am not sure where you\ngot the \"auto-repack\" from.\n\nOne of the reasons you may have misread me may be because I made it\nsound as if \"this may work and when it works you will be happy, but\nif it does not work you did not lose very much\" by mentioning \"mv &&\nfsck\". That wasn't what I meant.\n\nThe \"repack -a\" procedure is to make the borrower repository no\nlonger dependent on the borrowee, and it is supposed to always work.\nIn fact, this behaviour was the whole reason why \"repack\" later\nlearned its \"-l\" option to disable it, because people who cloned\nwith \"--reference\" in order to reduce the disk footprint by sharing\nolder and more common objects [*1*] were rightfully surprised to see\nthat the borrowed objects were copied over to their borrower\nrepository when they ran \"repack\" [*2*].\n\nBecause this is \"clone\", there is nothing complex to \"undo\". Either\nit succeeds, or you remove the whole new directory if anything\nfails.\n\nI said \"even in the final version\" for a simple reason: you cannot\ncannot do realistically any better than the \"clone --reference &&\nrepack -a d && rm alternates\" sequence.\n\nBut you would need to know a few things about how Git works in order\nto come to that realisation. Here are some:\n\n * \"clone --borrow\" (or whatever we end up calling the option) must\n talk to two repositories:\n\n - We will need to have one upload-pack session with the distant\n origin repository over the network, which will send a complete\n pack.\n\n - We need to also copy objects that weren't sent from the\n distant origin to our repository from the reference one.\n\n * A single \"repack -a -d\" (without \"-l\") after \"clone --reference\"\n is already a way to do exactly what you need---enumerate what are\n missing in the packfile that was received from the distant origin\n and come up with packfile(s) that contain all and only objects\n the cloned repository needs.\n\n * You cannot easily concatenate multiple packfiles into a single\n one (or append runs of objects to an existing packfile) to come\n up with a single packfile.\n\nYou _could_ shoehorn the logic to \"enumerate and read from the\nreference, and append them at the end of the packfile received from\nthe distant origin repository\" into the part that talks to the\ndistant origin repository, but the object layout in the resulting\npackfile will be suboptimal [*3*] and the code complexity required\nto do so is not worth it [*4*].\n\n\n[Footnotes]\n\n*1* From the point of view of supporting both camps, i.e. those who\n want their borrower repositories to keep sharing the objects\n with the borrowee repository and those who want to use a\n borrowee repository temporarily while cloning only to reduce the\n network cost from the distant upstream, the current option name\n \"--reference\" and the proposed name \"--borrow\" are backwards.\n The folks who want the original behaviour of keep depending want\n to \"borrow\" from the borrowee repository; those who want to\n utilize the mechanism temporarily only while cloning would want\n to merely \"reference\" it only while cloning.\n\n*2* A repository created with \"clone --reference\" may want to set a\n configuration variable in it to tell future invocations of \"git\n repack\" to use the \"-l\" option by default, while allowing those\n who want to fatten such a repository to override it with \"repack\n --no-local\". Without such an arrangement, we would risk the\n people who wanted that \"-l\" option for \"repack\" in the first\n place to accidentally fatten their lean repositories by mistake,\n forgetting to pass the \"-l\" option. Luckily \"gc\" always runs\n \"repack\" with \"-l\", so the risk is limited to those who run\n \"repack\" themselves, which may be why we heard no complaints on\n this point.\n\n*3* In fact, another program invoked during the object transfer\n \"index-pack --fix-thin\" does have \"append selected objects at\n the end of the packfile received over the wire and fix up the\n whole thing\" logic in it). The pack that results from it\n suffers from the suboptimal layout because the appended objects\n are \"appended\", not placed in their optimal positions in the\n packfile to reduce seeks. \n\n If \"--borrow\" did the same, the pack layout issue will be worse,\n because the whole point of \"--borrow\" is to borrow the majority\n of objects from reference repository---we will be appending a\n lot from the reference to a relatively small pack we receive\n over the wire from the distant origin.\n\n*4* The complexity of the code to implement \"index-pack --fix-thin\"\n is not pretty, but it will be worse for \"--borrow\", as the\n former at least does not have to walk the dag to find out what\n objects need to be appended but the latter does.\n"},{"id":"238009","messageId":"8030ADEA-B11F-47E8-AFE7-8F46E861F560@kellerfarm.com","threadId":"36130","inReplyTo":"xmqqbnwskgwd.fsf@gitster.dls.corp.google.com","subject":"Re: Borrowing objects from nearby repositories","fromName":"Andrew Keller","fromEmail":"andrew@kellerfarm.com","sentAt":"2014-03-28T14:52:20Z","receivedAt":"2014-03-28T14:52:20Z","isPatch":false,"sender":{"key":"andrew@kellerfarm.com","avatar":"https://avatars.githubusercontent.com/u/304204?v=4"},"body":"On Mar 26, 2014, at 1:29 PM, Junio C Hamano wrote:\n\n> Andrew Keller writes:\n> \n>> On Mar 25, 2014, at 6:17 PM, Junio C Hamano wrote:\n>> ...\n>>>> I think that the standard practice with the existing toolset is to\n>>>> clone with reference and then repack. That is:\n>>>> \n>>>> $ git clone --reference git://over/there mine\n>>>> $ cd mine\n>>>> $ git repack -a -d\n>>>> \n>>>> And then you can try this:\n>>>> \n>>>> $ mv .git/objects/info/alternates .git/objects/info/alternates.disabled\n>>>> $ git fsck\n>>>> \n>>>> to make sure that you are no longer borrowing anything from the\n>>>> borrowee. Once you are satisfied, you can remove the saved-away\n>>>> alternates.disabled file.\n>>> \n>>> Oh, I forgot to say that I am not opposed if somebody wants to teach\n>>> \"git clone\" a new option to copy its objects from two places,\n>>> (hopefully) the majority from near-by reference repository and the\n>>> remainder over the network, without permanently relying on the\n>>> former via the alternates mechanism. The implementation of such a\n>>> feature could even literally be \"clone with reference first and then\n>>> repack\" at least initially but even in the final version.\n> \n> [Administrivia: please wrap your lines to a reasonable length]\n> \n>> That was actually one of my first ideas - adding some sort of\n>> '--auto-repack' option to git-clone. It's a relatively small\n>> change, and would work. However, keeping in mind my end goal of\n>> automating the feature to the point where you could run simply\n>> 'git clone ', an '--auto-repack' option is more difficult to\n>> undo. You would need a new parameter to disable the automatic\n>> adding of reference repositories, and a new parameter to undo\n>> '--auto-repack', and you'd have to remember to actually undo both\n>> of those settings.\n>> \n>> In contrast, if the new feature was '--borrow', and the evolution\n>> of the feature was a global configuration 'fetch.autoBorrow', then\n>> to turn it off temporarily, one only needs a single new parameter\n>> '--no-auto-borrow'. I think this is a cleaner approach than the\n>> former, although much more work.\n> \n> I think you may have misread me. With the \"new option\", I was\n> hinting that the \"clone --reference && repack && rm alternates\"\n> will be an acceptable internal implementation of the \"--borrow\"\n> option that was mentioned in the thread. I am not sure where you\n> got the \"auto-repack\" from.\n\nAh, yes - that is better than what I was thinking. I was thinking a bit\ntoo low-level, and using two arguments in the place of your one.\n\n> One of the reasons you may have misread me may be because I made it\n> sound as if \"this may work and when it works you will be happy, but\n> if it does not work you did not lose very much\" by mentioning \"mv &&\n> fsck\". That wasn't what I meant.\n> \n> The \"repack -a\" procedure is to make the borrower repository no\n> longer dependent on the borrowee, and it is supposed to always work.\n> In fact, this behaviour was the whole reason why \"repack\" later\n> learned its \"-l\" option to disable it, because people who cloned\n> with \"--reference\" in order to reduce the disk footprint by sharing\n> older and more common objects [*1*] were rightfully surprised to see\n> that the borrowed objects were copied over to their borrower\n> repository when they ran \"repack\" [*2*].\n> \n> Because this is \"clone\", there is nothing complex to \"undo\". Either\n> it succeeds, or you remove the whole new directory if anything\n> fails.\n> \n> I said \"even in the final version\" for a simple reason: you cannot\n> cannot do realistically any better than the \"clone --reference &&\n> repack -a d && rm alternates\" sequence.\n\nWow, that's very insightful - thanks! So, it sounds like I was right about\nthe general areas of concern when trying to do this during a fetch, but\nI underestimated just how complicated it would be.\n\nOkay, so to re-frame my idea, like you said, the goal is to find a user-\nfriendly way for the user to tell git-clone to set up the alternates file\n(or perhaps just use the --alternates parameter), and run a repack,\nand disconnect the alternate. And yet, we still want to be able to use\n--reference on its own, because there are existing use cases for that.\n\nThanks!\n - Andrew Keller\n"},{"id":"238015","messageId":"xmqq8urucl3s.fsf@gitster.dls.corp.google.com","threadId":"36130","inReplyTo":"8030ADEA-B11F-47E8-AFE7-8F46E861F560@kellerfarm.com","subject":"Re: Borrowing objects from nearby repositories","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2014-03-28T17:02:31Z","receivedAt":"2014-03-28T17:02:31Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Andrew Keller writes:\n\n> Okay, so to re-frame my idea, like you said, the goal is to find a user-\n> friendly way for the user to tell git-clone to set up the alternates file\n> (or perhaps just use the --alternates parameter), and run a repack,\n> and disconnect the alternate. And yet, we still want to be able to use\n> --reference on its own, because there are existing use cases for that.\n\nHere are a few possible action items that came out of this\ndiscussion:\n\n 1. Introduce a new \"--borrow\" option to \"git clone\".\n\n The updates to the SYNOPSIS section may go like this:\n\n -'git clone' [--reference ] ...other options...\n +'git clone' [[--reference|--borrow] ] ...other options...\n\n The new option can be used instead of \"--reference\" and they\n will be mutually incompatible. The first implementation of the\n \"--borrow\" option would do the following:\n\n (1) run the same \"git clone\" with the same command line but\n replacing \"--borrow\" with \"--reference\"; if this fails, exit\n with the same failure.\n\n (2) in the resulting repository, run \"git repack -a -d\"; if this\n fails, remove the entire directory the first step created,\n and exit with failure.\n\n (3) remove .git/objects/info/alternates from the resulting\n repository and exit with success.\n\n and it may be acceptable as the final implementation as well.\n\n\n 2. Make \"git repack\" safer for the users of \"clone --reference\" who\n want to keep sharing objects from the original.\n\n - Introduce the \"repack.local\" configuration variable that can\n be set to either true or false. Missing variable defaults to\n \"false\". \n\n - A \"repack\" that is run without \"-l\" option on the command line\n will pretend as if it was given \"-l\" from the command line if\n \"repack.local\" is set to \"true\". Add \"repack --no-local\"\n option to countermand this configuration variable from the\n command line.\n\n - Teach \"git clone --reference\" (but not \"git clone --borrow\")\n to set \"repack.local = true\" in the configuration of the\n resulting repository.\n"}]}