{"thread":{"id":"46362","subject":"Mirroring for offline use - best practices?","startedAt":"2017-07-12T10:54:31Z","lastAt":"2017-07-12T22:14:10Z","messageCount":3,"participants":["Joachim Durchholz","Stefan Beller"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"324259","messageId":"81052ff7-cece-400d-1742-e13693fae7e3@durchholz.org","threadId":"46362","inReplyTo":null,"subject":"Mirroring for offline use - best practices?","fromName":"Joachim Durchholz","fromEmail":"jo@durchholz.org","sentAt":"2017-07-12T10:47:52Z","receivedAt":"2017-07-12T10:54:31Z","isPatch":false,"sender":{"key":"jo@durchholz.org","avatar":null},"body":"Hi all,\n\nI'm pretty sure this is a FAQ, but articles I found on the Internet were \neither mere \"recipes\" (i.e. tell you how, but don't explain why), or \nbogged down in so many details that I was never sure how to proceed from \nthere.\n\n\nBasic situation:\n\nThere's a master repository (Github or corporate or whatever), and I \nwant to set up a local mirror so that I can create clones without having \nto access the original upstream.\nI'd like to set the mirror up so that creating a clone from it will \nautomatically set up things to \"just work\": I.e. branches will track the \nmirror, not upstream, possibly other settings that I'm not aware of.\n\nI gather that local clones are fast because hardlinked - is that correct?\nIs that correct on Windows? (I can't easily avoid Windows.)\n\n\nRamification 1:\n\nI'm not sure how best to prepare patches for push-to-upstream.\nIs there value in collecting them locally into a push-to-upstream repo, \nor is it better to just push from each local clone individually?\n\n\nRamification 2:\n\nSome of the repos I work with use submodules. Sometimes they use \nsubmodules that I'm not aware of. Or a submodule was used historically, \nand git bisect breaks/misbehaves because it can't get the submodule in \noffline mode.\nIs there a way to get these, without writing a script that recurses \nthrough all versions of .gitmodules?\nI'm seeing the --recurse-submodules option for git fetch, so this might \n(or might not) be the Right Thing.\n\n\nAny thoughts welcome, thanks!\n\nRegards,\nJo\n"},{"id":"324275","messageId":"CAGZ79ka59qkekqGCSmVHb0=H--4HtjC7Ts9eXj_hb6f2cRQ4ww@mail.gmail.com","threadId":"46362","inReplyTo":"81052ff7-cece-400d-1742-e13693fae7e3@durchholz.org","subject":"Re: Mirroring for offline use - best practices?","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-07-12T17:40:58Z","receivedAt":"2017-07-12T17:41:06Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Jul 12, 2017 at 3:47 AM, Joachim Durchholz <jo@durchholz.org> wrote:\n> Hi all,\n>\n> I'm pretty sure this is a FAQ, but articles I found on the Internet were\n> either mere \"recipes\" (i.e. tell you how, but don't explain why), or bogged\n> down in so many details that I was never sure how to proceed from there.\n>\n>\n> Basic situation:\n>\n> There's a master repository (Github or corporate or whatever), and I want to\n> set up a local mirror so that I can create clones without having to access\n> the original upstream.\n\n'git clone --mirror' should accomplish the mirroring part.\n\n> I'd like to set the mirror up so that creating a clone from it will\n> automatically set up things to \"just work\": I.e. branches will track the\n> mirror, not upstream, possibly other settings that I'm not aware of.\n\nAnd then 'git clone <local-path-to-mirror>'. This would setup the local\nmirror as upstream, such that git-fetch would fetch from the\nlocal mirror. However git-push would also go to the mirror. I am not\nsure if this is desired or if you rather desire a triangular workflow, i.e.\nthe local clone would directly push back to the real upstream.\nThat can be configured with url.<base>.pushInsteadOf, but there\nis no way to have that setup by default when cloning from the local\nmirror as the config is not copied over.\n\n>\n> I gather that local clones are fast because hardlinked - is that correct?\n\nYes, a local path implies --local in git-clone, which (a) uses hardlinks\nand (b) avoids some other protocol overhead.\n\n> Is that correct on Windows? (I can't easily avoid Windows.)\n\nLet's see if a Windows expert shows up, I cannot tell.\n\n> Ramification 1:\n>\n> I'm not sure how best to prepare patches for push-to-upstream.\n> Is there value in collecting them locally into a push-to-upstream repo, or\n> is it better to just push from each local clone individually?\n\nIt depends on a lot of things:\n* How critical is the latency in the desired workflow?\n\n  Say you have this setup on a cruise ship and only push once when\n  you are in a harbor, then (a) you want to make sure you pushed everything\n  and (b) you care less about latency. Hence you would prefer to collect\n  everything in one repo so nothing gets lost.\n\n  Say you are in a fast paced environment, where you want instant feedback\n  on your patches as they are mostly exploratory designs. Then you want to\n  push directly from the local clone individually to minimize latency, I would\n  imagine.\n\n* Does a local clone have any value for having the work from\n  another local clone available? In that case you may want to\n  have all your changes accumulated into the mirror.\n\n> Ramification 2:\n>\n> Some of the repos I work with use submodules. Sometimes they use submodules\n> that I'm not aware of. Or a submodule was used historically, and git bisect\n> breaks/misbehaves because it can't get the submodule in offline mode.\n\nOh!\n\n> Is there a way to get these, without writing a script that recurses through\n> all versions of .gitmodules?\n\nNot, that I am aware of. You need to find all submodules.\n\nWhen a submodule gets deleted (git rm <submodule> && git commit),\nthen all entries for that submodule in the .gitmodules file are also removed.\nThat seems ok, but in an ideal world we may have a tombstone in there\n(e.g. the submodule.NAME.path still set) that would help for tasks like finding\nall submodules in the future.\n\n> I'm seeing the --recurse-submodules option for git fetch, so this might (or\n> might not) be the Right Thing.\n\nThat only works for currently initialized (active) submodules. The submodules\nof the past and those which you do not have, are not fetched.\n\nWithout the submodule ramifications, I would have advised to have\nhave the local mirror a 'bare' repo.\n\nHope that helps,\nStefan\n"},{"id":"324314","messageId":"42f0df64-5747-c2d2-832f-10d921fa2efa@durchholz.org","threadId":"46362","inReplyTo":"CAGZ79ka59qkekqGCSmVHb0=H--4HtjC7Ts9eXj_hb6f2cRQ4ww@mail.gmail.com","subject":"Re: Mirroring for offline use - best practices?","fromName":"Joachim Durchholz","fromEmail":"jo@durchholz.org","sentAt":"2017-07-12T22:14:01Z","receivedAt":"2017-07-12T22:14:10Z","isPatch":false,"sender":{"key":"jo@durchholz.org","avatar":null},"body":"Am 12.07.2017 um 19:40 schrieb Stefan Beller:\n\nThanks for the feedback - it's been very, very useful to me!\n\n > Yes, a local path implies --local in git-clone, which (a) uses hardlinks\n > and (b) avoids some other protocol overhead.\n\nI guess (a) is the most important one for repositories large enough to \nmake this kind of stuff matter.\nI had gathered so much, but I wasn't sure - my own repos aren't large \nenough to take any measurements.\n\n >> Ramification 1:\n >>\n >> I'm not sure how best to prepare patches for push-to-upstream.\n >> Is there value in collecting them locally into a push-to-upstream \nrepo, or\n >> is it better to just push from each local clone individually?\n >\n > It depends on a lot of things:\n > * How critical is the latency in the desired workflow?\n >\n >    Say you have this setup on a cruise ship and only push once when\n >    you are in a harbor, then (a) you want to make sure you pushed \neverything\n >    and (b) you care less about latency. Hence you would prefer to collect\n >    everything in one repo so nothing gets lost.\n\nYeah, that's the kind of scenario I'm having.\nLess cruise ship (I wish!) and more on-train work during commutes, but \nit's similar.\n\nBut I think the \"make sure nothing gets lost\" aspect is the most \nimportant one. It's so easy to forget to push some sideline branch, \nparticularly if you're in a workflow that uses many branches.\n\nSo... an \"outbox\" repository would be the Right Thing for me.\nNot sure how generally applicable this is - what do other people think?\n\n >    Say you are in a fast paced environment, where you want instant \nfeedback\n >    on your patches as they are mostly exploratory designs. Then you \nwant to\n >    push directly from the local clone individually to minimize \nlatency, I would\n >    imagine.\n\nThat's for online work I think.\n\nOf course, there's the situation where you're sometimes offline and \nsometimes online.\nI'm not sure how to best handle that - have a script that switches \nconfigurations? Just stick with that outbox workflow because switching \nworkflows would invite error?\n\nOne thing that's specific to me is that I tend to be active in multiple \nprojects, so I might get home and have queued up pushes in multiple repos.\nI'm not sure how to best make sure that I don't forget a push.\nOTOH maybe I'm overly optimistic about for how many repositories I might \nbe working on a day, and it's a non-issue.\n\n > * Does a local clone have any value for having the work from\n >    another local clone available? In that case you may want to\n >    have all your changes accumulated into the mirror.\n\nYeah, definitely.\nA repository I work on might be used as a submodule, and I might want to \ncheck the ramifications.\n\n > When a submodule gets deleted (git rm <submodule> && git commit),\n > then all entries for that submodule in the .gitmodules file are also \nremoved.\n > That seems ok, but in an ideal world we may have a tombstone in there\n > (e.g. the submodule.NAME.path still set) that would help for tasks \nlike finding\n > all submodules in the future.\n\nI wouldn't want to use tombstone entries actually, because the content \nof .gitconfig and .gitmodules might have been modified for any number of \nreasons.\n\nThe incantations that I'm using for my own \"gitmirror\" script are:\n1. Get all commits that touch .gitmodules via\n   git rev-list --all --full-history -- .gitmodules\n2. Get a list of all module names mentioned in that .gitmodules version via\n   git config \\\n   --blob \"${commit}:.gitmodules\" \\\n   --name-only \\\n   --get-regexp \"^submodule\\..*\\.path$\"\n3. Given the module name, extract path and url via\n   git config \\\n     --blob \"${commit}:.gitmodules\" \\\n     --get \"submodule.${module_name}.path\"\nresp.\n     --get \"submodule.${module_name}.url\" \\\n\nIt's not the most efficient way conceivable, but it's using git's \nconfiguration parser, and it won't get put off by manual edits in \nconfiguration files .\n\nIt's nothing you'd do on the command line, hence the scripting :-)\n\n >> I'm seeing the --recurse-submodules option for git fetch, so this \nmight (or\n >> might not) be the Right Thing.\n >\n > That only works for currently initialized (active) submodules. The \nsubmodules\n > of the past and those which you do not have, are not fetched.\n\nAww.\nAh well.\n\n > Without the submodule ramifications, I would have advised to have\n > have the local mirror a 'bare' repo.\nI'm currently steering towards having a cache for all repositories I \never downloaded, which would live in ~/.cache/gitmirror.\n\nI can turn this script into a public, maintained project if there's \ninterest.\nCurrent state is two files, the script itself and a unit test script. \nIt's still in a state of flux so sharing would be for the code reviews \nat this time, general usefulness would come later.\n\nRegards,\nJo\n"}]}