{"thread":{"id":"38932","subject":"how to make \"full\" copy of a repo","startedAt":"2015-03-28T02:56:37Z","lastAt":"2015-03-30T17:37:39Z","messageCount":9,"participants":["Christoph Anton Mitterer","Kevin D","Torsten Bögershausen","Ævar Arnfjörð Bjarmason","Duy Nguyen","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"258629","messageId":"1427511397.19633.52.camel@scientia.net","threadId":"38932","inReplyTo":null,"subject":"how to make \"full\" copy of a repo","fromName":"Christoph Anton Mitterer","fromEmail":"calestyo@scientia.net","sentAt":"2015-03-28T02:56:37Z","receivedAt":"2015-03-28T02:56:37Z","isPatch":false,"sender":{"key":"calestyo@scientia.net","avatar":"https://gravatar.com/avatar/74cd78048081efac1fcd21e681751ed55968f949a50e33adbe1a41c572a605cd?d=mp&s=160"},"body":"Hey.\n\nI was looking for an ideally simple way to make a \"full\" copy of a git\nrepo. Many howtos are floating around on this on the web, with also lots\nof voodoo.\n\n\nFirst, it shouldn't be just a clone, i.o.w.\n- I want to have all refs (local/remote branches/tags) and of course all\nobjects from the source repo copied as is.\nSo it's local branches should become my local branches and not remote\nbranches as well - and so on.\nBasically I want to be able to delete the source afterwards (and all\nbackups ;) ) and not having anything lost.\n\n- It shouldn't set the source repo as origin or it's branches as remote\ntracking branches, as said it should be identical the source repo, just\n\"freshly copied\" via the \"Git aware transport mechanisms\".\n\n- Whether GC or repacking happens, I don't care, as long as nothing that\nis still reachable in the source repo wouldn't get lost (or get lost\nonce I run a GC in the copied repo).\n\n- Whether anything that other tools have added to .git (e.g. git-svn\nstuff) get's lost, I don't care.\n\n- It should work for both, bare and non-bare repos, but it's okay when\nit doesn't copy anything that is not committed or stashed.\n\n\n\nI'd have said that either:\n$ git clone --mirror URl-to-source-repo copy\nfor the direction from \"outside\" the source to a copy,\nor alternatively:\n$ cd source-repo\n$ git push --mirror URl-to-copy\nfor the direction from \"within\" the source to a copy with copy being an\nempty bare or non-bare repo,\nwould do the job.\n\nBut:\n\na) but the git-clone(1) part for --mirror:\n   >and sets up a refspec configuration such that all these refs are\n   >overwritten by a git remote update in the target repository.\n   kinda confuses me since I wanted to get independent of the source\n   repo and this ssems to set up a remote to it?\n\nb) do I need --all --tags for the push as well?\n\nc) When following\n   https://help.github.com/articles/duplicating-a-repository/\n   it doesn't seem as if --mirror is what I want because they seem to\n   advertise it rather as having the copy tracking the source repo.\n   Of course I read about just using git-clone --bare, but that seems to\n   not copy everything that --mirror does (remote-tracking branches,\n   notes).\n\n   So I'm a bit confused...\n\n\n1) Is it working like I assumed above?\n2) Does that also copy things like git-config, hooks, etc.?\n3) Does it copy the configured remotes from the source?\n4) What else is not copied by that? I'd assume anything that is not\n   tracked by git and the stash of the source?\n\n\n\nThanks a lot,\nChris.\n"},{"id":"258635","messageId":"20150328143152.GB9387@vps892.directvps.nl","threadId":"38932","inReplyTo":"1427511397.19633.52.camel@scientia.net","subject":"Re: how to make \"full\" copy of a repo","fromName":"Kevin D","fromEmail":"me@ikke.info","sentAt":"2015-03-28T14:31:52Z","receivedAt":"2015-03-28T14:31:52Z","isPatch":false,"sender":{"key":"me@ikke.info","avatar":"https://avatars.githubusercontent.com/u/135698?v=4"},"body":"On Sat, Mar 28, 2015 at 03:56:37AM +0100, Christoph Anton Mitterer wrote:\n> Hey.\n> \n> I was looking for an ideally simple way to make a \"full\" copy of a git\n> repo. Many howtos are floating around on this on the web, with also lots\n> of voodoo.\n> \n> \n> First, it shouldn't be just a clone, i.o.w.\n> - I want to have all refs (local/remote branches/tags) and of course all\n> objects from the source repo copied as is.\n> So it's local branches should become my local branches and not remote\n> branches as well - and so on.\n> Basically I want to be able to delete the source afterwards (and all\n> backups ;) ) and not having anything lost.\n> \n> - It shouldn't set the source repo as origin or it's branches as remote\n> tracking branches, as said it should be identical the source repo, just\n> \"freshly copied\" via the \"Git aware transport mechanisms\".\n> \n> - Whether GC or repacking happens, I don't care, as long as nothing that\n> is still reachable in the source repo wouldn't get lost (or get lost\n> once I run a GC in the copied repo).\n> \n> - Whether anything that other tools have added to .git (e.g. git-svn\n> stuff) get's lost, I don't care.\n> \n> - It should work for both, bare and non-bare repos, but it's okay when\n> it doesn't copy anything that is not committed or stashed.\n> \n> \n> \n> I'd have said that either:\n> $ git clone --mirror URl-to-source-repo copy\n> for the direction from \"outside\" the source to a copy,\n> or alternatively:\n> $ cd source-repo\n> $ git push --mirror URl-to-copy\n> for the direction from \"within\" the source to a copy with copy being an\n> empty bare or non-bare repo,\n> would do the job.\n> \n> But:\n> \n> a) but the git-clone(1) part for --mirror:\n>    >and sets up a refspec configuration such that all these refs are\n>    >overwritten by a git remote update in the target repository.\n>    kinda confuses me since I wanted to get independent of the source\n>    repo and this ssems to set up a remote to it?\n> \n> b) do I need --all --tags for the push as well?\n> \n> c) When following\n>    https://help.github.com/articles/duplicating-a-repository/\n>    it doesn't seem as if --mirror is what I want because they seem to\n>    advertise it rather as having the copy tracking the source repo.\n>    Of course I read about just using git-clone --bare, but that seems to\n>    not copy everything that --mirror does (remote-tracking branches,\n>    notes).\n> \n>    So I'm a bit confused...\n> \n> \n> 1) Is it working like I assumed above?\n> 2) Does that also copy things like git-config, hooks, etc.?\n> 3) Does it copy the configured remotes from the source?\n> 4) What else is not copied by that? I'd assume anything that is not\n>    tracked by git and the stash of the source?\n> \n> \n> \n> Thanks a lot,\n> Chris.\n\nGit clone is never going to get you a copy where nothing is lost.\n\nWhat you are losing on clone is:\n\n* config settings (this includes the configures remotes)\n* hooks\n* reflog (history of refs, though, by default disabled for bare\n  repositories)\n* Stashes, because the reference to them is stored in the reflog\n* unreferenced objects (though you said those are not a requirement, it\n  is still something that is lost)\n\ngit clone --mirror is used for repositories that regularly get updates\nfrom the repositories they were cloned from. Though this is not what you\nwant, it's not difficult to reset the refspecs to the default refspecs.\nBecause it fetches all refs, it's not necessary to add --all --tags\n(because tags are also refs).\n\ngit clone --mirror is the closest you are going to get by only using\ngit.\n\nI guess you are aware of this, but if you want to retain more\ninformation, you have to rely on other means, like scp to get the other\nthings\n\nSo to summarize, git clone is only used for cloning history, which means\nobjects and refs, the rest is not part of cloning. To get more, you have\nto go outside git.\n\nHope this helps to clear some confussion.\n\nKevin\n"},{"id":"258640","messageId":"5516F856.9010100@web.de","threadId":"38932","inReplyTo":"1427511397.19633.52.camel@scientia.net","subject":"Re: how to make \"full\" copy of a repo","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2015-03-28T18:52:06Z","receivedAt":"2015-03-28T18:52:06Z","isPatch":false,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On 2015-03-28 03.56, Christoph Anton Mitterer wrote:\n> Hey.\n> \n> I was looking for an ideally simple way to make a \"full\" copy of a git\n> repo. Many howtos are floating around on this on the web, with also lots\n> of voodoo.\n> \n> \n> First, it shouldn't be just a clone, i.o.w.\n> - I want to have all refs (local/remote branches/tags) and of course all\n> objects from the source repo copied as is.\n> So it's local branches should become my local branches and not remote\n> branches as well - and so on.\n> Basically I want to be able to delete the source afterwards (and all\n> backups ;) ) and not having anything lost.\n> \n> - It shouldn't set the source repo as origin or it's branches as remote\n> tracking branches, as said it should be identical the source repo, just\n> \"freshly copied\" via the \"Git aware transport mechanisms\".\n> \n> - Whether GC or repacking happens, I don't care, as long as nothing that\n> is still reachable in the source repo wouldn't get lost (or get lost\n> once I run a GC in the copied repo).\n> \n> - Whether anything that other tools have added to .git (e.g. git-svn\n> stuff) get's lost, I don't care.\n> \n> - It should work for both, bare and non-bare repos, but it's okay when\n> it doesn't copy anything that is not committed or stashed.\n> \n> \n> \n> I'd have said that either:\n> $ git clone --mirror URl-to-source-repo copy\n> for the direction from \"outside\" the source to a copy,\n> or alternatively:\n> $ cd source-repo\n> $ git push --mirror URl-to-copy\n> for the direction from \"within\" the source to a copy with copy being an\n> empty bare or non-bare repo,\n> would do the job.\n> \n> But:\n> \n> a) but the git-clone(1) part for --mirror:\n>    >and sets up a refspec configuration such that all these refs are\n>    >overwritten by a git remote update in the target repository.\n>    kinda confuses me since I wanted to get independent of the source\n>    repo and this ssems to set up a remote to it?\n> \n> b) do I need --all --tags for the push as well?\n> \n> c) When following\n>    https://help.github.com/articles/duplicating-a-repository/\n>    it doesn't seem as if --mirror is what I want because they seem to\n>    advertise it rather as having the copy tracking the source repo.\n>    Of course I read about just using git-clone --bare, but that seems to\n>    not copy everything that --mirror does (remote-tracking branches,\n>    notes).\n> \n>    So I'm a bit confused...\nThis instructions have 3 repos:\nthe source, \"old\", the destination \"new\" and a temporary one.\nAs you only push to \"new\", \"new\" should have no information about\n\"old\" or \"temp\".\n> \n> \n> 1) Is it working like I assumed above?\n> 2) Does that also copy things like git-config, hooks, etc.?\n> 3) Does it copy the configured remotes from the source?\n> 4) What else is not copied by that? I'd assume anything that is not\n>    tracked by git and the stash of the source?\n\nYou didn't write if this is a bare repository,\nif it is on a local disc, if it is reachable by rsync ?\nLinux or Windows ?\n\nFor a \"full clone\" (in the sense of having everything, bit for bit)\nI would probably use rsync. (After stopping all activities on the repo)\n\nBut I don't know where you repos life, are they bare or not, so there\nmay be other ways to go.\n\n> \n> \n> Thanks a lot,\n> Chris.\n> \n"},{"id":"258643","messageId":"CACBZZX6y3OB3y+V6VetHD0ZOxG3T+xsCFXy_-Qj2erLK7-wpEg@mail.gmail.com","threadId":"38932","inReplyTo":"5516F856.9010100@web.de","subject":"Re: how to make \"full\" copy of a repo","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2015-03-28T20:33:31Z","receivedAt":"2015-03-28T20:33:31Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Sat, Mar 28, 2015 at 7:52 PM, Torsten Bögershausen <tboegi@web.de> wrote:\n> On 2015-03-28 03.56, Christoph Anton Mitterer wrote:\n>> Hey.\n>>\n>> I was looking for an ideally simple way to make a \"full\" copy of a git\n>> repo. Many howtos are floating around on this on the web, with also lots\n>> of voodoo.\n>>\n>>\n>> First, it shouldn't be just a clone, i.o.w.\n>> - I want to have all refs (local/remote branches/tags) and of course all\n>> objects from the source repo copied as is.\n>> So it's local branches should become my local branches and not remote\n>> branches as well - and so on.\n>> Basically I want to be able to delete the source afterwards (and all\n>> backups ;) ) and not having anything lost.\n>>\n>> - It shouldn't set the source repo as origin or it's branches as remote\n>> tracking branches, as said it should be identical the source repo, just\n>> \"freshly copied\" via the \"Git aware transport mechanisms\".\n>>\n>> - Whether GC or repacking happens, I don't care, as long as nothing that\n>> is still reachable in the source repo wouldn't get lost (or get lost\n>> once I run a GC in the copied repo).\n>>\n>> - Whether anything that other tools have added to .git (e.g. git-svn\n>> stuff) get's lost, I don't care.\n>>\n>> - It should work for both, bare and non-bare repos, but it's okay when\n>> it doesn't copy anything that is not committed or stashed.\n>>\n>>\n>>\n>> I'd have said that either:\n>> $ git clone --mirror URl-to-source-repo copy\n>> for the direction from \"outside\" the source to a copy,\n>> or alternatively:\n>> $ cd source-repo\n>> $ git push --mirror URl-to-copy\n>> for the direction from \"within\" the source to a copy with copy being an\n>> empty bare or non-bare repo,\n>> would do the job.\n>>\n>> But:\n>>\n>> a) but the git-clone(1) part for --mirror:\n>>    >and sets up a refspec configuration such that all these refs are\n>>    >overwritten by a git remote update in the target repository.\n>>    kinda confuses me since I wanted to get independent of the source\n>>    repo and this ssems to set up a remote to it?\n>>\n>> b) do I need --all --tags for the push as well?\n>>\n>> c) When following\n>>    https://help.github.com/articles/duplicating-a-repository/\n>>    it doesn't seem as if --mirror is what I want because they seem to\n>>    advertise it rather as having the copy tracking the source repo.\n>>    Of course I read about just using git-clone --bare, but that seems to\n>>    not copy everything that --mirror does (remote-tracking branches,\n>>    notes).\n>>\n>>    So I'm a bit confused...\n> This instructions have 3 repos:\n> the source, \"old\", the destination \"new\" and a temporary one.\n> As you only push to \"new\", \"new\" should have no information about\n> \"old\" or \"temp\".\n>>\n>>\n>> 1) Is it working like I assumed above?\n>> 2) Does that also copy things like git-config, hooks, etc.?\n>> 3) Does it copy the configured remotes from the source?\n>> 4) What else is not copied by that? I'd assume anything that is not\n>>    tracked by git and the stash of the source?\n>\n> You didn't write if this is a bare repository,\n> if it is on a local disc, if it is reachable by rsync ?\n> Linux or Windows ?\n>\n> For a \"full clone\" (in the sense of having everything, bit for bit)\n> I would probably use rsync. (After stopping all activities on the repo)\n\nThis warrants more emphasis. If you rsync a repository that's\n\"active\", i.e. getting pushes you *will* get corrupt copies. E.g. you\ncan easily copy something out of the objects directory that's in the\nmiddle of being written, or copy the \"refs\" namespace after you copy\n\"objects\" and end up with an unreachable object.\n\nThere's unfortunately no good solution to this other than doing both\ngit --mirror backups and rsync backups (for hooks etc.) and combining\nthe two, or pushing a hook for the duration that bans all updates.\n"},{"id":"258646","messageId":"1427595686.4912.17.camel@scientia.net","threadId":"38932","inReplyTo":"20150328143152.GB9387@vps892.directvps.nl","subject":"Re: how to make \"full\" copy of a repo","fromName":"Christoph Anton Mitterer","fromEmail":"calestyo@scientia.net","sentAt":"2015-03-29T02:21:26Z","receivedAt":"2015-03-29T02:21:26Z","isPatch":false,"sender":{"key":"calestyo@scientia.net","avatar":"https://gravatar.com/avatar/74cd78048081efac1fcd21e681751ed55968f949a50e33adbe1a41c572a605cd?d=mp&s=160"},"body":"On Sat, 2015-03-28 at 15:31 +0100, Kevin D wrote: \n> What you are losing on clone is:\n> * config settings (this includes the configures remotes)\n> * hooks\nthat would be okay...\n\n\n> * reflog (history of refs, though, by default disabled for bare\n>   repositories)\nis there a way to get this copied?\n\n\n> * Stashes, because the reference to them is stored in the reflog\n> * unreferenced objects (though you said those are not a requirement, it\n>   is still something that is lost)\nthat would be okay for me either.\n\n\n> git clone --mirror is used for repositories that regularly get updates\n> from the repositories they were cloned from. Though this is not what you\n> want, it's not difficult to reset the refspecs to the default refspecs.\nWhat do you mean here? What would I need to reset exactly?\n\n\n> git clone --mirror is the closest you are going to get by only using\n> git.\nI see, thanks :)\n\n> So to summarize, git clone is only used for cloning history, which means\n> objects and refs, the rest is not part of cloning. To get more, you have\n> to go outside git.\n\nThanks :)\nChris.\n"},{"id":"258647","messageId":"1427595767.4912.19.camel@scientia.net","threadId":"38932","inReplyTo":"5516F856.9010100@web.de","subject":"Re: how to make \"full\" copy of a repo","fromName":"Christoph Anton Mitterer","fromEmail":"calestyo@scientia.net","sentAt":"2015-03-29T02:22:47Z","receivedAt":"2015-03-29T02:22:47Z","isPatch":false,"sender":{"key":"calestyo@scientia.net","avatar":"https://gravatar.com/avatar/74cd78048081efac1fcd21e681751ed55968f949a50e33adbe1a41c572a605cd?d=mp&s=160"},"body":"On Sat, 2015-03-28 at 19:52 +0100, Torsten Bögershausen wrote: \n> As you only push to \"new\", \"new\" should have no information about\n> \"old\" or \"temp\".\nExactly, that would be the goal.\n\n \n> > 1) Is it working like I assumed above?\n> > 2) Does that also copy things like git-config, hooks, etc.?\n> > 3) Does it copy the configured remotes from the source?\n> > 4) What else is not copied by that? I'd assume anything that is not\n> >    tracked by git and the stash of the source?\n> You didn't write if this is a bare repository,\n> if it is on a local disc, if it is reachable by rsync ?\n> Linux or Windows ?\nLinux.\nAnd in principle I have both cases, but mostly non-bare repos.\n\n\nCheers,\nChris.\n"},{"id":"258652","messageId":"20150329110506.GC9387@vps892.directvps.nl","threadId":"38932","inReplyTo":"1427595686.4912.17.camel@scientia.net","subject":"Re: how to make \"full\" copy of a repo","fromName":"Kevin D","fromEmail":"me@ikke.info","sentAt":"2015-03-29T11:05:06Z","receivedAt":"2015-03-29T11:05:06Z","isPatch":false,"sender":{"key":"me@ikke.info","avatar":"https://avatars.githubusercontent.com/u/135698?v=4"},"body":"On Sun, Mar 29, 2015 at 04:21:26AM +0200, Christoph Anton Mitterer wrote:\n> On Sat, 2015-03-28 at 15:31 +0100, Kevin D wrote: \n> [..]\n> \n> > * reflog (history of refs, though, by default disabled for bare\n> >   repositories)\n> is there a way to get this copied?\n> \n> \n\nNo, the reflog is considered something private to the repository, so\nthere is no way to git it through git clone.\n\n> [..] \n> \n> > git clone --mirror is used for repositories that regularly get updates\n> > from the repositories they were cloned from. Though this is not what you\n> > want, it's not difficult to reset the refspecs to the default refspecs.\n> What do you mean here? What would I need to reset exactly?\n\ngit clone --mirror sets up the fetch refspec in such a way that local\nrefs would get reset to whatever upstream has:\n\n+refs/*:refs/*\n\nSo every time you would fetch / pull, all your branches would reflect\nthe way they are on the mirrored repo (which is why it's called mirror).\n\nThe default refspec is:\n\n+refs/heads/*:refs/remotes/origin/*\n\nWhich would only fetch heads (branches), and maps them as remote\ntracking branches, so that your local branches are left alone.\n\n> > git clone --mirror is the closest you are going to get by only using\n> > git.\n> I see, thanks :)\n> \n> > So to summarize, git clone is only used for cloning history, which means\n> > objects and refs, the rest is not part of cloning. To get more, you have\n> > to go outside git.\n> \n> Thanks :)\n> Chris.\n"},{"id":"258673","messageId":"CACsJy8AN6X3oh2=yLn-szEApEMC+uQMmO6qBYbPrVNK1_d5tEQ@mail.gmail.com","threadId":"38932","inReplyTo":"20150328143152.GB9387@vps892.directvps.nl","subject":"Re: how to make \"full\" copy of a repo","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2015-03-30T15:22:07Z","receivedAt":"2015-03-30T15:22:07Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sat, Mar 28, 2015 at 9:31 PM, Kevin D <me@ikke.info> wrote:\n> Git clone is never going to get you a copy where nothing is lost.\n>\n> What you are losing on clone is:\n>\n> * config settings (this includes the configures remotes)\n> * hooks\n> * reflog (history of refs, though, by default disabled for bare\n>   repositories)\n> * Stashes, because the reference to them is stored in the reflog\n> * unreferenced objects (though you said those are not a requirement, it\n>   is still something that is lost)\n\nThis is true. But I wonder if we should (and can) support\n--super-mirror option (disabled by default), where reflog and stashes\nare kept, for backup purposes. We might keep unreferenced objects as\nwell if it's not hard to do.\n-- \nDuy\n"},{"id":"258683","messageId":"xmqqbnjaxu70.fsf@gitster.dls.corp.google.com","threadId":"38932","inReplyTo":"CACsJy8AN6X3oh2=yLn-szEApEMC+uQMmO6qBYbPrVNK1_d5tEQ@mail.gmail.com","subject":"Re: how to make \"full\" copy of a repo","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2015-03-30T17:37:39Z","receivedAt":"2015-03-30T17:37:39Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen <pclouds@gmail.com> writes:\n\n> This is true. But I wonder if we should (and can) support\n> --super-mirror option (disabled by default), where reflog and stashes\n> are kept, for backup purposes. We might keep unreferenced objects as\n> well if it's not hard to do.\n\nI doubt that we want to be in the business of filesystem backup.  Is\nthere a practical use case that is *not* \"I am relocating out of\nthis directory on this machine and will be using the other one I am\nmaking by copying\"?\n\nFor the \"I am relocating\" scenario, rsync is the most suitable\noption.  The caveat \"activity at the original will leave the copied\nresult incomplete\" will apply to whatever transport method you use,\nbut in the \"I am relocating\" scenario, you will have a period that\nthe original is quiet (i.e. you stop using the original at some\npoint before you start the copied one, and do not expect that the\nsequence to take zero down time).\n\nIn a sense, \"super-mirror\" is even worse, if it is doing some \"Git\nactivity\" on the source which we may want to log, which means the\noriginal will never be quiet during the copying.  Sure, send-pack\nmay not currently not do any logging in the original repository, but\ndepending on the reason why such a copy is being made, the original\nmay even have a custom hook-based logging data left somewhere in the\nrepository and for copying such a repository the repository owner\nwould want to keep the logged data.\n\nAnd if what super-mirror does is not considered a \"Git activity\" and\nsomehow bypasses all the Git rules in the original repository, then\nwhat is the advantage of having it in Git in the first place, over\nusing something like rsync?\n"}]}