{"thread":{"id":"24186","subject":"[GSoC update extra!] git-remote-svn: Week 8","startedAt":"2010-06-24T13:33:28Z","lastAt":"2010-07-01T03:38:54Z","messageCount":8,"participants":["Ramkumar Ramachandra","Jonathan Nieder","Eric Wong","Sam Vilain"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"144152","messageId":"1277386408-29943-1-git-send-email-artagnon@gmail.com","threadId":"24186","inReplyTo":null,"subject":"[GSoC update extra!] git-remote-svn: Week 8","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2010-06-24T13:33:28Z","receivedAt":"2010-06-24T13:33:28Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi,\n\nI built a client that generates a deltified dumpfile, but David's\nexporter can't build full text from it. We got an idea, brought Sverre\ninto the loop and discussed it. David and I felt that the chatlog was\nvaluable enough to go on the Git mailing list for later reference, so\nhere it is.\n\nThis proposal will take a lot of time to implement. The current plan\nis to get my client to dump full text (rather inefficiently), and get\nsomething working while Sverre (and possibly others) work on this\nproposal in the meantime.\n\nSigned-off-by: Ramkumar Ramachandra \n---\n notes | 117 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 files changed, 117 insertions(+), 0 deletions(-)\n create mode 100644 notes\n\ndiff --git a/notes b/notes\nnew file mode 100644\nindex 0000000..224a5be\n--- /dev/null\n+++ b/notes\n@@ -0,0 +1,117 @@\n+14:19 *** SRabbelier JOIN\n+14:19 artagnon: backstory please :)\n+14:20 SRabbelier: welcome\n+14:20 sup :)\n+14:20 SRabbelier: Background -- I generated a delta dumpfile fine. barrbrain's exporter can't build full text from that, and I can't do it in my client either. We are looking for some kind of filesystem -- the Git\n+ store is the answer. Can we import revisions one by one through fast-import and ask Git to generate the full text?\n+14:22 SRabbelier: Note that barrbrain's recent work on persistence makes it support incremental dumps, so throwing it into fast-import is not the problem at all.\n+14:22 artagnon: well, that kind of goes against fast-import's nature, since if you do one revision at a time you lose the benefits of using it (low memory profile, etc)\n+14:22 Hm.\n+14:23 Is there some option to tweak it to create just loose objects and not pack them for every revision?\n+14:23 And then pack them all together after we produce a full text in the end?\n+14:23 artagnon: it doesn't create a packfile until you call 'checkpoint'\n+14:24 Excellent. Then what's the problem? I won't call checkpoint until we get the full text out.\n+14:25 perhaps I'm not quite understanding the problem then\n+14:25 you can definitely call 'git cat-file' after each exported file\n+14:25 and fast-import would just hang waiting for more input I guess\n+14:26 barrbrain: Does that ^^ sound good?\n+14:26 artagnon: sounds good\n+14:27 SRabbelier: Thanks, that's what we wanted to know :)\n+14:27 SRabbelier: how does the exporter know when the checkpoint has completed?\n+14:27 barrbrain: it doesn't matter\n+14:27 barrbrain: the object is available regardless of whether you checkpoint\n+14:28 oh really?\n+14:28 barrbrain: it doesn't matter if the object is loose or in a pack\n+14:28 barrbrain: it's transparent to whoever is calling git\n+14:28 loose object and pack are equivalent to Git infrastructure\n+14:28 right, only packs are faster\n+14:28 even in the middle of a fast-import?\n+14:28 [artagnon nods]\n+14:29 the catch is that the only reference we have for a blob is a mark\n+14:29 You have to keep all the marks until the fulltext is built?\n+14:30 barrbrain: ah, well, that is a problem\n+14:30 also, remind me why you need the fulltext?\n+14:30 the only way out of that is to hash them as we go\n+14:31 input stream is in deltas\n+14:32 barrbrain: ... also, what about text-content-length? You need that to parse the dumpfile, right?\n+14:32 yeah\n+14:32 Erm.\n+14:32 It'll be horrible if replay doesn't supply that.\n+14:32 indeed\n+14:33 I told SRabbelier about that too... loading the whole thing into memory is a terrible option.\n+14:34 artagnon: well, git cat-file-ing it will require it to be loaded too?\n+14:34 Oh, damn.\n+14:35 SRabbelier: Is there a way to keep dynamically write it from a stream? I don't want git-cat-file then.\n+14:35 artagnon: oh like that\n+14:35 artagnon: uhm, I assume it streams it\n+14:35 artagnon: but you'd have to double check that\n+14:35 [artagnon checks]\n+14:40 so, if you have pipes to fast-import and cat-file --batch, can you read-write from the repo safely?\n+14:40 barrbrain: yes, git is concurrency safe\n+14:41 what about the timing issue - if I write to fast import the blob might be delayed in a buffer somewhere\n+14:41 barrbrain: I think what we'll need to do is to extend fast-import to also write the object names to stdout\n+14:42 barrbrain: as soon as it's done writing the object\n+14:42 nvm, that's what the checkpoint/progress comment is all about\n+14:42 barrbrain: and then you can wait till you get the name on stdout, and then you'll be sure that it's safe to git cat-file that object\n+14:42 barrbrain: no\n+14:42 barrbrain: checkpoint is only about packing\n+14:42 except no need for all that work\n+14:42 barrbrain: you don't want to do checkpoint after each commit though\n+14:43 SRabbelier: I totally agree\n+14:43 barrbrain: but, you can read the marks from the export file after a checkpoint, that is true\n+14:43 so what I guess you could do\n+14:43 is to keep feeding it commits till you need to get the hashes\n+14:44 then do a checkpoint and a progress\n+14:44 watch it's stdout till you see the progress message\n+14:44 it'd be nice to have a command to write the hash of a mark to stdout\n+14:44 (so that you know that the checkpoint has completed)\n+14:44 and then you can read the markfile\n+14:44 barrbrain: if it would make things easier I could probably fairly easily hack fast-import to do that always\n+14:45 barrbrain: so the other workflow I described, where it prints the hash of each mark ti stdiyt as it processes it\n+14:45 that's potentially a lot of detail in the output though :)\n+14:45 barrbrain: yes, so it would be guarded by a flag\n+14:45 SRabbelier: oh, nice\n+14:45 barrbrain: --print-marks\n+14:45 or of cvourse\n+14:45 barrbrain: in the stream you could say\n+14:46 barrbrain: option git print-marks\n+14:46 Better.\n+14:46 this is why we talk :)\n+14:46 IRC is superb for some discussions :)\n+14:46 ok, so you'd have to capture fast-import's stdout\n+14:47 and you'd have to wait till you got the mark line\n+14:47 and then you could feed that to the `git cat-file --batch` pipe\n+14:47 SRabbelier: I hope it doesn't break the workflow of the remote helper.\n+14:47 artagnon: ah, hmmm\n+14:48 that's a problem\n+14:48 I thought so.\n+14:48 :|\n+14:48 since in that case git starts the fast-import stream\n+14:48 Exactly.\n+14:48 **the fast-improt process\n+14:49 well, I guess we could add a capability to the remote-helper\n+14:49 that would make git just forward those marks to the remote-helper process?\n+14:49 (on it's stdin)\n+14:50 artagnon, barrbrain: that would work, no?\n+14:50 [artagnon is thinking]\n+14:51 I think I've lost track of which pipe goes where\n+14:51 artagnon: the remote-helper could just read from it's stdin, instead of from fast-import's stdout\n+14:51 Right, got it.\n+14:51 Yeah, a new capability should work.\n+14:51 barrbrain: git-remote-svn would be hooked up both stdin and stdout to git\n+14:51 barrbrain: when it starts, it tells git through a capability that it needs to have the marks hashes\n+14:52 barrbrain: when git tells it to start exporting, git-remote-svn will write it's stream to stdout\n+14:52 barrbrain: after each mark, git will read from the stdout of the fast-import process that it starts, and write that mark back to git-remote-svn's stdin\n+14:53 barrbrain: so git-remote-svn can then read from it's stdin, and write that hash back to git cat-file --batch\n+14:53 right, can it then request blobs to be received on stdin later?\n+14:53 barrbrain: yeah, that would work to, but I think it'd be better to keep it generic\n+14:53 barrbrain: and instead have git write the hashes to git-remote-svn's stdin\n+14:53 barrbrain: since then you can do whatever you want with those hashes\n+14:54 barrbrain: for example, you might want to create a tag for those hashes, or such, in which case a hash is useful, and the contents of the blob isn't\n+14:54 or it just gets hashes and opens its own cat-file process?\n+14:54 barrbrain: right\n+14:54 barrbrain: that\n+14:55 artagnon, barrbrain: anyway, if this is what we want, please mail the git list (especially Shawn), and propose this\n+14:55 see what he thinks of it\n+14:55 ask Shawn if he's ok with having git-fast-import learn a new '--print-marks' flag\n+14:55 if so, I'll get on that :)\n-- \n1.7.1\n"},{"id":"144165","messageId":"20100624173956.GA1600@burratino","threadId":"24186","inReplyTo":"1277386408-29943-1-git-send-email-artagnon@gmail.com","subject":"Re: [GSoC update extra!] git-remote-svn: Week 8","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-06-24T17:39:56Z","receivedAt":"2010-06-24T17:39:56Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ramkumar Ramachandra wrote:\n\n> David and I felt that the chatlog was\n> valuable enough to go on the Git mailing list for later reference, so\n> here it is.\n\nThanks. :)\n\n> SRabbelier: Background -- I generated a delta dumpfile\n> fine. barrbrain's exporter can't build full text from that, and I\n> can't do it in my client either. We are looking for some kind of\n> filesystem -- the Git store is the answer. Can we import revisions\n> one by one through fast-import and ask Git to generate the full\n> text?\n\nAh, this is something I was worried about with respect to persistence.\nGit has all the blobs and all the trees, so except for the mapping\nbetween marks, subversion revs, and git revs, svn-fe does not need to\npersist much data at all.\n\nOf course, that requires that the fast-import stream is going directly\nto git. fast-import streams can be used by other VCSes, too, but that\nproblem can be addressed later, I think.\n\n> so, if you have pipes to fast-import and cat-file\n> --batch, can you read-write from the repo safely?\n> barrbrain: yes, git is concurrency safe\n> what about the timing issue - if I write to fast import\n> the blob might be delayed in a buffer somewhere\n> barrbrain: I think what we'll need to do is to extend\n> fast-import to also write the object names to stdout\n> barrbrain: as soon as it's done writing the object\n\nFWIW, I like the idea.\n\nIf you want to keep stdout unpolluted, this could work like\n\n\tgit fast-import --print-marks=\n\nWe would have to make sure output to the fd is always flushed to\nprevent deadlock.\n\n> I think I've lost track of which pipe goes where\n\nYeah.\n\n> ask Shawn if he's ok with having git-fast-import learn a new '--print-marks' flag\n> if so, I'll get on that :)\n\nThanks for the insights.\n\nRegards,\nJonathan\n"},{"id":"144173","messageId":"20100624180752.GA1642@burratino","threadId":"24186","inReplyTo":"20100624173956.GA1600@burratino","subject":"Re: [GSoC update extra!] git-remote-svn: Week 8","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-06-24T18:07:52Z","receivedAt":"2010-06-24T18:07:52Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jonathan Nieder wrote:\n\n> Git has all the blobs and all the trees, so except for the mapping\n> between marks, subversion revs, and git revs, svn-fe does not need to\n> persist much data at all.\n> \n> Of course, that requires that the fast-import stream is going directly\n> to git.\n\nOne more thought.\n\nIf we are tracking the history of separate subversion branches separately,\nthen reading back trees includes an oddity:\n\nSuppose someone tries to reimplement git-svn on top of svn-fe[1].\n\n $ git svn --fe clone --stdlayout http://path/to/some/svn/root\n\nBehind the scenes, git-svn processes the fast-import stream it is\nreceiving and writes its _own_ fast-import stream with paths munged\nand commits split up into separate commits on each branch. Good.\n\nNow the oddity: suppose that in the repository, svn-fe finds an\n\n svn copy branches@r11 branches-old\n\noperation. In other words, it needs the tree for\nhttp://path/to/some/svn/root/branches@r11. This does not correspond\nto a single git tree, since the content of each branch has been given\nits own commit.\n\nHowever, this does not seem to be fatal: one could just make\n‘git svn --fe’ build a branch with the full history at the same time\nas it builds the other branches. Ugly, but I don’t see another way\naround it without making svn-fe and ‘git svn --fe’ know more about\neach other than I would like.\n\nJonathan\n\n[1] Eric, we are discussing the remote-svn series[2] and especially\nRam, Sverre, and David’s recent comments[3]. Apologies for not\nkeeping you in the loop sooner; your insights have always been\nhelpful in the past.\n\nAs for the idea of reimplementing git-svn on top of svn-fe: yes, the\nfast-import stream would need more information to support\n--follow-parent, but that piece is not so hard to add AFAICT. Of\ncourse, I am mentioning this not because it is important to keep the\ngit-svn interface but because the --stdlayout feature is very useful\nand we may want to port it over some day.\n\n[2] http://thread.gmane.org/gmane.comp.version-control.git/149571\n[3] http://thread.gmane.org/gmane.comp.version-control.git/149594\n"},{"id":"144212","messageId":"20100624213204.GA22756@dcvr.yhbt.net","threadId":"24186","inReplyTo":"20100624180752.GA1642@burratino","subject":"Re: [GSoC update extra!] git-remote-svn: Week 8","fromName":"Eric Wong","fromEmail":"normalperson@yhbt.net","sentAt":"2010-06-24T21:32:04Z","receivedAt":"2010-06-24T21:32:04Z","isPatch":false,"sender":{"key":"e@80x24.org","avatar":null},"body":"Jonathan Nieder wrote:\n> [1] Eric, we are discussing the remote-svn series[2] and especially\n> Ram, Sverre, and David’s recent comments[3]. Apologies for not\n> keeping you in the loop sooner; your insights have always been\n> helpful in the past.\n\nNo worries Jonathan, I've been preoccupied with several other projects,\nnone of which require dealing with SVN :)\n\n> As for the idea of reimplementing git-svn on top of svn-fe: yes, the\n> fast-import stream would need more information to support\n> --follow-parent, but that piece is not so hard to add AFAICT. Of\n> course, I am mentioning this not because it is important to keep the\n> git-svn interface but because the --stdlayout feature is very useful\n> and we may want to port it over some day.\n\nYes, I've always found --stdlayout very convenient and hope it\nlives on.\n\n-- \nEric Wong\n"},{"id":"144489","messageId":"1277862665.23613.8.camel@wilber","threadId":"24186","inReplyTo":"20100624180752.GA1642@burratino","subject":"Re: [GSoC update extra!] git-remote-svn: Week 8","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2010-06-30T01:51:05Z","receivedAt":"2010-06-30T01:51:05Z","isPatch":false,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"On Thu, 2010-06-24 at 13:07 -0500, Jonathan Nieder wrote:\n> operation. In other words, it needs the tree for\n> http://path/to/some/svn/root/branches@r11. This does not correspond\n> to a single git tree, since the content of each branch has been given\n> its own commit.\n\nI wrote at length about this near the beginning of the project;\nessentially, figuring out whether particular paths are roots or not is\nnot defined, as SVN does not distinguish between them (a misfeature\ncargo culted from Perforce). It becomes a data mining problem, you have\nthis scattered data, and you have to find a history inside.\n\nAs I recommended before, it probably makes more sense to keep a \"remote\ntracking\" branch which mirrors the *entire* repository, and sort out\nefficient ways to convert SVN revision paths like the above into tree\nIDs.\n\nI consider it very important to separate the data import and tracking\nstage from the data mining stage.\n\nOnce the data mining stage is well solved, then it makes sense to look\nat ways that a tracking branch which only tracks a part of the\nSubversion repository can be achieved. In the simple case, where no\nrepository re-organisation or cross-project renames have occurred it is\nrelatively simple. But in general I think this is a harder problem,\nwhich cannot always be solved without intervention - and so not\nnecessary to be solved in short-term milestones. As you are\ndiscovering, it is a can of worms which you avoid if you know you always\nhave the complete SVN repository available.\n\nSam\n"},{"id":"144491","messageId":"1277864434.23613.15.camel@wilber","threadId":"24186","inReplyTo":"20100624173956.GA1600@burratino","subject":"Re: [GSoC update extra!] git-remote-svn: Week 8","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2010-06-30T02:20:34Z","receivedAt":"2010-06-30T02:20:34Z","isPatch":false,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"On Thu, 2010-06-24 at 12:39 -0500, Jonathan Nieder wrote:\n> FWIW, I like the idea.\n> \n> If you want to keep stdout unpolluted, this could work like\n> \n> \tgit fast-import --print-marks=\n> \n> We would have to make sure output to the fd is always flushed to\n> prevent deadlock.\n\nThat works, it could default to --print-marks=0 if STDIN happens to be a\nsocket. Works for FastCGI :-)\n\nSamn\n"},{"id":"144513","messageId":"20100630124553.GA30999@debian","threadId":"24186","inReplyTo":"1277862665.23613.8.camel@wilber","subject":"Re: [GSoC update extra!] git-remote-svn: Week 8","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2010-06-30T12:45:53Z","receivedAt":"2010-06-30T12:45:53Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi Sam,\n\nSam Vilain writes:\n> On Thu, 2010-06-24 at 13:07 -0500, Jonathan Nieder wrote:\n> > operation. In other words, it needs the tree for\n> > http://path/to/some/svn/root/branches@r11. This does not correspond\n> > to a single git tree, since the content of each branch has been given\n> > its own commit.\n> \n> I wrote at length about this near the beginning of the project;\n> essentially, figuring out whether particular paths are roots or not is\n> not defined, as SVN does not distinguish between them (a misfeature\n> cargo culted from Perforce). It becomes a data mining problem, you have\n> this scattered data, and you have to find a history inside.\n\nRight. Implementing git-svn on top of git-remote-svn might not be a\nbad idea.\n\n> As I recommended before, it probably makes more sense to keep a \"remote\n> tracking\" branch which mirrors the *entire* repository, and sort out\n> efficient ways to convert SVN revision paths like the above into tree\n> IDs.\n> \n> I consider it very important to separate the data import and tracking\n> stage from the data mining stage.\n\nWe're following this approach. At the moment, we're just focusing on\ngetting all the data directly from SVN into the Git store. Instead of\nbuilding trees for each SVN revision, we've found a way to do it\ninside the Git object store: we're currently ironing out the details,\nand I'll post an update about this shortly.\n\n> Once the data mining stage is well solved, then it makes sense to look\n> at ways that a tracking branch which only tracks a part of the\n> Subversion repository can be achieved. In the simple case, where no\n> repository re-organisation or cross-project renames have occurred it is\n> relatively simple. But in general I think this is a harder problem,\n> which cannot always be solved without intervention - and so not\n> necessary to be solved in short-term milestones. As you are\n> discovering, it is a can of worms which you avoid if you know you always\n> have the complete SVN repository available.\n\nRight. I'm not convinced that it necessarily requires user\nintervention though: can you systematically prove that enough\ninformation is not available without user intervention using an\nexample? Or is it possible, but simply too difficult (and not worth\nthe effort) to mine out the data?\n\n-- Ram\n"},{"id":"144562","messageId":"1277955534.6605.81.camel@wilber","threadId":"24186","inReplyTo":"20100630124553.GA30999@debian","subject":"Re: [GSoC update extra!] git-remote-svn: Week 8","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2010-07-01T03:38:54Z","receivedAt":"2010-07-01T03:38:54Z","isPatch":false,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"On Wed, 2010-06-30 at 14:45 +0200, Ramkumar Ramachandra wrote: \n> > I wrote at length about this near the beginning of the project;\n> > essentially, figuring out whether particular paths are roots or not is\n> > not defined, as SVN does not distinguish between them (a misfeature\n> > cargo culted from Perforce). It becomes a data mining problem, you have\n> > this scattered data, and you have to find a history inside.\n> \n> Right. Implementing git-svn on top of git-remote-svn might not be a\n> bad idea.\n\nThat's a good way to look at it, yes. Probably git-svn has more\nsvn-specific code than import rules, so just the interface like\n--stdlayout etc is worth keeping, as well as checking that the svn\nimport data miner could do all the same things as git-svn.\n\n> > I consider it very important to separate the data import and tracking\n> > stage from the data mining stage.\n> \n> We're following this approach. At the moment, we're just focusing on\n> getting all the data directly from SVN into the Git store. Instead of\n> building trees for each SVN revision, we've found a way to do it\n> inside the Git object store: we're currently ironing out the details,\n> and I'll post an update about this shortly.\n\nOf course no working copy need exist with these contents within; that\nwould hardly be 'cheap copy' would it? :) But it's probably worth\nsticking to the standard tree/blob/commit object convention for ease of\ndebugging etc.\n\n> > Once the data mining stage is well solved, then it makes sense to look\n> > at ways that a tracking branch which only tracks a part of the\n> > Subversion repository can be achieved. In the simple case, where no\n> > repository re-organisation or cross-project renames have occurred it is\n> > relatively simple. But in general I think this is a harder problem,\n> > which cannot always be solved without intervention - and so not\n> > necessary to be solved in short-term milestones. As you are\n> > discovering, it is a can of worms which you avoid if you know you always\n> > have the complete SVN repository available.\n> \n> Right. I'm not convinced that it necessarily requires user\n> intervention though: can you systematically prove that enough\n> information is not available without user intervention using an\n> example? Or is it possible, but simply too difficult (and not worth\n> the effort) to mine out the data?\n\nSure, well all you really need to do is try it with a few real-world\nrepositories.\n\nBut I can give you a few examples of where all attempts at heuristics\nwill fail.\n\nThe first is where someone puts a file somewhere in the repository,\nperhaps a README.txt or something, somewhere outside the regular\nlocation.\n\n r1:\n add /README.txt\n\nThen, someone comes along and starts making their project:\n\n r2:\n add /trunk/README.txt\n\nHow do you know that the first commit is not part of any project, but\nsome out-of-band notes to people working with the repository?\n\nThe way I approached all this in my perforce converter (remember,\nPerforce is like SVN in almost every way) is to progressively scan the\nhistory and build up two tables which trace the \"mined\" history.\n\nYou can see the table definitions at\nhttp://utsl.gen.nz/gitweb/?p=git-p4raw;a=blob;f=tables.sql;h=259c243;hb=7e4fc4a#l205\n\nThe first, change_branches - records that a logical branch exists at a\nrevision and path.\n\n (branchpath, change)\n\n(you might want another 'column' in your conceptual data model: the\nproject name; I was dealing with a single project).\n\nThere are also cases where someone does something dumb, and then it is\nrepaired on the next commit.\n\neg\n\n /trunk/ProjectA\n /trunk/ProjectB\n /branches/ProjectA/foo\n /branches/ProjectB/bar\n\nSomeone comes along and does something like:\n\n rm /trunk\n mv /branches/ProjectA/foo /trunk\n\nWhoops! The /trunk path just got wiped. How do we fix it? In a hurry,\nthe system administrator checks out the old revision, tars them all up,\nthen uses 'svn add' to put them back.\n\n rm -r /trunk/*\n add /trunk/ProjectA\n add /trunk/ProjectB\n\nAfter this, people working on it realise the mistake: the disconnected\nhistory won't merge, etc. But the change is permanent, and they work\naround this error in the history. They don't want to do the more\ncorrect thing, which is restore the history from the broken commit,\n\n rm -r /trunk/*\n cp /trunk@42 /trunk\n\nThey don't want to do this because SVN has taught them that version\ncontrol is a fragile thing, and you don't want to monkey with it.\nBecause it can break and then your whole world is changed as the\nprecious black box which all your work is going into doesn't work quite\nas before. Because there is no \"undo\". Because it has all these opaque\nfiles inside it no-one can understand. What happened before with the\nrename upset and embarrassed you, and you don't want to risk making it\nworse.\n\nThis sort of thing does actually happen. The lesson is that you can't\ntrust heuristics, or the revision control breadcrumbs (copied-from etc)\nto be perfect. They are invisible - impossible to inspect directly\nusing the SVN command-line API, and impossible to revise once they are\nthere. by contrast, with git we have grafts, refs/replace/OBJID,\nfilter-branch, rebase, etc. We have visualization tools, git-add -p,\ngit gui. We have an object store which is robust, simple and widely\nunderstood. We have a simple data model, so that the actual information\ncan be understood by people and not just buggy software.\n\nOf course with SVN you have the fact that for the entire of its life as\na relevant version control system, it did not support merge tracking.\nSo, most history being imported will not have any reliable merge\ninformation. If you read early versions of the SVN manual, they\nactually advocate recording, in natural language, a human-readable\ndescription of the work done, in the commit message. I've seen people\nworking around this lack of functionality by developing their own\nsystems, sometimes not even being able to reconstruct what what merged\nwhere (eg, in parrot SVN).\n\nYet another situation is partial merging; unlike SVN, Perforce had\ndetailed merge tracking from the very beginning. With Perforce it\nworked on a per-file level only, so it is slightly different in that\nrespect. But what you find is that sometimes, people will merge only a\npart of another tree in to their \"trunk\" at a time.\n\n r45: merged\n /branches/ProjectA/src -> /trunk/ProjectA/dest\n\n r46: merged\n /branches/ProjectB/doc -> /trunk/ProjectB/doc\n\nWhat I normally find in this case is that there is no useful history\nrecorded in those intermediate commits; they were just committing to\nsave their intermediate work from being lost. This doesn't happen quite\nso much in Perforce, because it has a concept of \"index\" missing\nentirely from (the user API of) Subversion. In that case, it makes more\nsense to omit the intermediate commits, and simply record a single merge\nand leave out the intermediate commits.\n\nTo work around parenting mistakes - both those caused by misuse and from\na lack of SVN functionality, you need to be able to readily and easily\nrevise the parent information.\n\nTo do this, the second important table in git-p4raw recorded parents;\n\n (branchpath, change, parent_branchpath, parent_change)\nor, if I was stitching on a pre-perforce or otherwise manually converted\nhistory;\n (branchpath, change, parent_sha1)\n\nSo, in the Perl code, I wrote a command called \"find_branches\", which\ncorrelates the information already there in the database with the\nchanges for that revision, and progressively looks for new revisions.\nIt also creates provisional parent information based on the integration\nbreadcrumbs.\n\nWhat I would then do is look at the result in 'gitk', and if there were\nproblems, they could usually be fixed by fiddling with the parent\ninformation, rewinding the export (see 'unexport_commits') and\nre-running it. Sometimes this meant adding a missing merge parent,\nsometimes my fuzzy logic for guessing the merge parents guessed badly.\nObviously, I was also developing the importer along the way, so as well\nas data errors there were bugfixes to make, etc.\n\nThis was not arduous; the speed of postgres' query evaluation (with some\ntuning) and git fast-import meant I was typically exporting at several\n*hundred* commits per second.\n\nAs I had a facility to graft manually converted history using git-p4raw\n(above, that's where the \"parent\" is a commit SHA1, not a Perforce\nrevision number and path), I even went back and found various changes in\nPerforce that looked more like incremental Perl releases, and run the\nscript I had for pre-perforce history over the diff and changelog\ncontained within (by then, it had a personality; it was called the\nTiminator: http://perl5.git.perl.org/perl.git/tag/timinator).\n\nAnyway, with that information in place, you then have all the\ninformation you need to do a test export. The exporter already has all\nof the blobs in the git repository; all it has to do is refer to these\nin a fast-export stream. It marks as it goes along; once it has\nfinished an export batch, it waits for the fast-import process to finish\nsuccessfully, reads all of the SHA1s corresponding to the marks it\nalready emitted, and then updates the database tables with the SHA1s\naccordingly. Due to extensive use of deferred check constraints, only\nthen will Postgres let it commit :-). That way, when I hit \"ctrl+c\"\nalong the way, I knew everything was safe. Restartability and\nrobustness in the face of crashes is very useful for this sort of tool.\n\nAnother strange case which affects some of the largest repositories in\nthe world I don't have an answer for, but suspect it can be represented\nby either subtree merging or by submodules:\n\n mv /trunk/ProjectA /trunk/ProjectB/lib/ProjectA\n\n\"ProjectA\" is now included in \"ProjectB\" - what is the intent of this?\nThe first possibility is a subtree merge, the second is that a submodule\nis desired. How to represent it in git will depend on what happens\nlater. If the directory is moved or copied elsewhere, then it is\nprobably going to be better to represent it as a submodule.\n\nAnd here, the lesson is: people use SVN in ways which defy a single\nmapping into git. This one in particular affects the KDE project\nheavily, as directories are copied around extensively. SVN can remember\nthe history and produce logs, but it requires the entire repository\navailable to be able to do so. Thiago wrote a tool called\nsvn-fast-export-all, which hoped to parse the svnadmin dump file and\nsplit the data into separate repositories as it went, but as it is a\nvery long batch job it is difficult to produce a high quality\nconversion.\n\nImportant points to take from this;\n\n * model the source data cleanly, completely and robustly.\n * start with heuristics, hopefully they will work for people following\nthe SVN guide, but allow for human input for when it doesn't.\n * aim for quick export/rewind, and robust operation.\n * this will make it very easy for revisionists to clean up the\nmistakes of the past\n\nKeep up the good work!\nSam\n"}]}