{"thread":{"id":"25660","subject":"Status of the svn remote helper project (Nov, 2010)","startedAt":"2010-11-07T11:21:29Z","lastAt":"2011-02-11T15:49:15Z","messageCount":22,"participants":["Jonathan Nieder","David Michael Barr","Ramkumar Ramachandra","David Barr","Tomas Carnecky","Sam Vilain"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"155320","messageId":"20101107112129.GA30042@burratino","threadId":"25660","inReplyTo":null,"subject":"Status of the svn remote helper project (Nov, 2010)","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-11-07T11:21:29Z","receivedAt":"2010-11-07T11:21:29Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"The svn remote helper project still has a long way to go. In the\nmeantime, the svn-fe plumbing and Tomas's scripted prototype are\nusable.\n\nHere are some topics that might be roughly in their final form. If\nyou would like to build on one of them, please let me know so I can\nrefrain from rewriting that piece of history.\n\nA merge of these branches is available as\n\n\tgit://repo.or.cz/git/jrn.git vcs-svn-pu\n\nand individual topic branches are also available in that repository,\nthough for convenience they are not in the refs/heads namespace.\n\nThoughts and improvements welcome.\n\n--------------------------------------------------\n[Cooking]\n* jn/svndiff0 (2010-11-06) 24 commits\n - vcs-svn: Allow deltas to copy from preimage\n - vcs-svn: Reject deltas that read past end of preimage\n - vcs-svn: Let deltas use data from postimage\n - vcs-svn: Reject deltas that do not consume all inline data\n - vcs-svn: Check declared number of output bytes\n - vcs-svn: Implement copyfrom_data delta instruction\n - vcs-svn: Read instructions from deltas\n - vcs-svn: Read inline data from deltas\n - vcs-svn: Read the preimage while applying deltas\n - vcs-svn: Skeleton of an svn delta parser\n - compat: helper for detecting unsigned overflow\n - vcs-svn: Learn to check for SVN\\0 magic\n - vcs-svn: Learn to parse variable-length integers\n - vcs-svn: Add code to maintain a sliding view of a file\n - vcs-svn: Allow character-oriented input\n - vcs-svn: Allow input errors to be detected early\n - vcs-svn: Let callers peek ahead to find stream end\n - vcs-svn: Add binary-safe read() function\n - vcs-svn: Improve support for reading large files\n - vcs-svn: Make buffer_skip_bytes() report partial reads\n - vcs-svn: Teach line_buffer to handle multiple input files\n - vcs-svn: Collect line_buffer data in a struct\n - vcs-svn: Replace buffer_read_string() memory pool with a strbuf\n - vcs-svn: Eliminate global byte_buffer[] array\n\nAn SVN-format delta applier. Seems okay, but it has not been heavily\nexercised with real-world deltas.\n\n* db/fast-import-cat-blob (2010-11-07) 3 commits\n - fast-import: let importers retrieve blobs\n - fast-import: clarify documentation of \"feature\" command\n - fast-import: stricter parsing of integer options\n\nAs David says: \"it has some significant consequences\".\n\nA start for bi-directional communication with fast-import (needed by\nsvn-fe to avoid keeping its own database of blobs). Seems to be in\nokay shape.\n\n* db/svn-fe-dumpfile3 (2010-11-07) 6 commits\n - vcs-svn: apply node text deltas\n - Merge branch 'jn/svndiff0' into db/svn-fe-dumpfile3\n - Merge branch 'db/fast-import-cat-blob' into db/svn-fe-dumpfile3\n - vcs-svn: Add output file param to buffer_copy_bytes()\n - vcs-svn: Find basis for deltified nodes; apply node prop deltas\n - vcs-svn: Teach dump parser about new header types\n (this branch uses jn/svndiff0 and db/fast-import-cat-blob.)\n\nAdding support for dumpfiles with deltas (which is pretty close to\nwhat the ra protocol sends over the wire) to svn-fe.\n\nThe tip commit could use some cleaning up.\n\n* rr/svnfe-tests-no-perl (2010-11-07) 1 commit\n - t9010 (svn-fe): Eliminate dependency on svn perl bindings\n\nEjected from the jn/svndiff0 topic. A noninvasive simplification;\nwhat more could one ask for?\n\n* jn/wrappers-no-libz (2010-11-06) 7 commits\n - Remove pack file handling dependency from wrapper.o\n - pack-objects: mark file-local variable static\n - wrapper: give zlib wrappers their own translation unit\n - strbuf: move strbuf_branchname to sha1_name.c\n - path helpers: move git_mkstemp* to wrapper.c\n - wrapper: move odb_* to environment.c\n - wrapper: move xmmap() to sha1_file.c\n\nApproach seems reasonable. More eyes on the tip commit would\nbe comforting.\n\n* xx/wrappers-no-libz-svndiff0 (2010-11-07) 2 commits\n - svn-fe: stop linking to libz and libxdiff\n - Merge branch 'jn/svndiff0' into xx/wrappers-no-libz-svndiff0\n (this branch uses jn/wrappers-no-libz and jn/svndiff0.)\n\nExample application of the jn/wrappers-no-libz topic.\n\n--------------------------------------------------\n[Not picked up yet]\n\n* db/branch-mapper: $gmane/158375\n . contrib/svn-fe: Fast script to remap svn history\n\nCould use a usage example (perhaps a test script).\n\n* tc/remote-helper-usability: $gmane/157860\n . Register new packs after the remote helper is done fetching\n . Properly record history of the notes ref\n . Fix ls-remote output when displaying impure refs\n . Add git-remote-svn\n . Introduce the git fast-import-helper\n . Rename get_mode() to decode_tree_mode() and export it\n . Allow the transport fetch command to add additional refs\n . Allow more than one keepfile in the transport\n . Remote helper: accept ': ' as a response to 'list'\n\nThe fourth-from-top seems a bit hard to review. If it really is\nnecessary to introduce a separate program with a separate interface,\nmaybe a compile-time flag to choose between them would help?\n\n* rr/remote-helper: http://github.com/artagnon/git\n . remote-svn: Write in fetch functionality\n . run-command: Protect the FD 3 from being grabbed\n . remote-svn: Build a pipeline for the import using svnrdump\n . run-command: Extend child_process to include a backchannel FD\n . Allow the transport fetch command to add additional refs\n . Remote helper: accept ': ' as a response to 'list'\n . test-svn-fe: Allow for a dumpfile on stdin\n . contrib/svn-fe: Fast script to remap svn history\n . Add Tom's remote helper for reference\n . Add a stubby remote-svn remote helper\n . Add a correct svndiff applier\n\nWork in progress, waiting on lower levels to be more functional\n(in particular, svn-fe does not support incremental imports yet).\n\n* sb/svn-fe-example: $gmane/159054\n\n--------------------------------------------------\n[Design note (vaporware)]\n\nSee $gmane/157141 for some hints about implementing incremental\nimports.\n\n$gmane means http://thread.gmane.org/gmane.comp.version-control.git\n"},{"id":"155323","messageId":"6A74D35D-A065-44B7-8DA4-6CA78DE067CF@cordelta.com","threadId":"25660","inReplyTo":"20101107112129.GA30042@burratino","subject":"Re: Status of the svn remote helper project (Nov, 2010)","fromName":"David Michael Barr","fromEmail":"david.barr@cordelta.com","sentAt":"2010-11-07T12:06:18Z","receivedAt":"2010-11-07T12:06:18Z","isPatch":false,"sender":{"key":"david.barr@cordelta.com","avatar":"https://avatars.githubusercontent.com/u/220594?v=4"},"body":"Hi all,\n\n> [Cooking]\n> * jn/svndiff0 (2010-11-06) 24 commits\n\n[...]\n\n> An SVN-format delta applier. Seems okay, but it has not been heavily\n> exercised with real-world deltas.\n\nI'm testing this version against the original ASF dump that I used previously.\nMaybe one day we can try against the KDE repo - which is epic in proportions.\n"},{"id":"155324","messageId":"20101107125054.GB16474@kytes","threadId":"25660","inReplyTo":"20101107112129.GA30042@burratino","subject":"Re: Status of the svn remote helper project (Nov, 2010)","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2010-11-07T12:50:56Z","receivedAt":"2010-11-07T12:50:56Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi,\n\nJonathan Nieder writes:\n> The svn remote helper project still has a long way to go. In the\n> meantime, the svn-fe plumbing and Tomas's scripted prototype are\n> usable.\n> \n> Here are some topics that might be roughly in their final form. If\n> you would like to build on one of them, please let me know so I can\n> refrain from rewriting that piece of history.\n> \n> A merge of these branches is available as\n> \n> \tgit://repo.or.cz/git/jrn.git vcs-svn-pu\n\nThanks for doing this! Now we have an up-to-date index that tracks all\nour work :)\n\nA note to the others: If we merge too early, we will be forced to\neither stick with bad decisions we made prematurely, or revert\nthem. Therefore, we have decided to develop this on the side, while\nreporting progress on the list.\n\n> * jn/svndiff0 (2010-11-06) 24 commits\n\nYeah, this requires some rigorous testing with real-world deltas. I'll\nprobably get some EC2 instaces to churn on it after my $EXAMS are over.\n\n> * db/fast-import-cat-blob (2010-11-07) 3 commits\n> As David says: \"it has some significant consequences\".\n\nrr/remote-helper already uses this, but there might be a better way to\ndo it- we should wait and see.\n\n> * db/svn-fe-dumpfile3 (2010-11-07) 6 commits\n\nYou earlier said that it might be possible to justify merging this now\nprovided we supply a wrapper script to make it easy to invoke.\n\n> * rr/svnfe-tests-no-perl (2010-11-07) 1 commit\n\nOk.\n\n> * jn/wrappers-no-libz (2010-11-06) 7 commits\n> * xx/wrappers-no-libz-svndiff0 (2010-11-07) 2 commits\n\nThese two are fairly independent of the rest of the series, no? Maybe\nget these merged separately?\n\n> * db/branch-mapper: $gmane/158375\n\nI've have some mapper ideas from the discussion thread following\nsb/svn-fe-example. I'll finish it after $EXAMS.\n\n> * tc/remote-helper-usability: $gmane/157860\n\nIt has some good ideas that I'm re-using in rr/remote-helper.\n\n> * rr/remote-helper: http://github.com/artagnon/git\n\nFirst, it's the wrong approach: I've hardcoded FD 3 into run-command\nto mean backflow. This ugly inelegant design must be thrown\naway. Second, it's very messy, and half the commits aren't even\nused.\n\nAnyway, I hope it gets the idea across- there's some functionality I\nintend reuse from tc/remote-helper-usability. Also, the \"fetch\"\ncommand works so long as fetching from revision 0 is requested. So the\nimmediate priority is to get svn-fe to support incremental imports.\n\n> * sb/svn-fe-example: $gmane/159054\n\nThe discussion thread following this has some good observations/\nideas.\n\n-- Ram\n"},{"id":"155336","messageId":"20101107174215.GA31242@burratino","threadId":"25660","inReplyTo":"20101107125054.GB16474@kytes","subject":"Re: Status of the svn remote helper project (Nov, 2010)","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-11-07T17:42:15Z","receivedAt":"2010-11-07T17:42:15Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Ramkumar Ramachandra wrote:\n> Jonathan Nieder writes:\n\n>> \tgit://repo.or.cz/git/jrn.git vcs-svn-pu\n>\n> Thanks for doing this! Now we have an up-to-date index that tracks all\n> our work :)\n>\n> A note to the others: If we merge too early, we will be forced to\n> either stick with bad decisions we made prematurely, or revert\n> them. Therefore, we have decided to develop this on the side, while\n> reporting progress on the list.\n\nMore precisely: it would be nice to see the usual flow of patches into\ngit.git; from my point of view, nothing is significantly different\nin that respect from before, and this is just a convenient place for\nremote-svn patches to park and be tested without bothering Junio too\nmuch.\n\nThis should make it easier to keep track of the current state of svn::\nsupport without regularly sending 46-commit patch series to the list.\n\nThanks for the updates and kind words.\nJonathan\n"},{"id":"155353","messageId":"201011081456.26618.david.barr@cordelta.com","threadId":"25660","inReplyTo":"6A74D35D-A065-44B7-8DA4-6CA78DE067CF@cordelta.com","subject":"Re: Status of the svn remote helper project (Nov, 2010)","fromName":"David Barr","fromEmail":"david.barr@cordelta.com","sentAt":"2010-11-08T03:56:26Z","receivedAt":"2010-11-08T03:56:26Z","isPatch":false,"sender":{"key":"david.barr@cordelta.com","avatar":"https://avatars.githubusercontent.com/u/220594?v=4"},"body":"Hi all,\n\n> > [Cooking]\n> > * jn/svndiff0 (2010-11-06) 24 commits\n> \n> [...]\n> \n> > An SVN-format delta applier. Seems okay, but it has not been heavily\n> > exercised with real-world deltas.\n> \n> I'm testing this version against the original ASF dump that I used \npreviously.\n> Maybe one day we can try against the KDE repo - which is epic in \nproportions.\n\nI've successfully tested this series against the ASF repository\n(940,166 revisions) and 5,636,613 blobs were faithfully reproduced.\n\n--\nDavid Barr.\n"},{"id":"155355","messageId":"20101108061140.GB29738@burratino","threadId":"25660","inReplyTo":"201011081456.26618.david.barr@cordelta.com","subject":"Re: Status of the svn remote helper project (Nov, 2010)","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-11-08T06:11:40Z","receivedAt":"2010-11-08T06:11:40Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"David Barr wrote:\n\n>>> [Cooking]\n>>> * jn/svndiff0 (2010-11-06) 24 commits\n[...]\n> I've successfully tested this series against the ASF repository\n> (940,166 revisions) and 5,636,613 blobs were faithfully reproduced.\n\nThanks! Was that using svn-fe, test-svn-fe -d, or some other program\nusing the vcs-svn lib?\n\nJonathan\n"},{"id":"155356","messageId":"201011081720.02961.david.barr@cordelta.com","threadId":"25660","inReplyTo":"20101108061140.GB29738@burratino","subject":"Re: Status of the svn remote helper project (Nov, 2010)","fromName":"David Barr","fromEmail":"david.barr@cordelta.com","sentAt":"2010-11-08T06:20:02Z","receivedAt":"2010-11-08T06:20:02Z","isPatch":false,"sender":{"key":"david.barr@cordelta.com","avatar":"https://avatars.githubusercontent.com/u/220594?v=4"},"body":"Hi,\n\n> >>> [Cooking]\n> >>> * jn/svndiff0 (2010-11-06) 24 commits\n> [...]\n> > I've successfully tested this series against the ASF repository\n> > (940,166 revisions) and 5,636,613 blobs were faithfully reproduced.\n> \n> Thanks! Was that using svn-fe, test-svn-fe -d, or some other program\n> using the vcs-svn lib?\n\nThat was using svn-fe:\nsvn-fe < svn-asf-public-r0:940166 3backflow\n\n--\nDavid Barr.\n"},{"id":"156283","messageId":"20101121063149.GA15449@burratino","threadId":"25660","inReplyTo":"20101107112129.GA30042@burratino","subject":"Status of the svn remote helper project (Nov 2010, #2)","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-11-21T06:31:49Z","receivedAt":"2010-11-21T06:31:49Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Not much to see here. There are lots of patches waiting for review;\nstill especially noteworthy are Tomas's fast-import changes.\n\nIncremental updates after a one-shot conversion by svn-fe are not\nsupported yet. A map from git revisions to svn revision numbers would\nbe needed for that, preferrably such that the time to look up the HEAD\ncommit does not scale with the number of revisions.\n\nA merge of the branches listed below is available as\n\n\tgit://repo.or.cz/git/jrn.git vcs-svn-pu\n\nand individual topic branches are also available in that repository\nin the refs/topics namespace. Please try to base your work on just\nthe topic branches you use; vcs-svn-pu itself is rebuilt each time it\nis updated.\n\nComplaints of all kinds welcome.\n\n--------------------------------------------------\n[Cooking]\n* jn/svndiff0 (2010-11-06) 24 commits\n - vcs-svn: Allow deltas to copy from preimage\n - vcs-svn: Reject deltas that read past end of preimage\n - vcs-svn: Let deltas use data from postimage\n - vcs-svn: Reject deltas that do not consume all inline data\n - vcs-svn: Check declared number of output bytes\n - vcs-svn: Implement copyfrom_data delta instruction\n - vcs-svn: Read instructions from deltas\n - vcs-svn: Read inline data from deltas\n - vcs-svn: Read the preimage while applying deltas\n - vcs-svn: Skeleton of an svn delta parser\n - compat: helper for detecting unsigned overflow\n - vcs-svn: Learn to check for SVN\\0 magic\n - vcs-svn: Learn to parse variable-length integers\n - vcs-svn: Add code to maintain a sliding view of a file\n - vcs-svn: Allow character-oriented input\n - vcs-svn: Allow input errors to be detected early\n - vcs-svn: Let callers peek ahead to find stream end\n - vcs-svn: Add binary-safe read() function\n - vcs-svn: Improve support for reading large files\n - vcs-svn: Make buffer_skip_bytes() report partial reads\n - vcs-svn: Teach line_buffer to handle multiple input files\n - vcs-svn: Collect line_buffer data in a struct\n - vcs-svn: Replace buffer_read_string() memory pool with a strbuf\n - vcs-svn: Eliminate global byte_buffer[] array\n\nWell tested. It's a library without a user except test-svn-fe -d, but\naside from that detail, this series should be ready for wide use.\n\n* db/fast-import-cat-blob (2010-11-07) 3 commits\n - fast-import: Allow cat-blob requests at arbitrary points in stream\n - fast-import: let importers retrieve blobs\n - fast-import: clarify documentation of \"feature\" command\n - fast-import: stricter parsing of integer options\n\nThere are plans for an additional command to print information in\nls-tree format about a path.\n\n* db/recognize-v3 (2010-11-20) 2 commits\n - vcs-svn: Allow simple v3 dumps (no deltas yet)\n - vcs-svn: Error out for v3 dumps\n\nA bugfix and the framework for a feature.\n\n* db/prop-delta (2010-11-20) 16 commits\n - vcs-svn: Simplify handling of deleted properties\n - vcs-svn: Implement Prop-delta handling\n - vcs-svn: Sharpen parsing of property lines\n - vcs-svn: Split off function for handling of individual properties\n - vcs-svn: Make source easier to read on small screens\n - vcs-svn: More dump format sanity checks\n - vcs-svn: Reject path nodes without Node-action\n - vcs-svn: Delay read of per-path properties\n - vcs-svn: Combine repo_replace and repo_modify functions\n - vcs-svn: Replace = Delete + Add\n - vcs-svn: handle_node: Handle deletion case early\n - vcs-svn: Use mark to indicate nodes with included text\n - vcs-svn: Unclutter handle_node by introducing have_props var\n - vcs-svn: Eliminate node_ctx.mark global\n - vcs-svn: Eliminate node_ctx.srcRev global\n - vcs-svn: Check for errors from open()\n (this branch uses db/recognize-v3.)\n\nNeeds review and testing.\n\n* db/text-delta (2010-11-20) 10 commits\n - svn-fe: Test script for handling of dumps with --deltas\n - vcs-svn: Implement text-delta handling\n - Merge branch 'db/fast-import-cat-blob' into db/text-delta\n - vcs-svn: Teach line_buffer about temporary files\n - vcs-svn: Let caller set up sliding window for delta preimage\n - vcs-svn: Read delta preimage from file descriptor\n - vcs-svn: Introduce fd_buffer routines\n - vcs-svn: Introduce repo_read_path to check the content at a path\n - vcs-svn: Internal fast_export_save_blob helper\n - Merge branch 'jn/svndiff0' into db/text-delta\n (this branch uses db/recognize-v3, db/prop-delta, db/fast-import-cat-blob,\n and jn/svndiff0.)\n\nA delta in r36 of does not apply\nwith this brand of svn-fe.\n\n* rr/svnfe-tests-no-perl (2010-11-07) 1 commit\n - t9010 (svn-fe): Eliminate dependency on svn perl bindings\n\nSent to list; hopefully will be in jch and we can stop tracking it\nsoon.\n\n* jn/thinner-wrapper (2010-11-06) 7 commits\n - Remove pack file handling dependency from wrapper.o\n - pack-objects: mark file-local variable static\n - wrapper: give zlib wrappers their own translation unit\n - strbuf: move strbuf_branchname to sha1_name.c\n - path helpers: move git_mkstemp* to wrapper.c\n - wrapper: move odb_* to environment.c\n - wrapper: move xmmap() to sha1_file.c\n\n>From pu.\n\n* xx/thinner-wrapper-svndiff0 (2010-11-07) 2 commits\n - svn-fe: stop linking to libz and libxdiff\n - Merge branch 'jn/svndiff0' into xx/thinner-wrapper-svndiff0\n (this branch uses jn/thinner-wrapper and jn/svndiff0.)\n\n---------------------------------------------------\n[Dropped]\n* db/svn-fe-dumpfile3 (2010-11-07) 6 commits\n - vcs-svn: apply node text deltas\n - Merge branch 'jn/svndiff0' into db/svn-fe-dumpfile3\n - Merge branch 'db/fast-import-cat-blob' into db/svn-fe-dumpfile3\n - vcs-svn: Add output file param to buffer_copy_bytes()\n - vcs-svn: Find basis for deltified nodes; apply node prop deltas\n - vcs-svn: Teach dump parser about new header types\n (this branch uses jn/svndiff0 and db/fast-import-cat-blob.)\n\nEjected in favor of db/recognize-v3, db/prop-delta, and db/text-delta.\n\n--------------------------------------------------\n[Not picked up yet]\n\n* db/branch-mapper: $gmane/158375\n . contrib/svn-fe: Fast script to remap svn history\n\nSent comments. The choices this script makes can be arbitrary at\ntimes.\n\n* tc/remote-helper-usability: $gmane/157860\n . Register new packs after the remote helper is done fetching\n . Properly record history of the notes ref\n . Fix ls-remote output when displaying impure refs\n . Add git-remote-svn\n . Introduce the git fast-import-helper\n . Rename get_mode() to decode_tree_mode() and export it\n . Allow the transport fetch command to add additional refs\n . Allow more than one keepfile in the transport\n . Remote helper: accept ': ' as a response to 'list'\n\nThe fourth-from-top seems a bit hard to review. If it really is\nnecessary to introduce a separate program with a separate interface,\nmaybe a compile-time flag to choose between them would help?\n\n* rr/remote-helper: http://github.com/artagnon/git\n . remote-svn: Write in fetch functionality\n . run-command: Protect the FD 3 from being grabbed\n . remote-svn: Build a pipeline for the import using svnrdump\n . run-command: Extend child_process to include a backchannel FD\n . Allow the transport fetch command to add additional refs\n . Remote helper: accept ': ' as a response to 'list'\n . test-svn-fe: Allow for a dumpfile on stdin\n . contrib/svn-fe: Fast script to remap svn history\n . Add Tom's remote helper for reference\n . Add a stubby remote-svn remote helper\n . Add a correct svndiff applier\n\nWork in progress, waiting on lower levels to be more functional\n(in particular, svn-fe does not support incremental imports yet).\n\n* sb/svn-fe-example: $gmane/159054\n"},{"id":"156284","messageId":"BB713021-7826-4E9E-8576-7D1704BF517C@cordelta.com","threadId":"25660","inReplyTo":"20101121063149.GA15449@burratino","subject":"Re: Status of the svn remote helper project (Nov 2010, #2)","fromName":"David Michael Barr","fromEmail":"david.barr@cordelta.com","sentAt":"2010-11-21T09:38:39Z","receivedAt":"2010-11-21T09:38:39Z","isPatch":false,"sender":{"key":"david.barr@cordelta.com","avatar":"https://avatars.githubusercontent.com/u/220594?v=4"},"body":"Hi all,\n\n> There are plans for an additional command to print information in\n> ls-tree format about a path.\n\nSorry that progress has been quite slow this week.\nMy work-in-progess lays out the syntax of the command,\nthe initial parsing and object lookup.\nI still need to break out the tree and path dereferencing logic.\nI haven't started on printing yet but I presume it should be easy.\n\n> * db/recognize-v3 (2010-11-20) 2 commits\n> - vcs-svn: Allow simple v3 dumps (no deltas yet)\n> - vcs-svn: Error out for v3 dumps\n> \n> A bugfix and the framework for a feature.\n\nThanks for this, it should have gone in much earlier.\n\n> * db/prop-delta (2010-11-20) 16 commits\n> - vcs-svn: Simplify handling of deleted properties\n> - vcs-svn: Implement Prop-delta handling\n> - vcs-svn: Sharpen parsing of property lines\n> - vcs-svn: Split off function for handling of individual properties\n> - vcs-svn: Make source easier to read on small screens\n> - vcs-svn: More dump format sanity checks\n> - vcs-svn: Reject path nodes without Node-action\n> - vcs-svn: Delay read of per-path properties\n> - vcs-svn: Combine repo_replace and repo_modify functions\n> - vcs-svn: Replace = Delete + Add\n> - vcs-svn: handle_node: Handle deletion case early\n> - vcs-svn: Use mark to indicate nodes with included text\n> - vcs-svn: Unclutter handle_node by introducing have_props var\n> - vcs-svn: Eliminate node_ctx.mark global\n> - vcs-svn: Eliminate node_ctx.srcRev global\n> - vcs-svn: Check for errors from open()\n> (this branch uses db/recognize-v3.)\n> \n> Needs review and testing.\n\nTesting against my favourite, the ASF repo, as I write this.\n\n> \n> * db/text-delta (2010-11-20) 10 commits\n> - svn-fe: Test script for handling of dumps with --deltas\n> - vcs-svn: Implement text-delta handling\n> - Merge branch 'db/fast-import-cat-blob' into db/text-delta\n> - vcs-svn: Teach line_buffer about temporary files\n> - vcs-svn: Let caller set up sliding window for delta preimage\n> - vcs-svn: Read delta preimage from file descriptor\n> - vcs-svn: Introduce fd_buffer routines\n> - vcs-svn: Introduce repo_read_path to check the content at a path\n> - vcs-svn: Internal fast_export_save_blob helper\n> - Merge branch 'jn/svndiff0' into db/text-delta\n> (this branch uses db/recognize-v3, db/prop-delta, db/fast-import-cat-blob,\n> and jn/svndiff0.)\n> \n> A delta in r36 of does not apply\n> with this brand of svn-fe.\n\nThat's odd, I was able to import up to r354 before receiving:\nfatal: missing newline after cat-blob response\n\nSo there's some regressions to chase down since the last roll-up.\n\nThat's all I've got for now.\n\n--\nDavid Barr.\n"},{"id":"156297","messageId":"20101121230613.GA24397@burratino","threadId":"25660","inReplyTo":"BB713021-7826-4E9E-8576-7D1704BF517C@cordelta.com","subject":"Re: Status of the svn remote helper project (Nov 2010, #2)","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-11-21T23:06:13Z","receivedAt":"2010-11-21T23:06:13Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"David Michael Barr wrote:\n> Jonathan Nieder wrote:\n\n>> A delta in r36 of does not apply\n>> with this brand of svn-fe.\n>\n> That's odd, I was able to import up to r354 before receiving:\n> fatal: missing newline after cat-blob response\n\nApparently sometimes deltas use the whole preimage and sometimes they\ndon't.\n\nHere's a fix (still needs a simple reproduction script).\n\nSigned-off-by: Jonathan Nieder \n---\ndiff --git a/test-svn-fe.c b/test-svn-fe.c\nindex 64f63cf..71de02b 100644\n--- a/test-svn-fe.c\n+++ b/test-svn-fe.c\n@@ -11,7 +11,7 @@\n int main(int argc, char *argv[])\n {\n \tstatic const char test_svnfe_usage[] =\n-\t\t\"test-svn-fe ( | [-d] )\";\n+\t\t\"test-svn-fe ( | [-d] [] )\";\n \tif (argc < 2)\n \t\tusage(test_svnfe_usage);\n \tif (argc == 2) {\n@@ -22,16 +22,26 @@ int main(int argc, char *argv[])\n \t\tsvndump_reset();\n \t\treturn 0;\n \t}\n-\tif (argc == 5 && !strcmp(argv[1], \"-d\")) {\n+\tif ((argc == 5 || argc == 6) && !strcmp(argv[1], \"-d\")) {\n+\t\tchar **arg = argv + 2;\n \t\tstruct line_buffer delta = LINE_BUFFER_INIT;\n-\t\tint preimage_fd = open(argv[2], O_RDONLY);\n+\t\tint preimage_fd = open(*arg++, O_RDONLY);\n \t\tstruct view preimage_view = {preimage_fd, 0, STRBUF_INIT};\n+\t\toff_t preimage_len;\n \t\tif (preimage_fd < 0)\n \t\t\tdie_errno(\"cannot open preimage\");\n-\t\tif (buffer_init(&delta, argv[3]))\n+\t\tif (argc == 6) {\n+\t\t\tpreimage_len = (off_t) strtoull(*arg++, NULL, 0);\n+\t\t} else {\n+\t\t\tstruct stat st;\n+\t\t\tif (fstat(preimage_fd, &st))\n+\t\t\t\tdie_errno(\"cannot stat preimage\");\n+\t\t\tpreimage_len = st.st_size;\n+\t\t}\n+\t\tif (buffer_init(&delta, *arg++))\n \t\t\tdie_errno(\"cannot open delta\");\n-\t\tif (svndiff0_apply(&delta, (off_t) strtoull(argv[4], NULL, 0),\n-\t\t\t\t &preimage_view, stdout))\n+\t\tif (svndiff0_apply(&delta, (off_t) strtoull(*arg++, NULL, 0),\n+\t\t\t\t &preimage_view, preimage_len, stdout))\n \t\t\treturn 1;\n \t\tif (close(preimage_fd))\n \t\t\tdie_errno(\"cannot close preimage\");\ndiff --git a/vcs-svn/fast_export.c b/vcs-svn/fast_export.c\nindex ceb1fc5..02456cf 100644\n--- a/vcs-svn/fast_export.c\n+++ b/vcs-svn/fast_export.c\n@@ -119,6 +119,7 @@ static long apply_delta(uint32_t mark, off_t len, struct line_buffer *input,\n \t\t\tuint32_t old_mark, uint32_t old_mode)\n {\n \tlong ret;\n+\toff_t preimage_len = 0;\n \tstruct view preimage = {REPORT_FILENO, 0, STRBUF_INIT};\n \tFILE *out;\n \n@@ -130,13 +131,12 @@ static long apply_delta(uint32_t mark, off_t len, struct line_buffer *input,\n \t\tprintf(\"cat-blob :%\"PRIu32\"\\n\", old_mark);\n \t\tfflush(stdout);\n \t\tresponse = get_response_line();\n-\t\t/* Not necessary, just for robustness */\n-\t\tif (parse_cat_response_line(response, &dummy))\n+\t\tif (parse_cat_response_line(response, &preimage_len))\n \t\t\tdie(\"invalid cat-blob response: %s\", response);\n \t}\n \tif (old_mode == REPO_MODE_LNK)\n \t\tstrbuf_addstr(&preimage.buf, \"link \");\n-\tif (svndiff0_apply(input, len, &preimage, out))\n+\tif (svndiff0_apply(input, len, &preimage, preimage_len, out))\n \t\tdie(\"cannot apply delta\");\n \tif (old_mark) {\n \t\t/* Discard trailing newline from cat-blob-fd. */\ndiff --git a/vcs-svn/svndiff.c b/vcs-svn/svndiff.c\nindex 308c734..8210561 100644\n--- a/vcs-svn/svndiff.c\n+++ b/vcs-svn/svndiff.c\n@@ -283,7 +283,8 @@ static int apply_one_window(struct line_buffer *delta, off_t *delta_len,\n }\n \n int svndiff0_apply(struct line_buffer *delta, off_t delta_len,\n-\t\t\tstruct view *preimage_view, FILE *postimage)\n+\t\t\tstruct view *preimage_view, off_t preimage_len,\n+\t\t\tFILE *postimage)\n {\n \tassert(delta && preimage_view && postimage);\n \n@@ -302,5 +303,7 @@ int svndiff0_apply(struct line_buffer *delta, off_t delta_len,\n \t\t\treturn error(\"Delta ends early! (%\"PRIu64\" bytes remaining)\",\n \t\t\t (uint64_t) delta_len);\n \t}\n+\tif (move_window(preimage_view, preimage_len, 0))\n+\t\treturn -1;\n \treturn 0;\n }\ndiff --git a/vcs-svn/svndiff.h b/vcs-svn/svndiff.h\nindex bb5afd0..640e04f 100644\n--- a/vcs-svn/svndiff.h\n+++ b/vcs-svn/svndiff.h\n@@ -5,6 +5,7 @@\n #include \"sliding_window.h\"\n \n extern int svndiff0_apply(struct line_buffer *delta, off_t delta_len,\n-\t\t\t\tstruct view *preimage_view, FILE *postimage);\n+\t\t\t\tstruct view *preimage_view, off_t preimage_len,\n+\t\t\t\tFILE *postimage);\n \n #endif\n"},{"id":"156301","messageId":"201011221306.45096.david.barr@cordelta.com","threadId":"25660","inReplyTo":"20101121230613.GA24397@burratino","subject":"Re: Status of the svn remote helper project (Nov 2010, #2)","fromName":"David Barr","fromEmail":"david.barr@cordelta.com","sentAt":"2010-11-22T02:06:44Z","receivedAt":"2010-11-22T02:06:44Z","isPatch":false,"sender":{"key":"david.barr@cordelta.com","avatar":"https://avatars.githubusercontent.com/u/220594?v=4"},"body":"Hi,\n\n> >> A delta in r36 of does not apply\n> >> with this brand of svn-fe.\n> >\n> > That's odd, I was able to import up to r354 before receiving:\n> > fatal: missing newline after cat-blob response\n> \n> Apparently sometimes deltas use the whole preimage and sometimes they\n> don't.\n> \n> Here's a fix (still needs a simple reproduction script).\n\nI'm testing this path along with the following changes.\nThe first just removes a compile-time warning.\nThe second fixes a memory leak.\nSorry, my send-email-fu is not up to scratch.\n\nSigned-off-by: David Barr \n---\ndiff --git a/vcs-svn/fast_export.c b/vcs-svn/fast_export.c\nindex 02456cf..a95a5c9 100644\n--- a/vcs-svn/fast_export.c\n+++ b/vcs-svn/fast_export.c\n@@ -127,7 +127,6 @@ static long apply_delta(uint32_t mark, off_t len, struct \nline_buffer *input,\n die(\"cannot open temporary file for blob retrieval\");\n if (old_mark) {\n const char *response;\n- off_t dummy;\n printf(\"cat-blob :%\"PRIu32\"\\n\", old_mark);\n fflush(stdout);\n response = get_response_line();\n@@ -147,6 +146,7 @@ static long apply_delta(uint32_t mark, off_t len, struct \nline_buffer *input,\n ret = buffer_tmpfile_prepare_to_read(&postimage);\n if (ret < 0)\n die(\"cannot read temporary file for blob retrieval\");\n+ strbuf_release(&preimage.buf);\n return ret;\n }\n \n"},{"id":"157351","messageId":"20101205113717.GH4332@burratino","threadId":"25660","inReplyTo":"20101121063149.GA15449@burratino","subject":"Status of the svn remote helper project (Dec 2010, #1)","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-12-05T11:37:17Z","receivedAt":"2010-12-05T11:37:17Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"It's been a couple of months since I mentioned that it would be\npossible for a simple svn remote helper on top of svn-fe to be usable\nin a couple of weeks. I still believe a read-only importer is _just_\nwithin grasp; but there are plenty of tempting and interesting topics\nto work on in the stack below that, too.\n\nThat said, the collection of topics cooking is already satisfying.\nTime for cleaning up additional branches for the tree seems short;\npatch series and branches that are ready for wider testing as-is would\nbe fantastic (and should be based on a merge of 'master' and as few of\nthe topics listed below as possible).\n\nExcluding changes taken from the 'jch' branch, relative to last time\nwe have\n\n Makefile | 4 +-\n contrib/svn-fe/svn-filter-root.py | 107 ++++++++++++++++++++++++\n contrib/svn-fe/testme.sh | 4 +-\n fast-import.c | 164 ++++++++++++++++++++++++++++++++++---\n t/t9011-svn-da.sh | 10 +-\n t/t9300-fast-import.sh | 6 +-\n test-treap.c | 11 ++-\n vcs-svn/repo_tree.c | 4 +-\n vcs-svn/sliding_window.c | 11 ++-\n vcs-svn/trp.h | 3 +-\n vcs-svn/trp.txt | 10 ++-\n 11 files changed, 298 insertions(+), 36 deletions(-)\n\nThe added fast-import code implements the 'ls' command.\n\nMost of the changes are one-liners and small cleanups. The most\nvisible change in practice should be the fix for the mysterious\nhistorical-files-appearing-as-directories bug in repo_tree.\n\nAs always, a merge of the branches listed below is available as\n\n\tgit://repo.or.cz/git/jrn.git vcs-svn-pu\n\nand individual topic branches are also available in that repository\nunder the refs/topics namespace.\n\n--------------------------------------------------\n[Cooking]\n* jn/svndiff0 (2010-11-06) 24 commits\n- vcs-svn: Allow deltas to copy from preimage\n- vcs-svn: Reject deltas that read past end of preimage\n- vcs-svn: Let deltas use data from postimage\n- vcs-svn: Reject deltas that do not consume all inline data\n- vcs-svn: Check declared number of output bytes\n- vcs-svn: Implement copyfrom_data delta instruction\n- vcs-svn: Read instructions from deltas\n- vcs-svn: Read inline data from deltas\n- vcs-svn: Read the preimage while applying deltas\n- vcs-svn: Skeleton of an svn delta parser\n- compat: helper for detecting unsigned overflow\n- vcs-svn: Learn to check for SVN\\0 magic\n- vcs-svn: Learn to parse variable-length integers\n- vcs-svn: Add code to maintain a sliding view of a file\n- vcs-svn: Allow character-oriented input\n- vcs-svn: Allow input errors to be detected early\n- vcs-svn: Let callers peek ahead to find stream end\n- vcs-svn: Add binary-safe read() function\n- vcs-svn: Improve support for reading large files\n- vcs-svn: Make buffer_skip_bytes() report partial reads\n- vcs-svn: Teach line_buffer to handle multiple input files\n- vcs-svn: Collect line_buffer data in a struct\n- vcs-svn: Replace buffer_read_string() memory pool with a strbuf\n- vcs-svn: Eliminate global byte_buffer[] array\n\nWell tested and should be ready for wide use.\n\nIs there some tool (a hex editor?) that can be used to easily read\nand write deltas? The \"printf and test against 'svnadmin load'\"\nmethod is a bit time confusing.\n\n* db/recognize-v3 (2010-11-20) 2 commits\n- vcs-svn: Allow simple v3 dumps (no deltas yet)\n- vcs-svn: Error out for v3 dumps\n\nA bugfix and the framework for a feature. Probably the bugfix\nshould be pushed toward 'maint'.\n\n* db/prop-delta (2010-11-20) 16 commits\n- vcs-svn: Simplify handling of deleted properties\n- vcs-svn: Implement Prop-delta handling\n- vcs-svn: Sharpen parsing of property lines\n- vcs-svn: Split off function for handling of individual properties\n- vcs-svn: Make source easier to read on small screens\n- vcs-svn: More dump format sanity checks\n- vcs-svn: Reject path nodes without Node-action\n- vcs-svn: Delay read of per-path properties\n- vcs-svn: Combine repo_replace and repo_modify functions\n- vcs-svn: Replace = Delete + Add\n- vcs-svn: handle_node: Handle deletion case early\n- vcs-svn: Use mark to indicate nodes with included text\n- vcs-svn: Unclutter handle_node by introducing have_props var\n- vcs-svn: Eliminate node_ctx.mark global\n- vcs-svn: Eliminate node_ctx.srcRev global\n- vcs-svn: Check for errors from open()\n(this branch uses db/recognize-v3.)\n\nAll but the tip commit are from 'pu'.\n\n* db/fast-import-blob-access (2010-12-04) 5 commits\n- fast-import: add 'ls' command\n- fast-import: Allow cat-blob requests at arbitrary points in stream\n- fast-import: let importers retrieve blobs\n- fast-import: clarify documentation of \"feature\" command\n- fast-import: stricter parsing of integer options\n\nA proof of concept for the protocol targetting another repository\nformat (hg?) would be a great comfort.\n\nSynchronization overhead seems to be a big problem. If someone can\nuse \"top -b\" output to produce a nice timechart explaining this then\nI would be happy to take a look.\n\n* db/fast-import-object-reuse (2010-11-24) 1 commit\n- fast-import: insert new object entries at start of hash bucket\n\nA speedup. Acked by Shawn, taken from 'pu'.\n\n* jn/fast-import-ondemand-checkpoint (2010-11-24) 1 commit\n- fast-import: treat SIGUSR1 as a request to access objects early\n\nTaken from 'pu'.\n\n* jn/svn-fe-makefile (2010-12-04) 1 commit\n- Makefile: dependencies for vcs-svn tests\n\nSeems to have been forgotten? Simplified commit message, otherwise\nunchanged.\n\n* xx/thinner-wrapper-svndiff0 (2010-11-07) 1 commit\n- svn-fe: stop linking to libz and libxdiff\n(this branch uses jn/svndiff0 and jn/thinner-wrapper.)\n\nSimplification. jn/thinner-wrapper is part of 'jch'.\n\n* rr/svnfe-tests-no-perl (2010-11-23) 1 commit\n- t9010 (svn-fe): Eliminate dependency on svn perl bindings\n\n>From 'pu'.\n\n* db/text-delta (2010-11-20) 10 commits\n- fixup! vcs-svn: tweak sliding window code to tolerate excessive\n readahead\n- fixup! svn-fe: Test script for handling of dumps with --deltas\n- svn-fe: Test script for handling of dumps with --deltas\n- vcs-svn: Implement text-delta handling\n- vcs-svn: Teach line_buffer about temporary files\n- vcs-svn: tweak sliding window code to tolerate excessive readahead\n- vcs-svn: Let caller set up sliding window for delta preimage\n- vcs-svn: Read delta preimage from file descriptor\n- vcs-svn: Introduce fd_buffer routines\n- vcs-svn: Introduce repo_read_path to check the content at a path\n- vcs-svn: Internal fast_export_save_blob helper\n- Merge branch 'db/fast-import-blob-access' (early part) into\n db/text-delta\n- Merge branch 'jn/svndiff0' into db/text-delta\n(this branch uses db/recognize-v3, db/prop-delta, db/fast-import-blob-access,\nand jn/svndiff0.)\n\nIt works!\n\n* db/svn-extract-branches (2010-11-20) 1 commit\n- svn-fe: Script to remap svn history\n\nVery rough but let's merge it so it doesn't get forgotten.\n\n* jn/maint-svn-fe (2010-12-05) 2 commits\n- vcs-svn: fix intermittent repo_tree corruption\n- treap: make treap_insert return inserted node\n\nFixes an old bug. Hoping for feedback or an ack from someone familiar\nsvn-fe internals; afterwards, would fast-track to maint.\n\n---------------------------------------------------\n[Graduated]\n* jn/thinner-wrapper (2010-11-06) 7 commits\n - Remove pack file handling dependency from wrapper.o\n - pack-objects: mark file-local variable static\n - wrapper: give zlib wrappers their own translation unit\n - strbuf: move strbuf_branchname to sha1_name.c\n - path helpers: move git_mkstemp* to wrapper.c\n - wrapper: move odb_* to environment.c\n - wrapper: move xmmap() to sha1_file.c\n\nPart of 'jch' now.\n\n--------------------------------------------------\n[Out of tree, stalled]\n\n* tc/remote-helper-usability: $gmane/157860\n . Register new packs after the remote helper is done fetching\n . Properly record history of the notes ref\n . Fix ls-remote output when displaying impure refs\n . Add git-remote-svn\n . Introduce the git fast-import-helper\n . Rename get_mode() to decode_tree_mode() and export it\n . Allow the transport fetch command to add additional refs\n . Allow more than one keepfile in the transport\n . Remote helper: accept ': ' as a response to 'list'\n\nThe fourth-from-top seems a bit hard to review. If it really is\nnecessary to introduce a separate program with a separate interface,\nmaybe a compile-time flag to choose between them would help?\n\n* rr/remote-helper: http://github.com/artagnon/git\n . remote-svn: Write in fetch functionality\n . run-command: Protect the FD 3 from being grabbed\n . remote-svn: Build a pipeline for the import using svnrdump\n . run-command: Extend child_process to include a backchannel FD\n . Allow the transport fetch command to add additional refs\n . Remote helper: accept ': ' as a response to 'list'\n . test-svn-fe: Allow for a dumpfile on stdin\n . contrib/svn-fe: Fast script to remap svn history\n . Add Tom's remote helper for reference\n . Add a stubby remote-svn remote helper\n . Add a correct svndiff applier\n\nWork in progress, waiting on lower levels to be more functional\n(in particular, svn-fe does not support incremental imports yet).\n\n* sb/svn-fe-example: $gmane/159054\n"},{"id":"157619","messageId":"4CFFCDCD.9060602@dbservice.com","threadId":"25660","inReplyTo":"20101205113717.GH4332@burratino","subject":"Re: Status of the svn remote helper project (Dec 2010, #1)","fromName":"Tomas Carnecky","fromEmail":"tom@dbservice.com","sentAt":"2010-12-08T18:26:21Z","receivedAt":"2010-12-08T18:26:21Z","isPatch":false,"sender":{"key":"tom@dbservice.com","avatar":"https://gravatar.com/avatar/900a300bdd1a8bbe086008ad78210bbee2ad2803b7d50a5cba04c1e9404bd6d2?d=mp&s=160"},"body":" On 12/5/10 12:37 PM, Jonathan Nieder wrote:\n> --------------------------------------------------\n> [Out of tree, stalled]\n>\n> * tc/remote-helper-usability: $gmane/157860\n> . Register new packs after the remote helper is done fetching\n> . Properly record history of the notes ref\n> . Fix ls-remote output when displaying impure refs\n> . Add git-remote-svn\n> . Introduce the git fast-import-helper\n> . Rename get_mode() to decode_tree_mode() and export it\n> . Allow the transport fetch command to add additional refs\n> . Allow more than one keepfile in the transport\n> . Remote helper: accept ': ' as a response to 'list'\n>\n> The fourth-from-top seems a bit hard to review. If it really is\n> necessary to introduce a separate program with a separate interface,\n> maybe a compile-time flag to choose between them would help?\n\nI simplified the code and the requirements on fast-import are much \nlighter now. All I need is a way to tell fast-import to stop writing \nrefs and after each commit write its sha1 to stdout. It's possible to \nmodify fast-import.c with a small patch to make it behave like that. \nHowever, I haven't followed the svn remote helper that much lately so I \ndon't know whether one of the other patches already modifies fast-import \nin this way.\n\n From the beginning my code was meant to be just an example how the \ninteraction between git and the svn remote helper could look like. For \nexample I save the svn rev <-> sha1 mapping in notes, which is appears \nto work well. I'll take a look if I'll be able to use the svn-fe in my \nscript.\n\ntom\n"},{"id":"157878","messageId":"20101212061437.GA17185@burratino","threadId":"25660","inReplyTo":"4CFFCDCD.9060602@dbservice.com","subject":"fast-import tweaks for remote helpers (Re: Status of the svn remote helper project (Dec 2010, #1))","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-12-12T06:14:37Z","receivedAt":"2010-12-12T06:14:37Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Tomas Carnecky wrote:\n\n> I simplified the code and the requirements on fast-import are much\n> lighter now. All I need is a way to tell fast-import to stop writing\n> refs and after each commit write its sha1 to stdout.\n\nThat's good to hear. What should be the syntax for asking fast-import\nnot to write to a ref? Something like this?\n\n\tcommit\n\tmark :1\n\tcommitter c o mitter now\n\tdata < It's possible to\n> to modify fast-import.c with a small patch to make it behave like\n> that. However, I haven't followed the svn remote helper that much\n> lately so I don't know whether one of the other patches already\n> modifies fast-import in this way.\n\nNo, the patches have mostly been adding commands that send information\nback to the frontend.\n\n cat-blob ( | ):\n\tSends back an old blob along with its length (in\n\tcat-file --batch format). svn-fe uses this to acquire\n\tthe preimage when applying deltas.\n\n ls :\n\tSends back information about the current state of a path\n\tin the commit being prepared (as a single line in ls-tree\n\tformat). svn-fe uses this to move around files and to find\n\ta to use with cat-blob when applying deltas.\n\n ls ( | ) :\n\tSends back information about a path in a previous revision\n\t(tag, commit, or tree), in ls-tree format.\n\n M 040000 ( | ) :\n\tLike \"M 100644 \", replaces an entry in the\n\tactive commit with content of the frontend's choice. This\n\tgets used to copy in old directories.\n\n> From the beginning my code was meant to be just an example how the\n> interaction between git and the svn remote helper could look like.\n\nIt makes a nice demo, too. :)\n\n> For example I save the svn rev <-> sha1 mapping in notes, which is\n> appears to work well. I'll take a look if I'll be able to use the\n> svn-fe in my script.\n\nsvn-fe needs a fast mapping svn rev -> sha1; it currently uses a marks\nfile for that. (In the back of my mind, I have the idea of using a\nfile that allows O(1) access, perhaps of the form\n\n\t NL\n\t NL\n\t...\n\nbut as Ram has noted, keeping the whole table in memory is pretty\ncheap already.) A remote helper needs a fast mapping sha1 -> svn rev,\nand imho notes are ideal for that[2].\n\nThe way I imagine it, the authoritative mapping is in notes and the\nreverse mapping (e.g. in a marks file) is rebuilt when needed.\n\n[1] remote-helper branch at git://github.com/wereHamster/git.git\n[2] Why? When a project switches from one svn server to another,\nrevision numbers tend to change, so revision numbers are not permanent\nenough to belong in the commit message imho. (If only git-notes had\nexisted when git svn was written...)\n"},{"id":"157882","messageId":"4D049BA5.1060509@vilain.net","threadId":"25660","inReplyTo":"20101212061437.GA17185@burratino","subject":"Re: fast-import tweaks for remote helpers (Re: Status of the svn remote helper project (Dec 2010, #1))","fromName":"Sam Vilain","fromEmail":"sam@vilain.net","sentAt":"2010-12-12T09:53:41Z","receivedAt":"2010-12-12T09:53:41Z","isPatch":false,"sender":{"key":"sam@vilain.net","avatar":"https://gravatar.com/avatar/8fc840ca854dbf6f7065b4335e3b934951c1dca3b11db688e95e471901f8f4a8?d=mp&s=160"},"body":"On 12/12/10 19:14, Jonathan Nieder wrote:\n> That's good to hear. What should be the syntax for asking fast-import\n> not to write to a ref? Something like this?\n>\n> \tcommit\n> \tmark :1\n> \tcommitter c o mitter now\n> \tdata< \t...\n>\n> Writing the sha1 as each commit is written: how early does the\n> frontend need access to the sha1? Would a facility to report marks\n> back to the frontend at the end of the stream take care of it?\n\nWhat happened to --report-fd ?\n\n> (In the back of my mind, I have the idea of using a\n> file that allows O(1) access, perhaps of the form\n>\n> \t NL\n> \t NL\n> \t...\n\nThis doesn't scale to many branches; git-svn started with that and had \nto use a b-tree in the end. Eg, consider a repository with 10,000 \nbranches and 600,000 revisions.\n\nThanks for continuing this work, it is most interesting to follow.\n\nCheers,\nSam\n"},{"id":"157892","messageId":"20101212171633.GB18847@burratino","threadId":"25660","inReplyTo":"4D049BA5.1060509@vilain.net","subject":"Re: fast-import tweaks for remote helpers","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2010-12-12T17:16:33Z","receivedAt":"2010-12-12T17:16:33Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Sam Vilain wrote:\n\n> What happened to --report-fd ?\n\nThe patch still works. The main problem with report-fd is that it\nintroduced a synchronization point after every commit: the frontend\nhas to read the commit id before fast-import will continue.\n\nSo if the reports can be made optional (\"report _this_ commit\") or\nbatched (\"report all marked commits\") then the result would be easier\nto use imho.\n\n>> (In the back of my mind, I have the idea of using a\n>> file that allows O(1) access, perhaps of the form\n>>\n>>\t NL\n>>\t NL\n>>\t...\n>\n> This doesn't scale to many branches\n\nRight. Another reason to delay getting rid of the git branch with the\nfull repo.\n"},{"id":"158994","messageId":"20110105212018.GA22975@burratino","threadId":"25660","inReplyTo":"20101212171633.GB18847@burratino","subject":"fast-import --report-fd (Re: fast-import tweaks for remote helpers)","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-01-05T21:20:18Z","receivedAt":"2011-01-05T21:20:18Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jonathan Nieder wrote:\n> Sam Vilain wrote:\n\n>> What happened to --report-fd ?\n>\n> The patch still works. The main problem with report-fd is that it\n> introduced a synchronization point after every commit: the frontend\n> has to read the commit id before fast-import will continue.\n\nCorrection: more precisely, that was and is the main problem with\nsvn-fe's use of bidirectional communication. An application like\nTom's remote helper would probably not suffer so much from it, since\ncommit ids are just queued up as long as the pipe doesn't fill before\nthe frontend reads any. It is transactions like\n\n\tFI>\tr5 = 734987a9878b97c879c798a897c897ac\n\tFE>\tcat 734987a9878b97c879c798a897c897ac\n\tFI>\t734987a9878b97c879c798a897c897ac commit 448\n\t\ttree 8d5bcf0f24bdfea1fdab8d39ba3c8ba91a52547c\n\t\tparent 84279592b8b5816d00300ba5d4412adf05cc80d6\n\t\tparent 3ca7353cab4ed6c7efac0c8d7477c87112fc7350\n\t\tauthor Junio C Hamano 1294187068 -0800\n\t\tcommitter Junio C Hamano 1294187068 -0800\n\n\t\tMerge branch 'sr/gitweb-hilite-more' into pu\n\n\t\t* sr/gitweb-hilite-more:\n\t\t gitweb: remove unnecessary test when closing file descriptor\n\t\t gitweb: add extensions to highlight feature map\n\n\tFE>\tcat 8d5bcf0f24bdfea1fdab8d39ba3c8ba91a52547c \"main.c\"\n\n(i.e., round-trips) that were and are creating overhead in svn-fe.\nSee [1] if curious about details.\n\nSo please don't be dissuaded by the nonsense I sent. :)\n\n[1] http://colabti.org/irclogger/irclogger_log_search/git-devel?search=overhead&action=search\n"},{"id":"159007","messageId":"20110105233915.GB22975@burratino","threadId":"25660","inReplyTo":"20101205113717.GH4332@burratino","subject":"Status of the svn remote helper project (Jan 2011, #1)","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-01-05T23:39:15Z","receivedAt":"2011-01-05T23:39:15Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Here are the topics that are cooking in vcs-svn-pu.\n\nHopefully v1.7.4-rc0 is treating you well and free git time has been\ngoing into finding and fixing regressions. So if you are bored and\nlooking for something to do, please skip to [1] and ignore the rest of\nthis message.\n\nDecember was a busy month. Excluding changes from the 'jch' branch:\n\n 39 files changed, 1472 insertions(+), 2020 deletions(-)\n\nThat breaks down as\n\n 1.7% Documentation/\n 1.8% contrib/svn-fe/\n 17.5% t/\n 65.4% vcs-svn/\n\nThe other 13%:\n\n .gitignore | 3 -\n Makefile | 17 +--\n fast-import.c | 134 ++++++++-----\n quote.h | 3 +-\n\nUsers will probably notice that svn-fe requires ls and cat-blob\nsupport from fast-import (alas); hopefully the lower memory footprint,\ncode simplification and incremental import support can justify that\ncost.\n\nThe fast-import changes are a mixture of enhancements to the \"ls\"\ncommand and general optimizations. The optimizations are not in their\nfinal form and likely have bugs but are being sent out now for some\nearly exposure.\n\nAs always, a merge of the branches listed below is available as\n\n\tgit://repo.or.cz/git/jrn.git vcs-svn-pu\n\nand individual topic branches are available in that repository under\nthe refs/topics namespace.\n\nLet's get svn-fe3 polished so when the next merge window comes around\nit is ready to be merged quickly. \n\n--------------------------------------------------\n[Graduated to \"master\"]\n* db/fast-import-object-reuse (2010-11-24) 1 commit\n - fast-import: insert new object entries at start of hash bucket\n\n* jn/fast-import-ondemand-checkpoint (2010-11-24) 1 commit\n - fast-import: treat SIGUSR1 as a request to access objects early\n\n* jn/svn-fe-makefile (2010-12-04) 1 commit\n - Makefile: dependencies for vcs-svn tests\n\n* rr/svnfe-tests-no-perl (2010-11-23) 1 commit\n - t9010 (svn-fe): Eliminate dependency on svn perl bindings\n\n* jn/maint-svn-fe (2010-12-05) 2 commits\n - vcs-svn: fix intermittent repo_tree corruption\n - treap: make treap_insert return inserted node\n\n* db/fast-import-blob-access (2010-12-04) 4 commits\n - fast-import: Allow cat-blob requests at arbitrary points in stream\n - fast-import: let importers retrieve blobs\n - fast-import: clarify documentation of \"feature\" command\n - fast-import: stricter parsing of integer options\n\nThe old tip commit that adds an 'ls' command was reworked (see below).\n\n* jn/fast-import-ondemand-checkpoint (2010-11-24) 1 commit\n - fast-import: treat SIGUSR1 as a request to access objects early\n\n--------------------------------------------------\n[New Topics]\n* jn/line-buffer-error (2010-12-28) 4 commits\n - vcs-svn: improve reporting of input errors\n - vcs-svn: make buffer_copy_bytes return length read\n - vcs-svn: make buffer_skip_bytes return length read\n - vcs-svn: allow input errors to be detected promptly\n\n>From jn/svndiff0 but expanded, waiting for feedback from the list.\nThese let the calling function take care of reporting the error with\nmore context; is that worth it or would it make more sense to die()\ndirectly?\n\n* jn/line-buffer-large-file (2010-12-24) 1 commit\n - vcs-svn: improve support for reading large files\n\n>From jn/svndiff0. Will merge soon if there are no objections.\n\n* jn/line-buffer (2011-01-02) 14 commits\n - Merge branch 'jn/line-buffer-large-file' into jn/line-buffer\n - Merge branch 'jn/line-buffer-error' into jn/line-buffer\n - vcs-svn: teach line_buffer about temporary files\n - vcs-svn: allow input from file descriptor\n - vcs-svn: allow character-oriented input\n - vcs-svn: add binary-safe read function\n - t0081 (line-buffer): add buffering tests\n - vcs-svn: tweak test-line-buffer to not assume line-oriented input\n - tests: give vcs-svn/line_buffer its own test script\n - vcs-svn: make test-line-buffer input format more flexible\n - vcs-svn: teach line_buffer to handle multiple input files\n - vcs-svn: collect line_buffer data in a struct\n - vcs-svn: replace buffer_read_string memory pool with a strbuf\n - vcs-svn: eliminate global byte_buffer\n (this branch uses jn/line-buffer-error and jn/line-buffer-large-file.)\n\n>From jn/svndiff0 and db/text-delta. Putting temporary files with\nmeaningless names in /tmp is unfortunate (maybe buffer_tmpfile_init\ncould use a filename prefix argument?).\n\n* jn/unsigned-overflow (2010-12-25) 1 commit\n - compat: helper for detecting unsigned overflow\n\n>From jn/svndiff0. Should submit for separate inclusion.\n\n* jn/sliding-window (2011-01-02) 1 commit\n - vcs-svn: learn to maintain a sliding view of a file\n (this branch uses jn/line-buffer, jn/line-buffer-error, and\n jn/line-buffer-large-file.)\n\n>From jn/svndiff0 but clarified somewhat.\n\n* db/vcs-svn-incremental (2011-01-05) 20 commits\n - svn-fe: WIP testme.sh performance enhancements\n - svn-fe: testme.sh update\n - vcs-svn: use mark from previous import for parent commit\n - vcs-svn: handle filenames with dq correctly\n - vcs-svn: quote paths correctly for ls command\n - vcs-svn: eliminate repo_tree structure\n - vcs-svn: add a comment before each commit\n - vcs-svn: simplify repo_modify_path and repo_copy\n - vcs-svn: prepare to eliminate repo_tree structure\n - vcs-svn: do not rely on marks for old blobs\n - vcs-svn: split off function to export result from delta application\n - vcs-svn: make apply_delta caller retrieve preimage\n - vcs-svn: explicitly close streams used for delta application at exit\n - vcs-svn: introduce cat_mark function to retrieve a marked blob\n - vcs-svn: save marks for imported commits\n - vcs-svn: use higher mark numbers for blobs\n - vcs-svn: check for errors reading from cat-blob-fd\n - quote.h: simplify the inclusion\n - Makefile: update dependencies for test-svn-fe.c\n - Merge branch 'db/fast-import-blob-access' into db/vcs-svn-incremental\n (this branch uses db/text-delta, db/prop-delta, jn/svndiff0,\n jn/sliding-window, jn/line-buffer, jn/line-buffer-error,\n jn/line-buffer-large-file, and db/fast-import-blob-access.)\n\nSupport for importing different revs in different svn-fe runs.\n\n* db/optimize-vcs-svn (2011-01-05) 9 commits\n - vcs-svn: use strchr to find RFC822 delimiter\n - vcs-svn: drop obj_pool.h\n - vcs-svn: drop trp.h\n - vcs-svn: drop string_pool\n - vcs-svn: factor out usage of string_pool\n - vcs-svn: implement perfect hash for top-level keys\n - vcs-svn: implement perfect hash for node-prop keys\n - vcs-svn: avoid using ls command twice\n - vcs-svn: pass paths through to fast-import\n (this branch uses db/vcs-svn-incremental, db/text-delta, db/prop-delta,\n jn/svndiff0, jn/sliding-window, jn/line-buffer, jn/line-buffer-error,\n jn/line-buffer-large-file, and db/fast-import-blob-access.)\n\nThe diffstat says it all. ;-)\n\n* db/optimize-fast-import (2011-01-05) 3 commits\n - WIP\n - WIP\n - WIP Hash/bitmap combo\n (this branch uses db/fast-import-blob-access.)\n\nVery rough. Testers beware.\n\nI suspect \"struct hash_table\" may provide a simpler approach to\navoiding filling fast-import's tables.\n\n--------------------------------------------------\n[Cooking]\n* jn/svndiff0 (2011-01-05) 11 commits\n - vcs-svn: microcleanup in svndiff0 window-reading code\n - vcs-svn: let deltas use data from preimage\n - vcs-svn: let deltas use data from postimage\n - vcs-svn: verify that deltas consume all inline data\n - vcs-svn: implement copyfrom_data delta instruction\n - vcs-svn: read instructions from deltas\n - vcs-svn: read inline data from deltas\n - vcs-svn: read the preimage when applying deltas\n - vcs-svn: parse svndiff0 window header\n - vcs-svn: skeleton of an svn delta parser\n - Merge branch 'jn/unsigned-overflow' into jn/svndiff0\n (this branch uses jn/sliding-window, jn/line-buffer,\n jn/line-buffer-error, and jn/line-buffer-large-file.)\n\nWell tested and should be ready for wide use.\n\nIs there some tool (a hex editor?) that can be used to read and write\ndeltas? The \"printf and test against 'svnadmin load'\" method is a bit\ntime confusing.\n\n* db/prop-delta (2010-12-09) 18 commits\n - vcs-svn: Simplify handling of deleted properties\n - vcs-svn: Allow change nodes for root of tree (/)\n - vcs-svn: Implement Prop-delta handling\n - vcs-svn: Sharpen parsing of property lines\n - vcs-svn: Split off function for handling of individual properties\n - vcs-svn: Make source easier to read on small screens\n - vcs-svn: More dump format sanity checks\n - vcs-svn: Reject path nodes without Node-action\n - vcs-svn: Delay read of per-path properties\n - vcs-svn: Combine repo_replace and repo_modify functions\n - vcs-svn: Replace = Delete + Add\n - vcs-svn: handle_node: Handle deletion case early\n - vcs-svn: Use mark to indicate nodes with included text\n - vcs-svn: Unclutter handle_node by introducing have_props var\n - vcs-svn: Eliminate node_ctx.mark global\n - vcs-svn: Eliminate node_ctx.srcRev global\n - vcs-svn: Check for errors from open()\n - Allow simple v3 dumps (no deltas yet)\n\nAll but the tip commit are in 'jch'.\n\n* db/fast-import-blob-access (2011-01-03) 3 commits\n - fast-import: add 'ls' command\n - fast-import: treat filemodify with empty tree as delete\n - fast-import: clarify handling of cat-blob feature\n\nA proof of concept for 'cat-blob' and 'ls' support targetting another\nrepository format (hg?) would be a great comfort.\n\nSynchronization overhead seems to be a problem. If someone can use\n\"top -b\" output to produce a nice timechart then I would be happy to\ntake a look.\n\n* db/text-delta (2011-01-04) 5 commits\n - svn-fe: Test script for handling of dumps with --deltas\n - vcs-svn: implement text-delta handling\n - vcs-svn: introduce repo_read_path to check the content at a path\n - Merge branch 'db/prop-delta' into db/text-delta\n - Merge branch 'jn/svndiff0' into db/text-delta\n (this branch uses db/prop-delta, jn/svndiff0, jn/sliding-window,\n jn/line-buffer, jn/line-buffer-error, and jn/line-buffer-large-file.)\n\nStill seems to work. ;-)\n\n* db/svn-extract-branches (2010-11-20) 1 commit\n - svn-fe: Script to remap svn history\n\nVery rough but let's merge it so it doesn't get forgotten.\n\n--------------------------------------------------\n[Ejected]\n* db/recognize-v3 (2010-11-20) 1 commit\n - vcs-svn: Allow simple v3 dumps (no deltas yet)\n\nNot worth maintaining as a separate branch from db/prop-delta.\n\n* xx/thinner-wrapper-svndiff0 (2010-11-07) 1 commit\n - svn-fe: stop linking to libz and libxdiff\n (this branch used jn/svndiff0.)\n\nFixed when jn/svndiff0 was rebased on top of jn/thinner-wrapper.\n\n--------------------------------------------------\n[Out of tree, stalled]\n\n* tc/remote-helper-usability: $gmane/157860\n . Register new packs after the remote helper is done fetching\n . Properly record history of the notes ref\n . Fix ls-remote output when displaying impure refs\n . Add git-remote-svn\n . Introduce the git fast-import-helper\n . Rename get_mode() to decode_tree_mode() and export it\n . Allow the transport fetch command to add additional refs\n . Allow more than one keepfile in the transport\n . Remote helper: accept ': ' as a response to 'list'\n\nI here there has been some work to make this work with the usual\nfast-import.\n\n* rr/remote-helper: http://github.com/artagnon/git\n . [WIP] Temporary commit\n . remote-svn: Write in fetch functionality\n . run-command: Protect the FD 3 from being grabbed\n . remote-svn: Build a pipeline for the import using svnrdump\n . run-command: Extend child_process to include a backchannel FD\n . Allow the transport fetch command to add additional refs\n . Remote helper: accept ': ' as a response to 'list'\n . test-svn-fe: Allow for a dumpfile on stdin\n . Add Tom's remote helper for reference\n . Add a stubby remote-svn remote helper\n . Add a correct svndiff applier\n\nWork in progress, waiting on lower levels to stabilize.\n\n* sb/svn-fe-example: $gmane/159054\n\n[1] Debian: 8 reports[2]\n Fedora: 6 reports[3]\n Gentoo: 1 report[4]\n[2] http://bugs.debian.org/cgi-bin/pkgreport.cgi?src=git;include=tags:upstream;exclude=tags:fixed-upstream;exclude=severity:wishlist\n[3] https://bugzilla.redhat.com/buglist.cgi?component=git&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED\n[4] http://bugs.gentoo.org/buglist.cgi?short_desc_type=allwordssubstr&short_desc=dev-vcs/git&product=Gentoo+Linux&bug_status=UNCONFIRMED&bug_status=NEW&bug_status=ASSIGNED&bug_status=REOPENED\n"},{"id":"159107","messageId":"1AE003AD-CEA1-4187-9778-769B38842597@cordelta.com","threadId":"25660","inReplyTo":"20110105233915.GB22975@burratino","subject":"Re: Status of the svn remote helper project (Jan 2011, #1)","fromName":"David Michael Barr","fromEmail":"david.barr@cordelta.com","sentAt":"2011-01-07T14:00:31Z","receivedAt":"2011-01-07T14:00:31Z","isPatch":false,"sender":{"key":"david.barr@cordelta.com","avatar":"https://avatars.githubusercontent.com/u/220594?v=4"},"body":"Hi,\n\nJonathan wrote:\n> Let's get svn-fe3 polished so when the next merge window comes around\n> it is ready to be merged quickly. \n\nIn short, I have tested the latest rollup and see no regressions when testing\nagainst the ASF and KDE repos.\n\nThank you Jonathan for the hard work, the outstanding patches are\nbecoming quite numerous.\n\n--\nDavid Barr.\n"},{"id":"160882","messageId":"20110211090931.GA27410@elie","threadId":"25660","inReplyTo":"20110105233915.GB22975@burratino","subject":"Plans for the vcs-svn-pu branch","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-02-11T09:09:31Z","receivedAt":"2011-02-11T09:09:31Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"(culled cc list for a new topic)\nHi,\n\nJonathan Nieder wrote:\n\n> Here are the topics that are cooking in vcs-svn-pu.\n\nLong time no pushout. My old plan was to maintain a \"vcs-svn\" branch\nand ask Junio to pull from it periodically. I still plan to do that\nsince I think it's the only feasible way to roll out the larger\nchanges but I might be submitting smaller pieces as patches to the git\nlist directly.\n\nThe purpose of this email is to request help --- if you are\nparticularly interested in some subtopic then I would love to see a\npatch series or pull request on the subject. If I can forget about\nvcs-svn-pu's existence and just review and accept ready-to-apply\npatches on the topic as they come, then building a suitable vcs-svn\nbranch will become very, very simple.\n\nThe good news is that it's been a calm time, so most of these topics\nare probably in good shape.\n\n> * db/fast-import-blob-access (2011-01-03)\n> - fast-import: add 'ls' command\n\nNeeds a feature name. Last seen at [1].\n\n> * db/optimize-vcs-svn (2011-01-05) 9 commits\n> - vcs-svn: drop obj_pool.h\n> - vcs-svn: drop trp.h\n> - vcs-svn: drop string_pool\n[...]\n\nApplying this code removal early would make your local branch juggler\nvery, very happy. Might make sense to pair it with some patches to\nsupport cat-blob-fd in transport-helper's \"import\" support, so the UI\ncould be\n\n\tgit fetch svn-dump::/tmp/file.dump\n\nThat could (perhaps) partially compensate for the loss of convenience\nfrom requiring the cat-blob-fd to be plumbed correctly. It's a shame\nwe weren't louder about svn-fe's command line usage being subject to\nchange. Live and learn. :/\n\n> (this branch uses db/vcs-svn-incremental, db/text-delta, db/prop-delta,\n> jn/svndiff0, jn/sliding-window, jn/line-buffer, jn/line-buffer-error,\n> jn/line-buffer-large-file, and db/fast-import-blob-access.)\n\nThat's silly --- this long overdue cleanup does not require anything\nexcept db/fast-import-blob-access. Building it on top of\ndb/prop-delta seems okay, since the latter is simple and known to work\nwell.\n\n> * db/vcs-svn-incremental (2011-01-05) 20 commits\n[...]\n> (this branch uses db/text-delta, db/prop-delta, jn/svndiff0,\n> jn/sliding-window, jn/line-buffer, jn/line-buffer-error,\n> jn/line-buffer-large-file, and db/fast-import-blob-access.)\n>\n> Support for importing different revs in different svn-fe runs.\n\nKiller feature. After db/optimize-vcs-svn it is easy. It does not\nneed to wait for Text-delta and svndiff0 support.\n\n[...]\n\nWith those three topics out of the way, the rest of the wishlist\nshould be easy. Most of the patches are already written.\n\n - line-buffer enhancements to support Text-delta (multiple buffers,\n sliding window, temporary files)\n - svndiff0 parser/interpreter\n - Text-delta support\n - Prop-delta support\n - support for large files\n - improved error handling\n - fast-import scalability fixes\n"},{"id":"160888","messageId":"20110211103643.GA29005@elie","threadId":"25660","inReplyTo":"20110211090931.GA27410@elie","subject":"[PATCH] svn-fe: warn about experimental status","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2011-02-11T10:36:44Z","receivedAt":"2011-02-11T10:36:44Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"svn-fe is young and some coming cleanups might involve backward\nincompatible UI changes. Add some words of warning to the manual so\nearly adopters that are not following the project closely don't get\nburned.\n\nSigned-off-by: Jonathan Nieder \n---\nJonathan Nieder wrote:\n\n> Applying this code removal early would make your local branch juggler\n> very, very happy. Might make sense to pair it with\n[...]\n> we weren't louder about svn-fe's command line usage being subject to\n> change. Live and learn. :/\n\nThat probably wasn't as clear as hoped. All I mean is that svn-fe is\nyoung and although the aforementioned cleanup involves a UI change\n(going from\n\n\t... | svn-fe | git fast-import\n\nto\n\n\tmkfifo replies\n\t... | svn-fe 3replies\n\n), such changes are to be expected. So hopefully no one will be\nconfused, but it would be better to mention the experimental status\nsomewhere in svn-fe.txt, like so:\n\n contrib/svn-fe/svn-fe.txt | 3 +++\n 1 files changed, 3 insertions(+), 0 deletions(-)\n\ndiff --git a/contrib/svn-fe/svn-fe.txt b/contrib/svn-fe/svn-fe.txt\nindex 35f84bd..cd075b9 100644\n--- a/contrib/svn-fe/svn-fe.txt\n+++ b/contrib/svn-fe/svn-fe.txt\n@@ -18,6 +18,9 @@ Subversion repository mirrored on the local disk. Remote Subversion\n repositories can be mirrored on local disk using the `svnsync`\n command.\n \n+Note: this tool is very young. The details of its commandline\n+interface may change in backward incompatible ways.\n+\n INPUT FORMAT\n ------------\n Subversion's repository dump format is documented in full in\n-- \n1.7.4\n"},{"id":"160900","messageId":"20110211154910.GB7335@kytes","threadId":"25660","inReplyTo":"20110211090931.GA27410@elie","subject":"Re: Plans for the vcs-svn-pu branch","fromName":"Ramkumar Ramachandra","fromEmail":"artagnon@gmail.com","sentAt":"2011-02-11T15:49:15Z","receivedAt":"2011-02-11T15:49:15Z","isPatch":false,"sender":{"key":"r@artagnon.com","avatar":"https://avatars.githubusercontent.com/u/37226?v=4"},"body":"Hi,\n\nJonathan Nieder writes:\n> Jonathan Nieder wrote:\n> \n> > Here are the topics that are cooking in vcs-svn-pu.\n\nThanks for the elaborate email. Some updates from my side:\n- I've rewritten most of the svnload parser to resemble fast-import,\n and I'd like some preliminary feedback on the design.\n- Although most of the dependent infrastructure is in place now, the\n remote-helper branch is still lagging. I'll shortly look into this.\n\nSigned-off-by: Ramkumar Ramachandra \n--8<--\n/*\n * Produce a dumpfile v3 from a fast-import stream.\n * Load the dump into the SVN repository with:\n * svnrdump load 1170199019 +0530 */\n\tstrbuf_reset(&(identp->name));\n\tstrbuf_reset(&(identp->email));\n\n\tif (!buf)\n\t\tgoto error;\n\tif (!(tz_off = strrchr(buf, ' ')))\n\t\tgoto error;\n\t*tz_off++ = '\\0';\n\tif (!(t = strrchr(buf, ' ')))\n\t\tgoto error;\n\t*(t - 1) = '\\0'; /* Ignore '>' from email */\n\tt++;\n\ttz_off_buf = atoi(tz_off);\n\tif (tz_off_buf > 1200 || tz_off_buf < -1200)\n\t\tgoto error;\n\ttm_time = time_to_tm(strtoul(t, NULL, 10), tz_off_buf);\n\tstrftime(identp->date, SVN_DATE_LEN + 1, SVN_DATE_FORMAT, tm_time);\n\tif (!(t = strchr(buf, '<')))\n\t\tgoto error;\n\t*(t - 1) = '\\0'; /* Ignore ' <' from email */\n\tt++;\n\n\tstrbuf_add(&(identp->email), t, strlen(t));\n\tstrbuf_add(&(identp->name), buf, strlen(buf));\n\treturn;\nerror:\n\tdie(\"Malformed ident line: %s\", buf);\n}\n\nstatic void skip_optional_lf(void)\n{\n\tint term_char = fgetc(stdin);\n\tif (term_char != '\\n' && term_char != EOF)\n\t\tungetc(term_char, stdin);\n}\n\nstatic void parse_data(struct strbuf *dst)\n{\n\tif (prefixcmp(command_buf.buf, \"data \"))\n\t\tdie(\"Expected 'data n' command, found: %s\", command_buf.buf);\n\n\tif (!prefixcmp(command_buf.buf + 5, \"<<\")) {\n\t\tchar *term = xstrdup(command_buf.buf + 5 + 2);\n\t\tsize_t term_len = command_buf.len - 5 - 2;\n\n\t\tstrbuf_reset(&command_buf);\n\t\tfor (;;) {\n\t\t\tif (read_next_command() == EOF)\n\t\t\t\tdie(\"EOF in data (terminator '%s' not found)\", term);\n\t\t\tif (term_len == command_buf.len\n\t\t\t && !memcmp(term, command_buf.buf, term_len))\n\t\t\t\tbreak;\n\t\t\tif (dst) {\n\t\t\t\tstrbuf_addbuf(dst, &command_buf);\n\t\t\t\tstrbuf_addch(dst, '\\n');\n\t\t\t} else\n\t\t\t\tprintf(\"%s\\n\", command_buf.buf);\n\t\t}\n\t\tfree(term);\n\t} else {\n\t\tuintmax_t length;\n\n\t\tlength = strtoumax(command_buf.buf + 5, NULL, 10);\n\t\tif ((size_t)length < length)\n\t\t\tdie(\"Data is too large to use in this context\");\n\t\tif (!dst) {\n\t\t\tstrbuf_reset(&command_buf);\n\t\t\t/* buffer_copy_bytes(&command_buf, (size_t)length); */\n\t\t} else\n\t\t\tstrbuf_fread(dst, (size_t)length, stdin);\n\t}\n\n\tskip_optional_lf();\n}\n\nstatic const char *get_mode(const char *str, uint16_t *modep)\n{\n\tunsigned char c;\n\tuint16_t mode = 0;\n\n\twhile ((c = *str++) != ' ') {\n\t\tif (c < '0' || c > '7')\n\t\t\treturn NULL;\n\t\tmode = (mode << 3) + (c - '0');\n\t}\n\t*modep = mode;\n\treturn str;\n}\n\nstatic void file_change_m(void)\n{\n\tconst char *p;\n\tconst char *endp;\n\tuint16_t mode;\n\tenum node_kind kind;\n\n\tp = get_mode(command_buf.buf + 2, &mode);\n\tif (!p)\n\t\tdie(\"Corrupt mode: %s\", command_buf.buf);\n\tswitch (mode) {\n\tcase 0644:\n\tcase 0755:\n\t\tmode |= S_IFREG;\n\tcase S_IFREG | 0644:\n\t\tkind = NODE_KIND_NORMAL;\n\t\tbreak;\n\tcase S_IFREG | 0755:\n\t\tkind = NODE_KIND_EXECUTABLE;\n\t\tbreak;\n\tcase S_IFLNK:\n\t\tkind = NODE_KIND_SYMLINK;\n\t\tbreak;\n\tcase S_IFGITLINK:\n\t\tdie(\"Gitlinks unsupported\"); /* TODO */\n\tcase S_IFDIR:\n\t\tdie(\"Subdirectories unsupported\"); /* TODO */\n\tdefault:\n\t\tdie(\"Corrupt mode: %s\", command_buf.buf);\n\t}\n\n\tif (!prefixcmp(p, \"inline\"))\n\t\tp += 6;\n\telse\n\t\tdie (\"Non-inlined data unsupported\");\n\tif (*p++ != ' ')\n\t\tdie(\"Missing space after dataref: %s\", command_buf.buf);\n\n\t/* parse out path into path_d */\n\tstrbuf_reset(&path_d);\n\tif (!unquote_c_style(&path_d, p, &endp)) {\n\t\tif (*endp)\n\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n\t} else\n\t\tstrbuf_addstr(&path_d, p);\n\n\tdump_export_m(path_d.buf, kind);\n\tread_next_command();\n\tparse_data(NULL); /* parse data and write it to stdout */\n}\n\nstatic void file_change_d(void)\n{\n\tconst char *p;\n\tconst char *endp;\n\t\n\tp = command_buf.buf + 2;\n\t/* parse out path into path_d */\n\tstrbuf_reset(&path_d);\n\tif (!unquote_c_style(&path_d, p, &endp)) {\n\t\tif (*endp)\n\t\t\tdie(\"Garbage after path in: %s\", command_buf.buf);\n\t} else\n\t\tstrbuf_addstr(&path_d, p);\n\tdump_export_d(path_d.buf);\n}\n\nstatic void file_change_cr(int rename)\n{\n\tconst char *p;\n\tconst char *endp;\n\n\tp = command_buf.buf + 2;\n\tstrbuf_reset(&path_s);\n\tif (!unquote_c_style(&path_s, p, &endp)) {\n\t\tif (*endp != ' ')\n\t\t\tdie(\"Missing space after source: %s\", command_buf.buf);\n\t} else {\n\t\tendp = strchr(p, ' ');\n\t\tif (!endp)\n\t\t\tdie(\"Missing space after source: %s\", command_buf.buf);\n\t\tstrbuf_add(&path_s, p, endp - p);\n\t}\n\n\tendp++;\n\tif (!*endp)\n\t\tdie(\"Missing destination: %s\", command_buf.buf);\n\n\tp = endp;\n\tstrbuf_reset(&path_d);\n\tif (!unquote_c_style(&path_d, p, &endp)) {\n\t\tif (*endp)\n\t\t\tdie(\"Garbage after destination in: %s\", command_buf.buf);\n\t} else\n\t\tstrbuf_addstr(&path_d, p);\n\n\t/* TODO: Check C \"path/to/subdir\" \"\" */\n\tif (rename)\n\t\tdump_export_d(path_s.buf);\n\tdump_export_c(path_d.buf, path_s.buf, 0);\n}\n\nstatic void parse_new_commit()\n{\n\tchar *branch;\n\n\t/* parse and ignore branch name */\n\tbranch = strchr(command_buf.buf, ' ') + 1;\n\tread_next_command();\n\tif (!prefixcmp(command_buf.buf, \"mark :\"))\n\t\t/* parse and ignore mark line */\n\t\tread_next_command();\n\tif (!prefixcmp(command_buf.buf, \"author \")) {\n\t\tparse_ident(command_buf.buf + 7, &author);\n\t\tread_next_command();\n\t}\n\tif (!prefixcmp(command_buf.buf, \"committer \")) {\n\t\tparse_ident(command_buf.buf + 10, &committer);\n\t\tread_next_command();\n\t}\n\tif (!committer.name.len)\n\t\tdie(\"Missing committer line in stream\");\n\tparse_data(&log_buf);\n\tread_next_command();\n\tif (!prefixcmp(command_buf.buf, \"from \"))\n\t\t/* TODO: Support copyfrom */\n\t\tread_next_command();\n\twhile (!prefixcmp(command_buf.buf, \"merge \"))\n\t\t/* TODO: Support merges */\n\t\tread_next_command();\n\n\t/* file_change_* */\n\twhile (command_buf.len > 0) {\n\t\tif (!prefixcmp(command_buf.buf, \"M \"))\n\t\t\tfile_change_m();\n\t\telse if (!prefixcmp(command_buf.buf, \"D \"))\n\t\t\tfile_change_d();\n\t\telse if (!prefixcmp(command_buf.buf, \"R \"))\n\t\t\tfile_change_cr(1);\n\t\telse if (!prefixcmp(command_buf.buf, \"C \"))\n\t\t\tfile_change_cr(0);\n\t\telse if (!prefixcmp(command_buf.buf, \"N \"))\n\t\t\t; /* ignored */\n\t\telse if (!prefixcmp(command_buf.buf, \"ls \"))\n\t\t\tgoto error; /* TODO */\n\t\telse if (!strcmp(\"deleteall\", command_buf.buf))\n\t\t\tgoto error; /* TODO */\n\t\telse\n\t\t\tbreak;\n\t\tif (read_next_command() == EOF)\n\t\t\tbreak;\n\t}\n\treturn;\nerror:\n\tdie(\"Unsupported command: %s\", command_buf.buf);\n}\n\nvoid parse_new_tag()\n{\n\t/* TODO: Support tags */\n\treturn;\n}\n\nvoid parse_reset_branch()\n{\n\t/* TODO */\n\treturn;\n}\n\nvoid build_svn_author(struct ident *author, struct ident *committer)\n{\n\tchar *t, *email;\n\n\tstrbuf_reset(&svn_author);\n\temail = author->email.len ? author->email.buf : committer->email.buf;\n\tif ((t = strchr(email, '@')))\n\t\tstrbuf_add(&svn_author, email, t - email);\n\telse\n\t\tstrbuf_addstr(&svn_author, t);\n}\n\nvoid svnload_read(void)\n{\n\tchar *val;\n\twhile (read_next_command() != EOF) {\n\t\tif ((val = strchr(command_buf.buf, ' ')))\n\t\t\t*val++ = '\\0';\n\n\t\tif (!strcmp(\"blob\", command_buf.buf))\n\t\t\tdie(\"Non-inlined blobs unsupported\");\n\t\telse if (!prefixcmp(command_buf.buf, \"ls \"))\n\t\t\tgoto error; /* TODO */\n\t\telse if (!prefixcmp(command_buf.buf, \"cat-blob \"))\n\t\t\tgoto error; /* TODO */\n\t\telse if (!prefixcmp(command_buf.buf, \"commit \"))\n\t\t\tparse_new_commit(val);\n\t\telse if (!prefixcmp(command_buf.buf, \"tag \"))\n\t\t\tparse_new_tag(val);\n\t\telse if (!prefixcmp(command_buf.buf, \"reset \"))\n\t\t\tparse_reset_branch(val);\n\t\telse if (!strcmp(command_buf.buf, \"checkpoint\")\n\t\t\t|| !prefixcmp(command_buf.buf, \"progress \")\n\t\t\t|| !prefixcmp(command_buf.buf, \"feature \")\n\t\t\t|| !prefixcmp(command_buf.buf, \"option \"))\n\t\t\t; /* ignored */\n\t\telse\n\t\t\tgoto error;\n\t}\nerror:\n\tdie(\"Unsupported command: %s\", command_buf.buf);\n}\n\nint svnload_init(const char *filename)\n{\n\tif (!(infile = filename ? fopen(filename, \"r\") : stdin))\n\t\tdie(\"Cannot open %s: %s\", filename, strerror(errno));\n\tdump_export_init();\n\treturn 0;\n}\n"}]}