{"thread":{"id":"8680","subject":"parsecvs fails even on simple input","startedAt":"2007-06-22T11:36:25Z","lastAt":"2007-06-27T17:20:21Z","messageCount":5,"participants":["Roman Kagan","Keith Packard","Shawn O. Pearce"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"45562","messageId":"20070622113625.GD12473@rkagan.sw.ru","threadId":"8680","inReplyTo":null,"subject":"parsecvs fails even on simple input","fromName":"Roman Kagan","fromEmail":"rkagan@sw.ru","sentAt":"2007-06-22T11:36:25Z","receivedAt":"2007-06-22T11:36:25Z","isPatch":false,"sender":{"key":"rkagan@sw.ru","avatar":null},"body":"One of the patches recently merged into parsecvs master, namely\n\ncommit f5b3cb849517adfd7790c1bfa84bbb84220e3e7b\nAuthor: Al Viro <viro@zeniv.linux.org.uk>\nDate:   Tue Jan 16 04:15:35 2007 -0500\n\n    [PATCH] generate tree objects just as we calculate changesets\n    \n    ... and don't store the fsckloads of rev_file in ExportGit mode; they\n    are only needed (for now) in ExportGraph one.\n    \n    Tree generation is done directly, without hitting on-disk index.  Very fast\n    now.\n\nbroke parsecvs entirely.  The reproducer (attached) is very simple:\ninitial commit of a just added file.  parsecvs now barfs on it:\n\n# parsecvs a,v\nInitialized empty shared Git repository in .git/\nLoad:                                 a,v ....................*     1 of     1\nPack pack-6b955e2d966143fc957ccd272e9dd822ceaccf25 created\nRemoving unused objects 81%...\nRemoving unused objects 100%...\nDone.\n\nerror: invalid object d0141680ee5324d51a558a0a48c8a867cbc6a47c\nerror: writing tree\nAuthors: No such file or directory\nSave:                              master ....................*     1 of     1\n\n\nThe problem is the following: after that commit parsecvs tries to add\nobjects to the git tree on its own via calls to libgit; however, in\nbetween it runs git-pack-objects.  Thus objects move to pack files\nwithout libgit being aware of it; this results in 'ivalid object'\nerrors.\n\nHowever, the object with that hash exists but is stored on the pack\nfile; if parsecvs is run in the same directory for the second time it\nfinds it there and happily completes.\n\nI haven't yet had the time to dig deeper into this problem and code a\npatch; however, the whole idea of doing part of the job through the\n(unpublished) libgit API and the rest via callouts to git utilities\nlooks like calling for trouble.  Wouldn't it be better to teach parsecvs\nto speak git-fast-import language instead?\n\nRoman.\n\n\nhead\t1.1;\naccess;\nsymbols;\nlocks; strict;\ncomment\t@# @;\n\n\n1.1\ndate\t2007.06.21.12.11.32;\tauthor tstuser;\tstate Exp;\nbranches;\nnext\t;\n\n\ndesc\n@@\n\n\n1.1\nlog\n@test commit\n@\ntext\n@this is test\n@\n"},{"id":"45729","messageId":"1182720667.13289.41.camel@neko.keithp.com","threadId":"8680","inReplyTo":"20070622113625.GD12473@rkagan.sw.ru","subject":"Re: parsecvs fails even on simple input","fromName":"Keith Packard","fromEmail":"keithp@keithp.com","sentAt":"2007-06-24T21:31:07Z","receivedAt":"2007-06-24T21:31:07Z","isPatch":false,"sender":{"key":"keithp@keithp.com","avatar":"https://gravatar.com/avatar/fa1f479cdd51322fe86215c955a81d296bbf66a1fe625f8a12d87a8ec7faf648?d=mp&s=160"},"body":"On Fri, 2007-06-22 at 15:36 +0400, Roman Kagan wrote:\n\n> The problem is the following: after that commit parsecvs tries to add\n> objects to the git tree on its own via calls to libgit; however, in\n> between it runs git-pack-objects.  Thus objects move to pack files\n> without libgit being aware of it; this results in 'ivalid object'\n> errors.\n\nSticking a call to reprepare_packed_git() after the pack creation fixes\nthis nicely.\n\n>  Wouldn't it be better to teach parsecvs\n> to speak git-fast-import language instead?\n\nAvoiding fork/exec is rather important for parsecvs perforamance.\n\n-- \nkeith.packard@intel.com\n"},{"id":"45730","messageId":"20070625045951.GB32223@spearce.org","threadId":"8680","inReplyTo":"1182720667.13289.41.camel@neko.keithp.com","subject":"Re: parsecvs fails even on simple input","fromName":"Shawn O. Pearce","fromEmail":"spearce@spearce.org","sentAt":"2007-06-25T04:59:51Z","receivedAt":"2007-06-25T04:59:51Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Keith Packard <keithp@keithp.com> wrote:\n> On Fri, 2007-06-22 at 15:36 +0400, Roman Kagan wrote:\n> >  Wouldn't it be better to teach parsecvs\n> > to speak git-fast-import language instead?\n> \n> Avoiding fork/exec is rather important for parsecvs perforamance.\n\nThat sort of thing is the entire point behind fast-import.  Its only\none fork+exec to setup the fast-import \"daemon\" in the background,\nand you do everything over a pipe to its stdin.  Including forcing\nit to finish its current packfile and open a new one on the next\nobject (the `checkpoint` command).\n\nfast-import is fast, its input language is fairly simple, and its\nquite stable.  And its only one fork+exec.  That's peanuts compared\nto the disk IO involved in any sizable import process.\n\n-- \nShawn.\n"},{"id":"45923","messageId":"20070627153300.GA27933@rkagan.sw.ru","threadId":"8680","inReplyTo":"1182720667.13289.41.camel@neko.keithp.com","subject":"Re: parsecvs fails even on simple input","fromName":"Roman Kagan","fromEmail":"rkagan@sw.ru","sentAt":"2007-06-27T15:33:00Z","receivedAt":"2007-06-27T15:33:00Z","isPatch":false,"sender":{"key":"rkagan@sw.ru","avatar":null},"body":"On Sun, Jun 24, 2007 at 10:31:07PM +0100, Keith Packard wrote:\n> On Fri, 2007-06-22 at 15:36 +0400, Roman Kagan wrote:\n> \n> > The problem is the following: after that commit parsecvs tries to add\n> > objects to the git tree on its own via calls to libgit; however, in\n> > between it runs git-pack-objects.  Thus objects move to pack files\n> > without libgit being aware of it; this results in 'ivalid object'\n> > errors.\n> \n> Sticking a call to reprepare_packed_git() after the pack creation fixes\n> this nicely.\n\nEhm sort of...  Except that I woudn't call that extern declaration\nnice.\n\nI'm now tracking down another problem which I didn't see before:\nparsecvs apparently doesn't close .git-cvs/log-XXX files and ends up\nexhausting the open file descriptor limit.  I'll update when I have more\ninfo.\n\n> >  Wouldn't it be better to teach parsecvs\n> > to speak git-fast-import language instead?\n> \n> Avoiding fork/exec is rather important for parsecvs perforamance.\n\nAvoiding _one_ fork/exec is certainly not.\n\nOTOH git-fast-import seems to be essentially the public API for the\nparsecvs kind of tasks.  It may be wiser from the maintenance POV to use\nthat instead of direct libgit calls (unless parsecvs is going to land in\nthe git tree).  I'll try to find the time and take a look at this\nsomewhere next week.\n\nRoman.\n"},{"id":"45931","messageId":"1182964822.4031.193.camel@neko.keithp.com","threadId":"8680","inReplyTo":"20070627153300.GA27933@rkagan.sw.ru","subject":"Re: parsecvs fails even on simple input","fromName":"Keith Packard","fromEmail":"keithp@keithp.com","sentAt":"2007-06-27T17:20:21Z","receivedAt":"2007-06-27T17:20:21Z","isPatch":false,"sender":{"key":"keithp@keithp.com","avatar":"https://gravatar.com/avatar/fa1f479cdd51322fe86215c955a81d296bbf66a1fe625f8a12d87a8ec7faf648?d=mp&s=160"},"body":"On Wed, 2007-06-27 at 19:33 +0400, Roman Kagan wrote:\n\n> OTOH git-fast-import seems to be essentially the public API for the\n> parsecvs kind of tasks.  It may be wiser from the maintenance POV to use\n> that instead of direct libgit calls (unless parsecvs is going to land in\n> the git tree).  I'll try to find the time and take a look at this\n> somewhere next week.\n\nYeah, I didn't quite understand how git-fast-import worked. Looks like\nit aligns with parsecvs's structure fairly well. Let me know if you get\nit working.\n\n-- \nkeith.packard@intel.com\n"}]}