{"thread":{"id":"41280","subject":"fast-import fails in read-only tree","startedAt":"2016-01-28T22:17:36Z","lastAt":"2016-01-30T13:56:48Z","messageCount":6,"participants":["Stefan Monnier","Jeff King","Andreas Schwab"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"277016","messageId":"jwvfuxhz72e.fsf-monnier+gmane.comp.version-control.git@gnu.org","threadId":"41280","inReplyTo":null,"subject":"fast-import fails in read-only tree","fromName":"Stefan Monnier","fromEmail":"monnier@iro.umontreal.ca","sentAt":"2016-01-28T22:17:36Z","receivedAt":"2016-01-28T22:17:36Z","isPatch":false,"sender":{"key":"monnier@iro.umontreal.ca","avatar":null},"body":"I recently discovered that \"git fast-import\" signals an error if used in\na tree to which we do not have write-access, because it tries to create\na \"objects/pack/tmp_pack_XXX\" file even before starting to process\nthe commands.\n\nUsually this is not a problem (we'll create new commits and such, so\nwrite-access is indeed necessary), but in my case I was using\nfast-import only for its \"reading\" operations (in order to combine\nseveral inter-dependent \"cat-file\" operations into a single git\nsession).\n\n\n        Stefan\n"},{"id":"277036","messageId":"20160129060802.GA23106@sigill.intra.peff.net","threadId":"41280","inReplyTo":"jwvfuxhz72e.fsf-monnier+gmane.comp.version-control.git@gnu.org","subject":"Re: fast-import fails in read-only tree","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-01-29T06:08:02Z","receivedAt":"2016-01-29T06:08:02Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jan 28, 2016 at 05:17:36PM -0500, Stefan Monnier wrote:\n\n> I recently discovered that \"git fast-import\" signals an error if used in\n> a tree to which we do not have write-access, because it tries to create\n> a \"objects/pack/tmp_pack_XXX\" file even before starting to process\n> the commands.\n> \n> Usually this is not a problem (we'll create new commits and such, so\n> write-access is indeed necessary), but in my case I was using\n> fast-import only for its \"reading\" operations (in order to combine\n> several inter-dependent \"cat-file\" operations into a single git\n> session).\n\nThe primary goal of fast-import is to write that packfile. It kind of\nsounds like you are using the wrong tool for the job.\n\nCan you elaborate on what you are sending to fast-import (preferably\nwith a concrete example)? There may be a way to accomplish the same\nthing with read-only tools like cat-file.\n\n-Peff\n"},{"id":"277048","messageId":"jwv7fisxyhz.fsf-monnier+gmane.comp.version-control.git@gnu.org","threadId":"41280","inReplyTo":"20160129060802.GA23106@sigill.intra.peff.net","subject":"Re: fast-import fails in read-only tree","fromName":"Stefan Monnier","fromEmail":"monnier@iro.umontreal.ca","sentAt":"2016-01-29T14:28:44Z","receivedAt":"2016-01-29T14:28:44Z","isPatch":false,"sender":{"key":"monnier@iro.umontreal.ca","avatar":null},"body":">> I recently discovered that \"git fast-import\" signals an error if used in\n>> a tree to which we do not have write-access, because it tries to create\n>> a \"objects/pack/tmp_pack_XXX\" file even before starting to process\n>> the commands.\n> The primary goal of fast-import is to write that packfile.  It kind of\n> sounds like you are using the wrong tool for the job.\n\nYes, I realize that.  But in some cases it's the best tool available.\n`fast-import' is very close to being a \"generic access API\" which can be\nused instead of something like libgit.  I think it'd be good to push it\nyet a bit closer.\n\nMy earlier \"cat-blob applied to a tree\" issue is another such case.\n\n> Can you elaborate on what you are sending to fast-import (preferably\n> with a concrete example)?\n\nI'm sending a stream of \"progress <foo>; cat-blob <foo>\", basically.\n\nThe concrete example is in [BuGit](https://gitlab.com/monnier/bugit),\nsee for example https://gitlab.com/monnier/bugit/commit/3678dcb8830a9c79c6f3404d75d63e6dd07bfe4c\n\n> There may be a way to accomplish the same thing with read-only tools\n> like cat-file.\n\nYes, I switched to using \"cat-file --batch\" instead, but it's less\nconvenient (I can't intersperse ad-hoc info in the output, the way I can\nwith \"progress\" in fast-import) and there are cases where the list of\nfiles I need to extract cannot be determined without first looking at\nsome of those extracted files (I currently have been able to avoid\nthis in BuGit, luckily).\n\nIf I could use \"cat-blob\" on directories, there would be even more cases\nwhere I'd want to use fast-import for read-only operations to reduce the\nnumber of Git processes I fork.\n\n\n        Stefan\n"},{"id":"277071","messageId":"20160130051340.GA1677@sigill.intra.peff.net","threadId":"41280","inReplyTo":"jwv7fisxyhz.fsf-monnier+gmane.comp.version-control.git@gnu.org","subject":"Re: fast-import fails in read-only tree","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-01-30T05:13:40Z","receivedAt":"2016-01-30T05:13:40Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 29, 2016 at 09:28:44AM -0500, Stefan Monnier wrote:\n\n> > The primary goal of fast-import is to write that packfile.  It kind of\n> > sounds like you are using the wrong tool for the job.\n> \n> Yes, I realize that.  But in some cases it's the best tool available.\n> `fast-import' is very close to being a \"generic access API\" which can be\n> used instead of something like libgit.  I think it'd be good to push it\n> yet a bit closer.\n\nI'm not sure I agree. Git tries to make its innards available via\nflexible plumbing commands. If we're not succeeding, I think that should\nbe fixed, rather than trying to shoe-horn an unrelated command to do the\njob, even if it would be less code.\n\n> > Can you elaborate on what you are sending to fast-import (preferably\n> > with a concrete example)?\n> \n> I'm sending a stream of \"progress <foo>; cat-blob <foo>\", basically.\n> \n> The concrete example is in [BuGit](https://gitlab.com/monnier/bugit),\n> see for example https://gitlab.com/monnier/bugit/commit/3678dcb8830a9c79c6f3404d75d63e6dd07bfe4c\n\nYou can use custom cat-file formatting to output your \"name\" strings as\npart of the same field. IOW, something like:\n\n  git cat-file -p refs/heads/bugit-master:numbers |\n  awk '{print $3 \" \" $4 }' |\n  git cat-file --batch=\"%(rest)\" |\n  while read number; do\n    read id; # assuming blob contents are single-line\n    read _junk; # assumes blob ended in its own newline\n    $fun \"$id\" \"$number\"\n  done\n\nThat's from a fairly cursory reading of that bugit patch, though, so I\nmight be missing some requirement.\n\n> Yes, I switched to using \"cat-file --batch\" instead, but it's less\n> convenient (I can't intersperse ad-hoc info in the output, the way I can\n> with \"progress\" in fast-import) and there are cases where the list of\n> files I need to extract cannot be determined without first looking at\n> some of those extracted files (I currently have been able to avoid\n> this in BuGit, luckily).\n\nI think the example above should handle the \"intersperse\" thing.\n\nIf you're really going to do a lot of interactive back-and-forth access\nof objects, though, I think you want to set up pipes to cat-file. It's a\nlittle tedious to allocate fifos, but something like:\n\n  mkfifo in out\n  (exec git cat-file --batch <in >out) &\n  exec 8>in\n  exec 9<out\n  echo $sha >&8\n  read mode type size <&9\n  read content ;# or read $size, or read until newline\n  echo $content >&8 ;# imagine content is another sha to look up\n  ...read from &9, etc..\n\nThe fifos and numbered descriptors are annoying, but that's shell for\nyou. I suspect using \"fast-import\" wouldn't be much different.\n\nOne feature I do think would be useful (and almost implemented when I\nadded --batch-check=<format>) is a formatter for the object content,\nwith a pretty modifier. I.e., it would be nice to do:\n\n  echo $some_tree |\n  git cat-file --batch-check=\"%(objectsize:pretty) %(contents:pretty)\"\n\nto work as the rough equivalent of \"git cat-file -p\" (but here you could\nfeed multiple trees and get multiple answers).\n\n-Peff\n"},{"id":"277079","messageId":"m2oac31lp2.fsf@linux-m68k.org","threadId":"41280","inReplyTo":"20160130051340.GA1677@sigill.intra.peff.net","subject":"Re: fast-import fails in read-only tree","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2016-01-30T09:05:45Z","receivedAt":"2016-01-30T09:05:45Z","isPatch":false,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> If you're really going to do a lot of interactive back-and-forth access\n> of objects, though, I think you want to set up pipes to cat-file. It's a\n> little tedious to allocate fifos, but something like:\n\nWith bash's coproc it's a bit less tedious:\n\n>   mkfifo in out\n>   (exec git cat-file --batch <in >out) &\n>   exec 8>in\n>   exec 9<out\n>   echo $sha >&8\n>   read mode type size <&9\n\n    coproc CAT_FILE git cat-file --batch\n    echo $sha >&${CAT_FILE[1]}\n    read mode type size <&${CAT_FILE[0]}\n\nAndreas.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 58CA 54C7 6D53 942B 1756  01D3 44D5 214B 8276 4ED5\n\"And now for something completely different.\"\n"},{"id":"277083","messageId":"jwvtwlv18jw.fsf-monnier+gmane.comp.version-control.git@gnu.org","threadId":"41280","inReplyTo":"20160130051340.GA1677@sigill.intra.peff.net","subject":"Re: fast-import fails in read-only tree","fromName":"Stefan Monnier","fromEmail":"monnier@iro.umontreal.ca","sentAt":"2016-01-30T13:56:48Z","receivedAt":"2016-01-30T13:56:48Z","isPatch":false,"sender":{"key":"monnier@iro.umontreal.ca","avatar":null},"body":"> You can use custom cat-file formatting to output your \"name\" strings as\n> part of the same field. IOW, something like:\n[...]\n> If you're really going to do a lot of interactive back-and-forth access\n> of objects, though, I think you want to set up pipes to cat-file.\n\nOMG, I didn't realize that cat-file doesn't buffer its output so it can\nbe read&write to/from the same process.  And the \"%(rest)\" thingy takes\ncare of the rest of my needs, indeed.\n\nThanks!\n\n> It's a little tedious to allocate fifos, but something like:\n\nThat's not a problem.\n\n> One feature I do think would be useful (and almost implemented when I\n> added --batch-check=<format>) is a formatter for the object content,\n> with a pretty modifier. I.e., it would be nice to do:\n>\n>   echo $some_tree |\n>   git cat-file --batch-check=\"%(objectsize:pretty) %(contents:pretty)\"\n>\n> to work as the rough equivalent of \"git cat-file -p\" (but here you could\n> feed multiple trees and get multiple answers).\n\nYes, that would be a good improvement,\n\n\n        Stefan\n"}]}