{"thread":{"id":"41107","subject":"How do I get the contents of a directory in fast-import","startedAt":"2016-01-01T15:54:00Z","lastAt":"2016-01-16T01:59:59Z","messageCount":4,"participants":["Stefan Monnier","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"275258","messageId":"jwvio3d7152.fsf-monnier+gmane.comp.version-control.git.user@gnu.org","threadId":"41107","inReplyTo":null,"subject":"How do I get the contents of a directory in fast-import","fromName":"Stefan Monnier","fromEmail":"monnier@iro.umontreal.ca","sentAt":"2016-01-01T15:54:00Z","receivedAt":"2016-01-01T15:54:00Z","isPatch":false,"sender":{"key":"monnier@iro.umontreal.ca","avatar":null},"body":"I have a program which tries to collect info from lots of branches and\ngenerate some table from that data into another branch.\n\nFor performance reasons, I'd like to do that from fast-import, and as\nlong as I know the name of all the files I need to consult, everything\nis fine since I can use the \"ls\" and \"cat-blob\" commands of fast-import\nto get efficiently the data I need.\n\nBut I also need to look at some files whose names I don't know beforehand\n(i.e. all the files in some directories). If I do \"cat-blob\" on those\ndirectories I get some binary \"thing\" which I don't understand.\n\nSo how do I get a directory listing from fast-inmport, i.e.\nlike I can get with \"git cat-file -p\", but without having to fork\na separate git process?\n\n\n Stefan\n"},{"id":"275600","messageId":"jwvegdqb946.fsf-monnier+gmane.comp.version-control.git@gnu.org","threadId":"41107","inReplyTo":"jwvio3d7152.fsf-monnier+gmane.comp.version-control.git.user@gnu.org","subject":"Re: How do I get the contents of a directory in fast-import","fromName":"Stefan Monnier","fromEmail":"monnier@iro.umontreal.ca","sentAt":"2016-01-09T23:56:49Z","receivedAt":"2016-01-09T23:56:49Z","isPatch":false,"sender":{"key":"monnier@iro.umontreal.ca","avatar":null},"body":"Any help would be greatly welcome, including \"sorry, can't do that\".\n\n\n Stefan\n\n\n>>>>> \"Stefan\" == Stefan Monnier writes:\n\n> I have a program which tries to collect info from lots of branches and\n> generate some table from that data into another branch.\n\n> For performance reasons, I'd like to do that from fast-import, and as\n> long as I know the name of all the files I need to consult, everything\n> is fine since I can use the \"ls\" and \"cat-blob\" commands of fast-import\n> to get efficiently the data I need.\n\n> But I also need to look at some files whose names I don't know beforehand\n> (i.e. all the files in some directories). If I do \"cat-blob\" on those\n> directories I get some binary \"thing\" which I don't understand.\n\n> So how do I get a directory listing from fast-inmport, i.e.\n> like I can get with \"git cat-file -p\", but without having to fork\n> a separate git process?\n\n\n> Stefan\n"},{"id":"276213","messageId":"20160115223922.GB32081@sigill.intra.peff.net","threadId":"41107","inReplyTo":"jwvio3d7152.fsf-monnier+gmane.comp.version-control.git.user@gnu.org","subject":"Re: How do I get the contents of a directory in fast-import","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2016-01-15T22:39:22Z","receivedAt":"2016-01-15T22:39:22Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Jan 01, 2016 at 10:54:00AM -0500, Stefan Monnier wrote:\n\n> I have a program which tries to collect info from lots of branches and\n> generate some table from that data into another branch.\n> \n> For performance reasons, I'd like to do that from fast-import, and as\n> long as I know the name of all the files I need to consult, everything\n> is fine since I can use the \"ls\" and \"cat-blob\" commands of fast-import\n> to get efficiently the data I need.\n> \n> But I also need to look at some files whose names I don't know beforehand\n> (i.e. all the files in some directories). If I do \"cat-blob\" on those\n> directories I get some binary \"thing\" which I don't understand.\n> \n> So how do I get a directory listing from fast-inmport, i.e.\n> like I can get with \"git cat-file -p\", but without having to fork\n> a separate git process?\n\nI'm not sure I understand your use case exactly, but is the directory\nlisting you want part of the newly-added objects from fast-import, or\ndoes it already exist in the branches you are collecting from?\n\nIf the latter, I wonder if a separate \"cat-file --batch\" process could\ngive you what you need (it's a separate process, but you can start a\nsingle process and make many queries of it; I assume your desire not to\nadd an extra process is to avoid the overhead).\n\nBut I think it won't pretty-print trees for you; it will give you the\nraw tree data (which I imagine is what you are getting from cat-blob,\ntoo). I'm not sure that's actually documented anywhere (it was part of\nthe original revisions of git, and hasn't changed since). But it is\nbasically:\n\n tree = tree_entry*\n tree_entry = mode SP path NUL sha1\n mode = ascii mode, in octal (e.g., \"100644\")\n path = *\n sha1 = {20}\n SP = ascii space (0x20)\n NUL = 0-byte\n\nSo it is pretty simple to parse.\n\nThere may be a better way to do what you want with fast-import. I'm not\nfamiliar enough with it to say.\n\n-Peff\n"},{"id":"276224","messageId":"jwvd1t270zv.fsf-monnier+gmane.comp.version-control.git@gnu.org","threadId":"41107","inReplyTo":"20160115223922.GB32081@sigill.intra.peff.net","subject":"Re: How do I get the contents of a directory in fast-import","fromName":"Stefan Monnier","fromEmail":"monnier@iro.umontreal.ca","sentAt":"2016-01-16T01:59:59Z","receivedAt":"2016-01-16T01:59:59Z","isPatch":false,"sender":{"key":"monnier@iro.umontreal.ca","avatar":null},"body":">> So how do I get a directory listing from fast-import, i.e.\n>> like I can get with \"git cat-file -p\", but without having to fork\n>> a separate git process?\n> I'm not sure I understand your use case exactly, but is the directory\n> listing you want part of the newly-added objects from fast-import, or\n> does it already exist in the branches you are collecting from?\n\nFor the most important cases, the relevant revision already exists\nbefore fast-import, yes.\n\n> If the latter, I wonder if a separate \"cat-file --batch\" process could\n> give you what you need (it's a separate process, but you can start a\n\nI'm not sure exactly how \"git cat-file --batch\" works internally\n(whether it tries to keep active revisions, like fast-import does), but\nI've indeed used it successfully (tho for files).\n\n> single process and make many queries of it; I assume your desire not to\n> add an extra process is to avoid the overhead).\n\nThe overhead of starting a new process is one part, but another is the\noverhead of re-reading the refs (I can have tens of thousands of\nbranches in my repository), etc..\n\n> But I think it won't pretty-print trees for you; it will give you the\n> raw tree data\n\nIndeed.\n\n> (which I imagine is what you are getting from cat-blob, too).\n\nActually no, \"cat-blob\" gives an error instead:\n\n fatal: Object 2ca1672d50c9dbfe582dc53af3c7ce9891a7a664 is a tree but a blob was expected.\n\n> I'm not sure that's actually documented anywhere (it was part of\n> the original revisions of git, and hasn't changed since). But it is\n> basically:\n\n> tree = tree_entry*\n> tree_entry = mode SP path NUL sha1\n> mode = ascii mode, in octal (e.g., \"100644\")\n> path = *\n> sha1 = {20}\n> SP = ascii space (0x20)\n> NUL = 0-byte\n\nAh, thanks. It'd be great if cat-blob could return this instead of\nsignalling an error.\n\n> So it is pretty simple to parse.\n\nMy program is written in /bin/sh so parsing the above is actually rather\ninconvenient, but it's much better than just getting an error.\n\n\n Stefan\n"}]}