{"thread":{"id":"376","subject":"[PATCH] GIT: Create tar archives of tree on the fly, take two","startedAt":"2005-04-28T17:40:38Z","lastAt":"2005-04-28T18:51:46Z","messageCount":3,"participants":["Rene Scharfe","Linus Torvalds"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"2060","messageId":"20050428174038.GA24352@lsrfire.ath.cx","threadId":"376","inReplyTo":null,"subject":"[PATCH] GIT: Create tar archives of tree on the fly, take two","fromName":"Rene Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2005-04-28T17:40:38Z","receivedAt":"2005-04-28T17:40:38Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Hi Linus,\n\nhere's an improved version of tar-tree, a streaming archive creator for\nGIT.  The major added feature is blocking; all write(2) calls now have a\nsize of 10240, just as GNU tar (and tape drives) likes them.  The\nbuffering overhead does not seem to degrade performance because most\nfiles in the repositories I tested this with are smaller than 10KB, so\nwe need fewer system calls.\n\nFile names are still restricted to 500 bytes and the archive format\ncurrently only allows for files up to 8GB.  Both restrictions can be\nlifted if need be with more pax extended headers.\n\nThe archive format used is the pax interchange format, i.e. POSIX tar\nformat.  It can be read by (and created with) GNU tar.  If I read the\nspecs correctly tar-tree should now be standards compliant (modulo\nbugs).\n\nBecause it streams the archive (think ls-tree merged with cat-file),\ntar-tree doesn't need to create any temporary files.  That makes it\nquite fast.\n\nIt accepts tree IDs and commit IDs as first parameter.  In the latter\ncase tar-tree tries to get the commit date out of the committer line.\nElse all files in the archive are time-stamped with the current time.\n\nAn optional second parameter is used as a path prefix for all files in\nthe archive.  Example:\n\n   $ tar-tree a2755a80f40e5794ddc20e00f781af9d6320fafb \\\n   linux-2.6.12-rc3 | bzip9 -9 > linux-2.6.12-rc3.tar.bz2\n\nI think it fits nicely into the core toolset of git.  What do you think?\n\nThanks,\nRene\n\n\nIndex: Makefile\n===================================================================\n--- a5d7e40207ee2bd713a15d5bc839168139055f5e/Makefile  (mode:100644 sha1:378d5ee9a24a9fb7690bfdf79fd3fde2b9563fad)\n+++ 0fa216c80d3ecf45d3de1552c03d95a21f8c2a11/Makefile  (mode:100644 sha1:0b330dc3fd05253b8fd20ca4f9e98e411074868f)\n@@ -18,7 +18,7 @@\n \tcat-file fsck-cache checkout-cache diff-tree rev-tree show-files \\\n \tcheck-files ls-tree merge-base merge-cache unpack-file git-export \\\n \tdiff-cache convert-cache http-pull rpush rpull rev-list git-mktag \\\n-\tdiff-tree-helper\n+\tdiff-tree-helper tar-tree\n \n all: $(PROG)\n \nIndex: tar-tree.c\n===================================================================\n--- /dev/null  (tree:a5d7e40207ee2bd713a15d5bc839168139055f5e)\n+++ 0fa216c80d3ecf45d3de1552c03d95a21f8c2a11/tar-tree.c  (mode:100644 sha1:d078cb5a8d4ab5bc2c774b889b602d179bde39b1)\n@@ -0,0 +1,363 @@\n+#include <time.h>\n+#include \"cache.h\"\n+\n+#define RECORDSIZE\t(512)\n+#define BLOCKSIZE\t(RECORDSIZE * 20)\n+\n+static const char *tar_tree_usage = \"tar-tree <key> [basedir]\";\n+\n+static char block[BLOCKSIZE];\n+static unsigned long offset;\n+\n+static const char *basedir;\n+static time_t archive_time;\n+\n+struct path_prefix {\n+\tstruct path_prefix *prev;\n+\tconst char *name;\n+};\n+\n+/* tries hard to write, either succeeds or dies in the attempt */\n+static void reliable_write(void *buf, unsigned long size)\n+{\n+\twhile (size > 0) {\n+\t\tlong ret = write(1, buf, size);\n+\t\tif (ret < 0) {\n+\t\t\tif (errno == EAGAIN)\n+\t\t\t\tcontinue;\n+\t\t\tif (errno == EPIPE)\n+\t\t\t\texit(0);\n+\t\t\tdie(\"tar-tree: %s\", strerror(errno));\n+\t\t} else if (!ret) {\n+\t\t\tdie(\"tar-tree: disk full?\");\n+\t\t}\n+\t\tsize -= ret;\n+\t\tbuf += ret;\n+\t}\n+}\n+\n+/* writes out the whole block, but only if it is full */\n+static void write_if_needed(void)\n+{\n+\tif (offset == BLOCKSIZE) {\n+\t\treliable_write(block, BLOCKSIZE);\n+\t\toffset = 0;\n+\t}\n+}\n+\n+/*\n+ * The end of tar archives is marked by 1024 nul bytes and after that\n+ * follows the rest of the block (if any).\n+ */\n+static void write_trailer(void)\n+{\n+\tmemset(block + offset, 0, RECORDSIZE);\n+\toffset += RECORDSIZE;\n+\twrite_if_needed();\n+\tmemset(block + offset, 0, RECORDSIZE);\n+\toffset += RECORDSIZE;\n+\twrite_if_needed();\n+\tif (offset) {\n+\t\tmemset(block + offset, 0, BLOCKSIZE - offset);\n+\t\treliable_write(block, BLOCKSIZE);\n+\t\toffset = 0;\n+\t}\n+}\n+\n+/*\n+ * queues up writes, so that all our write(2) calls write exactly one\n+ * full block; pads writes to RECORDSIZE\n+ */\n+static void write_blocked(void *buf, unsigned long size)\n+{\n+\tunsigned long tail;\n+\n+\tif (offset) {\n+\t\tunsigned long chunk = BLOCKSIZE - offset;\n+\t\tif (size < chunk)\n+\t\t\tchunk = size;\n+\t\tmemcpy(block + offset, buf, chunk);\n+\t\tsize -= chunk;\n+\t\toffset += chunk;\n+\t\tbuf += chunk;\n+\t\twrite_if_needed();\n+\t}\n+\twhile (size >= BLOCKSIZE) {\n+\t\treliable_write(buf, BLOCKSIZE);\n+\t\tsize -= BLOCKSIZE;\n+\t\tbuf += BLOCKSIZE;\n+\t}\n+\tif (size) {\n+\t\tmemcpy(block + offset, buf, size);\n+\t\tbuf += size;\n+\t\toffset += size;\n+\t}\n+\ttail = offset % RECORDSIZE;\n+\tif (tail)  {\n+\t\tmemset(block + offset, 0, RECORDSIZE - tail);\n+\t\toffset += RECORDSIZE - tail;\n+\t}\n+\twrite_if_needed();\n+}\n+\n+static void append_string(char **p, const char *s)\n+{\n+\tunsigned int len = strlen(s);\n+\tmemcpy(*p, s, len);\n+\t*p += len;\n+}\n+\n+static void append_char(char **p, char c)\n+{\n+\t**p = c;\n+\t*p += 1;\n+}\n+\n+static void append_long(char **p, long n)\n+{\n+\tint len = sprintf(*p, \"%ld\", n);\n+\t*p += len;\n+}\n+\n+static void append_path_prefix(char **buffer, struct path_prefix *prefix)\n+{\n+\tif (!prefix)\n+\t\treturn;\n+\tappend_path_prefix(buffer, prefix->prev);\n+\tappend_string(buffer, prefix->name);\n+\tappend_char(buffer, '/');\n+}\n+\n+static unsigned int path_prefix_len(struct path_prefix *prefix)\n+{\n+\tif (!prefix)\n+\t\treturn 0;\n+\treturn path_prefix_len(prefix->prev) + strlen(prefix->name) + 1;\n+}\n+\n+static void append_path(char **p, int is_dir, const char *basepath,\n+                        struct path_prefix *prefix, const char *path)\n+{\n+\tif (basepath) {\n+\t\tappend_string(p, basepath);\n+\t\tappend_char(p, '/');\n+\t}\n+\tappend_path_prefix(p, prefix);\n+\tappend_string(p, path);\n+\tif (is_dir)\n+\t\tappend_char(p, '/');\n+}\n+\n+static unsigned int path_len(int is_dir, const char *basepath,\n+                             struct path_prefix *prefix, const char *path)\n+{\n+\tunsigned int len = 0;\n+\tif (basepath)\n+\t\tlen += strlen(basepath) + 1;\n+\tlen += path_prefix_len(prefix) + strlen(path);\n+\tif (is_dir)\n+\t\tlen++;\n+\treturn len;\n+}\n+\n+static void write_header(const char *, const char *, struct path_prefix *,\n+                         const char *, unsigned int, unsigned long);\n+\n+/* stores a pax extended header directly in the block buffer */\n+static void write_extended_header(const char *headerfilename, int is_dir,\n+                                  const char *basepath,\n+                                  struct path_prefix *prefix,\n+                                  const char *path, unsigned int namelen)\n+{\n+\tchar *records, *p;\n+\tunsigned int size = 1 + 6 + namelen + 1;\n+\tif (size > 9)\n+\t\tsize++;\n+\tif (size > 99)\n+\t\tsize++;\n+\tif (size > RECORDSIZE)\n+\t\tdie(\"tar-tree: extended header too big, wtf?\");\n+\twrite_header(NULL, NULL, NULL, headerfilename, 0100600, size);\n+\n+\trecords = block + offset;\n+\tmemset(records, 0, RECORDSIZE);\n+\toffset += RECORDSIZE;\n+\tp = records;\n+\tappend_long(&p, size);\n+\tappend_string(&p, \" path=\");\n+\tappend_path(&p, is_dir, basepath, prefix, path);\n+\tappend_char(&p, '\\n');\n+\twrite_if_needed();\n+}\n+\n+/* stores a ustar header directly in the block buffer */\n+static void write_header(const char *sha1, const char *basepath,\n+                         struct path_prefix *prefix, const char *path,\n+                         unsigned int mode, unsigned long size)\n+{\n+\tunsigned int namelen; \n+\tchar *p, *header = NULL;\n+\tunsigned int checksum = 0;\n+\tint i;\n+\n+\tnamelen = path_len(S_ISDIR(mode), basepath, prefix, path);\n+\tif (namelen > 500) {\n+\t\tdie(\"tar-tree: name too log of object %s\\n\", sha1_to_hex(sha1));\n+\t} else if (namelen > 100) {\n+\t\tchar *sha1_hex = sha1_to_hex(sha1);\n+\t\tchar headerfilename[51];\n+\t\tsprintf(headerfilename, \"%s.paxheader\", sha1_hex);\n+\t\t/* the extended header must be written before the normal one */\n+\t\twrite_extended_header(headerfilename, S_ISDIR(mode), basepath,\n+\t\t\t\t      prefix, path, namelen);\n+\n+\t\theader = block + offset;\n+\t\tmemset(header, 0, RECORDSIZE);\n+\t\toffset += RECORDSIZE;\n+\t\tsprintf(header, \"%s.data\", sha1_hex);\n+\t} else {\n+\t\theader = block + offset;\n+\t\tmemset(header, 0, RECORDSIZE);\n+\t\toffset += RECORDSIZE;\n+\t\tp = header;\n+\t\tappend_path(&p, S_ISDIR(mode), basepath, prefix, path);\n+\t}\n+\n+\tif (S_ISDIR(mode))\n+\t\tmode |= 0755;\t/* GIT doesn't store permissions of dirs */\n+\tsprintf(&header[100], \"%07o\", mode & 07777);\n+\n+\t/* XXX: should we provide more meaningful info here? */\n+\tsprintf(&header[108], \"%07o\", 0);\t/* uid */\n+\tsprintf(&header[116], \"%07o\", 0);\t/* gid */\n+\tstrncpy(&header[265], \"git\", 31);\t/* uname */\n+\tstrncpy(&header[297], \"git\", 31);\t/* gname */\n+\n+\tsprintf(&header[124], \"%011lo\", S_ISDIR(mode) ? 0 : size);\n+\tsprintf(&header[136], \"%011lo\", archive_time);\n+\n+\t/* typeflag */\n+\tif (!sha1)\n+\t\theader[156] = 'x';\t/* extended header */\n+\telse\n+\t\theader[156] = S_ISDIR(mode) ? '5' : '0';\n+\n+\tmemcpy(&header[257], \"ustar\", 6);\n+\tmemcpy(&header[263], \"00\", 2);\n+\n+\tprintf(&header[329], \"%07o\", 0);\t/* devmajor */\n+\tprintf(&header[337], \"%07o\", 0);\t/* devminor */\n+\n+\tmemset(&header[148], ' ', 8);\n+\tfor (i = 0; i < RECORDSIZE; i++)\n+\t\tchecksum += header[i];\n+\tsprintf(&header[148], \"%07o\", checksum & 0x1fffff);\n+\n+\twrite_if_needed();\n+}\n+\n+static void traverse_tree(void *buffer, unsigned long size,\n+\t\t\t  struct path_prefix *prefix)\n+{\n+\tstruct path_prefix this_prefix;\n+\tthis_prefix.prev = prefix;\n+\n+\twhile (size) {\n+\t\tint namelen = strlen(buffer)+1;\n+\t\tvoid *eltbuf;\n+\t\tchar elttype[20];\n+\t\tunsigned long eltsize;\n+\t\tunsigned char *sha1 = buffer + namelen;\n+\t\tchar *path = strchr(buffer, ' ') + 1;\n+\t\tunsigned int mode;\n+\n+\t\tif (size < namelen + 20 || sscanf(buffer, \"%o\", &mode) != 1)\n+\t\t\tdie(\"corrupt 'tree' file\");\n+\t\tbuffer = sha1 + 20;\n+\t\tsize -= namelen + 20;\n+\n+\t\teltbuf = read_sha1_file(sha1, elttype, &eltsize);\n+\t\tif (!eltbuf)\n+\t\t\tdie(\"cannot read %s\", sha1_to_hex(sha1));\n+\t\twrite_header(sha1, basedir, prefix, path, mode, eltsize);\n+\t\tif (!strcmp(elttype, \"tree\")) {\n+\t\t\tthis_prefix.name = path;\n+\t\t\ttraverse_tree(eltbuf, eltsize, &this_prefix);\n+\t\t} else if (!strcmp(elttype, \"blob\")) {\n+\t\t\twrite_blocked(eltbuf, eltsize);\n+\t\t}\n+\t\tfree(eltbuf);\n+\t}\n+}\n+\n+/* get commit time from committer line of commit object */\n+time_t commit_time(const unsigned char *sha1)\n+{\n+        char type[20];\n+        void *buffer;\n+        unsigned long size;\n+\ttime_t result = 0;\n+\n+        buffer = read_sha1_file(sha1, type, &size);\n+\tif (buffer) {\n+\t\tchar *p = buffer;\n+\t\twhile (size > 0) {\n+\t\t\tchar *endp = memchr(p, '\\n', size);\n+\t\t\tif (!endp || endp == p)\n+\t\t\t\tbreak;\n+\t\t\t*endp = '\\0';\n+\t\t\tif (endp - p > 10 && !memcmp(p, \"committer \", 10)) {\n+\t\t\t\tchar *nump = strrchr(p, '>');\n+\t\t\t\tif (!nump)\n+\t\t\t\t\tbreak;\n+\t\t\t\tnump++;\n+\t\t\t\tresult = strtoul(nump, &endp, 10);\n+\t\t\t\tif (*endp != ' ')\n+\t\t\t\t\tresult = 0;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tsize -= endp - p - 1;\n+\t\t\tp = endp + 1;\n+\t\t}\n+\t\tfree(buffer);\n+\t}\n+\treturn result;\n+}\n+\n+int main(int argc, char **argv)\n+{\n+\tunsigned char sha1[20];\n+\tvoid *buffer;\n+\tunsigned long size;\n+\tunsigned char tree_sha1[20];\n+\n+\tswitch (argc) {\n+\tcase 3:\n+\t\tbasedir = argv[2];\n+\t\t/* FALLTHROUGH */\n+\tcase 2:\n+\t\tif (get_sha1_hex(argv[1], sha1) < 0)\n+\t\t\tusage(tar_tree_usage);\n+\t\tbreak;\n+\tdefault:\n+\t\tusage(tar_tree_usage);\n+\t}\n+\n+\tsha1_file_directory = getenv(DB_ENVIRONMENT);\n+\tif (!sha1_file_directory)\n+\t\tsha1_file_directory = DEFAULT_DB_ENVIRONMENT;\n+\n+\tbuffer = read_tree_with_tree_or_commit_sha1(sha1, &size, tree_sha1);\n+\tif (!buffer)\n+\t\tdie(\"unable to read sha1 file\");\n+\tif (memcmp(sha1, tree_sha1, 20))\t/* is sha1 a commit object? */\n+\t\tarchive_time = commit_time(sha1);\n+\tif (!archive_time)\n+\t\tarchive_time = time(NULL);\n+\tif (basedir)\n+\t\twrite_header(\"0\", NULL, NULL, basedir, 040755, 0);\n+\ttraverse_tree(buffer, size, NULL);\n+\tfree(buffer);\n+\twrite_trailer();\n+\treturn 0;\n+}\n"},{"id":"2061","messageId":"Pine.LNX.4.58.0504281121070.18901@ppc970.osdl.org","threadId":"376","inReplyTo":"20050428174038.GA24352@lsrfire.ath.cx","subject":"Re: [PATCH] GIT: Create tar archives of tree on the fly, take two","fromName":"Linus Torvalds","fromEmail":"torvalds@osdl.org","sentAt":"2005-04-28T18:21:29Z","receivedAt":"2005-04-28T18:21:29Z","isPatch":true,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Thu, 28 Apr 2005, Rene Scharfe wrote:\n> \n> I think it fits nicely into the core toolset of git.  What do you think?\n\nYes. Mind sending in a sign-off too, and I'll apply it.\n\n\t\tLinus\n"},{"id":"2063","messageId":"20050428185146.GA26019@lsrfire.ath.cx","threadId":"376","inReplyTo":"Pine.LNX.4.58.0504281121070.18901@ppc970.osdl.org","subject":"Re: [PATCH] GIT: Create tar archives of tree on the fly, take two","fromName":"Rene Scharfe","fromEmail":"rene.scharfe@lsrfire.ath.cx","sentAt":"2005-04-28T18:51:46Z","receivedAt":"2005-04-28T18:51:46Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"On Thu, Apr 28, 2005 at 11:21:29AM -0700, Linus Torvalds wrote:\n> \n> Yes. Mind sending in a sign-off too, and I'll apply it.\n\nSure thing.  Hope it's the only thing I stupidly forgot. :)\n\nSigned-off-by: Rene Scharfe <rene.scharfe@lsrfire.ath.cx>\n\nIndex: Makefile\n===================================================================\n--- a5d7e40207ee2bd713a15d5bc839168139055f5e/Makefile  (mode:100644 sha1:378d5ee9a24a9fb7690bfdf79fd3fde2b9563fad)\n+++ 0fa216c80d3ecf45d3de1552c03d95a21f8c2a11/Makefile  (mode:100644 sha1:0b330dc3fd05253b8fd20ca4f9e98e411074868f)\n@@ -18,7 +18,7 @@\n \tcat-file fsck-cache checkout-cache diff-tree rev-tree show-files \\\n \tcheck-files ls-tree merge-base merge-cache unpack-file git-export \\\n \tdiff-cache convert-cache http-pull rpush rpull rev-list git-mktag \\\n-\tdiff-tree-helper\n+\tdiff-tree-helper tar-tree\n \n all: $(PROG)\n \nIndex: tar-tree.c\n===================================================================\n--- /dev/null  (tree:a5d7e40207ee2bd713a15d5bc839168139055f5e)\n+++ 0fa216c80d3ecf45d3de1552c03d95a21f8c2a11/tar-tree.c  (mode:100644 sha1:d078cb5a8d4ab5bc2c774b889b602d179bde39b1)\n@@ -0,0 +1,363 @@\n+#include <time.h>\n+#include \"cache.h\"\n+\n+#define RECORDSIZE\t(512)\n+#define BLOCKSIZE\t(RECORDSIZE * 20)\n+\n+static const char *tar_tree_usage = \"tar-tree <key> [basedir]\";\n+\n+static char block[BLOCKSIZE];\n+static unsigned long offset;\n+\n+static const char *basedir;\n+static time_t archive_time;\n+\n+struct path_prefix {\n+\tstruct path_prefix *prev;\n+\tconst char *name;\n+};\n+\n+/* tries hard to write, either succeeds or dies in the attempt */\n+static void reliable_write(void *buf, unsigned long size)\n+{\n+\twhile (size > 0) {\n+\t\tlong ret = write(1, buf, size);\n+\t\tif (ret < 0) {\n+\t\t\tif (errno == EAGAIN)\n+\t\t\t\tcontinue;\n+\t\t\tif (errno == EPIPE)\n+\t\t\t\texit(0);\n+\t\t\tdie(\"tar-tree: %s\", strerror(errno));\n+\t\t} else if (!ret) {\n+\t\t\tdie(\"tar-tree: disk full?\");\n+\t\t}\n+\t\tsize -= ret;\n+\t\tbuf += ret;\n+\t}\n+}\n+\n+/* writes out the whole block, but only if it is full */\n+static void write_if_needed(void)\n+{\n+\tif (offset == BLOCKSIZE) {\n+\t\treliable_write(block, BLOCKSIZE);\n+\t\toffset = 0;\n+\t}\n+}\n+\n+/*\n+ * The end of tar archives is marked by 1024 nul bytes and after that\n+ * follows the rest of the block (if any).\n+ */\n+static void write_trailer(void)\n+{\n+\tmemset(block + offset, 0, RECORDSIZE);\n+\toffset += RECORDSIZE;\n+\twrite_if_needed();\n+\tmemset(block + offset, 0, RECORDSIZE);\n+\toffset += RECORDSIZE;\n+\twrite_if_needed();\n+\tif (offset) {\n+\t\tmemset(block + offset, 0, BLOCKSIZE - offset);\n+\t\treliable_write(block, BLOCKSIZE);\n+\t\toffset = 0;\n+\t}\n+}\n+\n+/*\n+ * queues up writes, so that all our write(2) calls write exactly one\n+ * full block; pads writes to RECORDSIZE\n+ */\n+static void write_blocked(void *buf, unsigned long size)\n+{\n+\tunsigned long tail;\n+\n+\tif (offset) {\n+\t\tunsigned long chunk = BLOCKSIZE - offset;\n+\t\tif (size < chunk)\n+\t\t\tchunk = size;\n+\t\tmemcpy(block + offset, buf, chunk);\n+\t\tsize -= chunk;\n+\t\toffset += chunk;\n+\t\tbuf += chunk;\n+\t\twrite_if_needed();\n+\t}\n+\twhile (size >= BLOCKSIZE) {\n+\t\treliable_write(buf, BLOCKSIZE);\n+\t\tsize -= BLOCKSIZE;\n+\t\tbuf += BLOCKSIZE;\n+\t}\n+\tif (size) {\n+\t\tmemcpy(block + offset, buf, size);\n+\t\tbuf += size;\n+\t\toffset += size;\n+\t}\n+\ttail = offset % RECORDSIZE;\n+\tif (tail)  {\n+\t\tmemset(block + offset, 0, RECORDSIZE - tail);\n+\t\toffset += RECORDSIZE - tail;\n+\t}\n+\twrite_if_needed();\n+}\n+\n+static void append_string(char **p, const char *s)\n+{\n+\tunsigned int len = strlen(s);\n+\tmemcpy(*p, s, len);\n+\t*p += len;\n+}\n+\n+static void append_char(char **p, char c)\n+{\n+\t**p = c;\n+\t*p += 1;\n+}\n+\n+static void append_long(char **p, long n)\n+{\n+\tint len = sprintf(*p, \"%ld\", n);\n+\t*p += len;\n+}\n+\n+static void append_path_prefix(char **buffer, struct path_prefix *prefix)\n+{\n+\tif (!prefix)\n+\t\treturn;\n+\tappend_path_prefix(buffer, prefix->prev);\n+\tappend_string(buffer, prefix->name);\n+\tappend_char(buffer, '/');\n+}\n+\n+static unsigned int path_prefix_len(struct path_prefix *prefix)\n+{\n+\tif (!prefix)\n+\t\treturn 0;\n+\treturn path_prefix_len(prefix->prev) + strlen(prefix->name) + 1;\n+}\n+\n+static void append_path(char **p, int is_dir, const char *basepath,\n+                        struct path_prefix *prefix, const char *path)\n+{\n+\tif (basepath) {\n+\t\tappend_string(p, basepath);\n+\t\tappend_char(p, '/');\n+\t}\n+\tappend_path_prefix(p, prefix);\n+\tappend_string(p, path);\n+\tif (is_dir)\n+\t\tappend_char(p, '/');\n+}\n+\n+static unsigned int path_len(int is_dir, const char *basepath,\n+                             struct path_prefix *prefix, const char *path)\n+{\n+\tunsigned int len = 0;\n+\tif (basepath)\n+\t\tlen += strlen(basepath) + 1;\n+\tlen += path_prefix_len(prefix) + strlen(path);\n+\tif (is_dir)\n+\t\tlen++;\n+\treturn len;\n+}\n+\n+static void write_header(const char *, const char *, struct path_prefix *,\n+                         const char *, unsigned int, unsigned long);\n+\n+/* stores a pax extended header directly in the block buffer */\n+static void write_extended_header(const char *headerfilename, int is_dir,\n+                                  const char *basepath,\n+                                  struct path_prefix *prefix,\n+                                  const char *path, unsigned int namelen)\n+{\n+\tchar *records, *p;\n+\tunsigned int size = 1 + 6 + namelen + 1;\n+\tif (size > 9)\n+\t\tsize++;\n+\tif (size > 99)\n+\t\tsize++;\n+\tif (size > RECORDSIZE)\n+\t\tdie(\"tar-tree: extended header too big, wtf?\");\n+\twrite_header(NULL, NULL, NULL, headerfilename, 0100600, size);\n+\n+\trecords = block + offset;\n+\tmemset(records, 0, RECORDSIZE);\n+\toffset += RECORDSIZE;\n+\tp = records;\n+\tappend_long(&p, size);\n+\tappend_string(&p, \" path=\");\n+\tappend_path(&p, is_dir, basepath, prefix, path);\n+\tappend_char(&p, '\\n');\n+\twrite_if_needed();\n+}\n+\n+/* stores a ustar header directly in the block buffer */\n+static void write_header(const char *sha1, const char *basepath,\n+                         struct path_prefix *prefix, const char *path,\n+                         unsigned int mode, unsigned long size)\n+{\n+\tunsigned int namelen; \n+\tchar *p, *header = NULL;\n+\tunsigned int checksum = 0;\n+\tint i;\n+\n+\tnamelen = path_len(S_ISDIR(mode), basepath, prefix, path);\n+\tif (namelen > 500) {\n+\t\tdie(\"tar-tree: name too log of object %s\\n\", sha1_to_hex(sha1));\n+\t} else if (namelen > 100) {\n+\t\tchar *sha1_hex = sha1_to_hex(sha1);\n+\t\tchar headerfilename[51];\n+\t\tsprintf(headerfilename, \"%s.paxheader\", sha1_hex);\n+\t\t/* the extended header must be written before the normal one */\n+\t\twrite_extended_header(headerfilename, S_ISDIR(mode), basepath,\n+\t\t\t\t      prefix, path, namelen);\n+\n+\t\theader = block + offset;\n+\t\tmemset(header, 0, RECORDSIZE);\n+\t\toffset += RECORDSIZE;\n+\t\tsprintf(header, \"%s.data\", sha1_hex);\n+\t} else {\n+\t\theader = block + offset;\n+\t\tmemset(header, 0, RECORDSIZE);\n+\t\toffset += RECORDSIZE;\n+\t\tp = header;\n+\t\tappend_path(&p, S_ISDIR(mode), basepath, prefix, path);\n+\t}\n+\n+\tif (S_ISDIR(mode))\n+\t\tmode |= 0755;\t/* GIT doesn't store permissions of dirs */\n+\tsprintf(&header[100], \"%07o\", mode & 07777);\n+\n+\t/* XXX: should we provide more meaningful info here? */\n+\tsprintf(&header[108], \"%07o\", 0);\t/* uid */\n+\tsprintf(&header[116], \"%07o\", 0);\t/* gid */\n+\tstrncpy(&header[265], \"git\", 31);\t/* uname */\n+\tstrncpy(&header[297], \"git\", 31);\t/* gname */\n+\n+\tsprintf(&header[124], \"%011lo\", S_ISDIR(mode) ? 0 : size);\n+\tsprintf(&header[136], \"%011lo\", archive_time);\n+\n+\t/* typeflag */\n+\tif (!sha1)\n+\t\theader[156] = 'x';\t/* extended header */\n+\telse\n+\t\theader[156] = S_ISDIR(mode) ? '5' : '0';\n+\n+\tmemcpy(&header[257], \"ustar\", 6);\n+\tmemcpy(&header[263], \"00\", 2);\n+\n+\tprintf(&header[329], \"%07o\", 0);\t/* devmajor */\n+\tprintf(&header[337], \"%07o\", 0);\t/* devminor */\n+\n+\tmemset(&header[148], ' ', 8);\n+\tfor (i = 0; i < RECORDSIZE; i++)\n+\t\tchecksum += header[i];\n+\tsprintf(&header[148], \"%07o\", checksum & 0x1fffff);\n+\n+\twrite_if_needed();\n+}\n+\n+static void traverse_tree(void *buffer, unsigned long size,\n+\t\t\t  struct path_prefix *prefix)\n+{\n+\tstruct path_prefix this_prefix;\n+\tthis_prefix.prev = prefix;\n+\n+\twhile (size) {\n+\t\tint namelen = strlen(buffer)+1;\n+\t\tvoid *eltbuf;\n+\t\tchar elttype[20];\n+\t\tunsigned long eltsize;\n+\t\tunsigned char *sha1 = buffer + namelen;\n+\t\tchar *path = strchr(buffer, ' ') + 1;\n+\t\tunsigned int mode;\n+\n+\t\tif (size < namelen + 20 || sscanf(buffer, \"%o\", &mode) != 1)\n+\t\t\tdie(\"corrupt 'tree' file\");\n+\t\tbuffer = sha1 + 20;\n+\t\tsize -= namelen + 20;\n+\n+\t\teltbuf = read_sha1_file(sha1, elttype, &eltsize);\n+\t\tif (!eltbuf)\n+\t\t\tdie(\"cannot read %s\", sha1_to_hex(sha1));\n+\t\twrite_header(sha1, basedir, prefix, path, mode, eltsize);\n+\t\tif (!strcmp(elttype, \"tree\")) {\n+\t\t\tthis_prefix.name = path;\n+\t\t\ttraverse_tree(eltbuf, eltsize, &this_prefix);\n+\t\t} else if (!strcmp(elttype, \"blob\")) {\n+\t\t\twrite_blocked(eltbuf, eltsize);\n+\t\t}\n+\t\tfree(eltbuf);\n+\t}\n+}\n+\n+/* get commit time from committer line of commit object */\n+time_t commit_time(const unsigned char *sha1)\n+{\n+        char type[20];\n+        void *buffer;\n+        unsigned long size;\n+\ttime_t result = 0;\n+\n+        buffer = read_sha1_file(sha1, type, &size);\n+\tif (buffer) {\n+\t\tchar *p = buffer;\n+\t\twhile (size > 0) {\n+\t\t\tchar *endp = memchr(p, '\\n', size);\n+\t\t\tif (!endp || endp == p)\n+\t\t\t\tbreak;\n+\t\t\t*endp = '\\0';\n+\t\t\tif (endp - p > 10 && !memcmp(p, \"committer \", 10)) {\n+\t\t\t\tchar *nump = strrchr(p, '>');\n+\t\t\t\tif (!nump)\n+\t\t\t\t\tbreak;\n+\t\t\t\tnump++;\n+\t\t\t\tresult = strtoul(nump, &endp, 10);\n+\t\t\t\tif (*endp != ' ')\n+\t\t\t\t\tresult = 0;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tsize -= endp - p - 1;\n+\t\t\tp = endp + 1;\n+\t\t}\n+\t\tfree(buffer);\n+\t}\n+\treturn result;\n+}\n+\n+int main(int argc, char **argv)\n+{\n+\tunsigned char sha1[20];\n+\tvoid *buffer;\n+\tunsigned long size;\n+\tunsigned char tree_sha1[20];\n+\n+\tswitch (argc) {\n+\tcase 3:\n+\t\tbasedir = argv[2];\n+\t\t/* FALLTHROUGH */\n+\tcase 2:\n+\t\tif (get_sha1_hex(argv[1], sha1) < 0)\n+\t\t\tusage(tar_tree_usage);\n+\t\tbreak;\n+\tdefault:\n+\t\tusage(tar_tree_usage);\n+\t}\n+\n+\tsha1_file_directory = getenv(DB_ENVIRONMENT);\n+\tif (!sha1_file_directory)\n+\t\tsha1_file_directory = DEFAULT_DB_ENVIRONMENT;\n+\n+\tbuffer = read_tree_with_tree_or_commit_sha1(sha1, &size, tree_sha1);\n+\tif (!buffer)\n+\t\tdie(\"unable to read sha1 file\");\n+\tif (memcmp(sha1, tree_sha1, 20))\t/* is sha1 a commit object? */\n+\t\tarchive_time = commit_time(sha1);\n+\tif (!archive_time)\n+\t\tarchive_time = time(NULL);\n+\tif (basedir)\n+\t\twrite_header(\"0\", NULL, NULL, basedir, 040755, 0);\n+\ttraverse_tree(buffer, size, NULL);\n+\tfree(buffer);\n+\twrite_trailer();\n+\treturn 0;\n+}\n"}]}