{"thread":{"id":"9794","subject":"[PATCH] allow bulk import to be packed","startedAt":"2007-09-06T01:17:19Z","lastAt":"2007-09-06T01:36:01Z","messageCount":2,"participants":["Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"52665","messageId":"7vps0wfv8g.fsf@gitster.siamese.dyndns.org","threadId":"9794","inReplyTo":null,"subject":"[PATCH] allow bulk import to be packed","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2007-09-06T01:17:19Z","receivedAt":"2007-09-06T01:17:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This will use fast-import as a backend to dump blobs resulting\nfrom a huge initial import into a packfile.\n\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n builtin-add.c |   79 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 files changed, 79 insertions(+), 0 deletions(-)\n\ndiff --git a/builtin-add.c b/builtin-add.c\nindex 105a9f0..f512108 100644\n--- a/builtin-add.c\n+++ b/builtin-add.c\n@@ -12,6 +12,7 @@\n #include \"diffcore.h\"\n #include \"commit.h\"\n #include \"revision.h\"\n+#include \"run-command.h\"\n \n static const char builtin_add_usage[] =\n \"git-add [-n] [-v] [-f] [--interactive | -i] [-u] [--refresh] [--] <filepattern>...\";\n@@ -153,6 +154,82 @@ static int git_add_config(const char *var, const char *value)\n \treturn git_default_config(var, value);\n }\n \n+static void pack_one_blob(FILE *g, size_t size, const char *path)\n+{\n+\tvoid *buf = NULL;\n+\tunsigned long nsize;\n+\tchar *nbuf;\n+\tint re_allocated = 0;\n+\n+\tif (size) {\n+\t\tint fd = open(path, O_RDONLY);\n+\t\tif (fd < 0)\n+\t\t\treturn;\n+\t\tbuf = xmmap(NULL, size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\t\tclose(fd);\n+\t}\n+\tnbuf = convert_to_git(path, buf, &nsize);\n+\tif (nbuf) {\n+\t\tmunmap(buf, size);\n+\t\tsize = nsize;\n+\t\tbuf = nbuf;\n+\t\tre_allocated = 1;\n+\t}\n+\n+\tfprintf(g, \"blob\\ndata %lu\\n\", size);\n+\tfwrite(buf, 1, size, g);\n+\tfputc('\\n', g);\n+}\n+\n+#ifndef BULK_ADD_LIMIT\n+#define BULK_ADD_LIMIT 200\n+#endif\n+\n+static void pack_bluk_blob(struct dir_struct *dir, int verbose)\n+{\n+\tstruct child_process gfi;\n+\tconst char *gfi_argv[] = { \"fast-import\", NULL };\n+\tint i, cnt;\n+\tFILE *g;\n+\tsize_t *addmap;\n+\n+\tif (dir->nr < BULK_ADD_LIMIT)\n+\t\treturn;\n+\n+\taddmap = xcalloc(dir->nr, sizeof(size_t));\n+\tfor (i = cnt = 0; i < dir->nr; i++) {\n+\t\tstruct stat st;\n+\t\tconst char *name = dir->entries[i]->name;\n+\t\tif (!lstat(name, &st) && S_ISREG(st.st_mode)) {\n+\t\t\tcnt++;\n+\t\t\taddmap[i] = xsize_t(st.st_size);\n+\t\t}\n+\t}\n+\tif (cnt < BULK_ADD_LIMIT)\n+\t\treturn;\n+\n+\tmemset(&gfi, 0, sizeof(gfi));\n+\tgfi.argv = gfi_argv;\n+\tgfi.no_stdout = 1;\n+\tgfi.in = -1;\n+\tgfi.git_cmd = 1;\n+\n+\tif (start_command(&gfi)) {\n+\t\twarning(\"could not run fast-import\");\n+\t\treturn;\n+\t}\n+\tg = fdopen(gfi.in, \"w\");\n+\n+\tfor (i = 0; i < dir->nr; i++) {\n+\t\tif (!addmap[i])\n+\t\t\tcontinue;\n+\t\tpack_one_blob(g, addmap[i], dir->entries[i]->name);\n+\t}\n+\tfflush(g);\n+\tfinish_command(&gfi);\n+\treprepare_packed_git();\n+}\n+\n static struct lock_file lock_file;\n \n static const char ignore_error[] =\n@@ -258,6 +335,8 @@ int cmd_add(int argc, const char **argv, const char *prefix)\n \t\tdie(\"no files added\");\n \t}\n \n+\tpack_bluk_blob(&dir, verbose);\n+\n \tfor (i = 0; i < dir.nr; i++)\n \t\tadd_file_to_cache(dir.entries[i]->name, verbose);\n \n"},{"id":"52667","messageId":"7vlkbkfuda.fsf@gitster.siamese.dyndns.org","threadId":"9794","inReplyTo":"7vps0wfv8g.fsf@gitster.siamese.dyndns.org","subject":"Re: [PATCH] allow bulk import to be packed","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2007-09-06T01:36:01Z","receivedAt":"2007-09-06T01:36:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> This will use fast-import as a backend to dump blobs resulting\n> from a huge initial import into a packfile.\n\nThis needs major work to be usable.\n\nAlthough fast-import knows object name of each blob, being a\nquick-and-dirty patch, git-add does not ask for it and still\ncomputes the blob names by itself.  The only thing this\nproof-of-concept patch buys is that the resulting object\ndatabase is packed, and does not get loose objects.\n"}]}