{"thread":{"id":"5153","subject":"fast-import and unique objects.","startedAt":"2006-08-06T12:32:06Z","lastAt":"2006-08-08T23:56:32Z","messageCount":15,"participants":["Jon Smirl","Shawn Pearce","Martin Langhoff","Ryan Anderson","Jakub Narebski"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"24887","messageId":"9e4733910608060532w51fca2c0r8038828df0d41eeb@mail.gmail.com","threadId":"5153","inReplyTo":null,"subject":"fast-import and unique objects.","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-08-06T12:32:06Z","receivedAt":"2006-08-06T12:32:06Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"This model has a lot of object duplication. I generated 949,305\nrevisions, but only 754,165 are unique. I'll modify my code to build a\nhash of the objects it has seen and then not send the duplicates to\nfast-import. Those 195,140 duplicated objects may be what is tripping\nindex-pack up.\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"24892","messageId":"9e4733910608060853ua0eabc1w9b35b8414d3c9bae@mail.gmail.com","threadId":"5153","inReplyTo":"9e4733910608060532w51fca2c0r8038828df0d41eeb@mail.gmail.com","subject":"Re: fast-import and unique objects.","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-08-06T15:53:43Z","receivedAt":"2006-08-06T15:53:43Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 8/6/06, Jon Smirl <jonsmirl@gmail.com> wrote:\n> This model has a lot of object duplication. I generated 949,305\n> revisions, but only 754,165 are unique. I'll modify my code to build a\n> hash of the objects it has seen and then not send the duplicates to\n> fast-import. Those 195,140 duplicated objects may be what is tripping\n> index-pack up.\n\nNew run is finished with duplicate removal.\n\nTime to run is unchanged, still 2hrs. Run time is IO bound not CPU.\nPack file is 845MB instead of 934MB.\ngit-index-pack works now, it takes 4 CPU minutes to run.\nIndex file is 18MB.\n\nSo it looks like the first stage code is working. Next I need to\nmodify cvs2svn to keep track of the sha-1 through it's sorting process\ninstead of file:revision.\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"24898","messageId":"20060806180323.GA19120@spearce.org","threadId":"5153","inReplyTo":"9e4733910608060853ua0eabc1w9b35b8414d3c9bae@mail.gmail.com","subject":"Re: fast-import and unique objects.","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-08-06T18:03:24Z","receivedAt":"2006-08-06T18:03:24Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jon Smirl <jonsmirl@gmail.com> wrote:\n> On 8/6/06, Jon Smirl <jonsmirl@gmail.com> wrote:\n> >This model has a lot of object duplication. I generated 949,305\n> >revisions, but only 754,165 are unique. I'll modify my code to build a\n> >hash of the objects it has seen and then not send the duplicates to\n> >fast-import. Those 195,140 duplicated objects may be what is tripping\n> >index-pack up.\n> \n> New run is finished with duplicate removal.\n> \n> Time to run is unchanged, still 2hrs. Run time is IO bound not CPU.\n> Pack file is 845MB instead of 934MB.\n> git-index-pack works now, it takes 4 CPU minutes to run.\n> Index file is 18MB.\n\nI'm attaching a new version of fast-import.c which generates the\nindex, and does duplicate removal.  However I think that it might\nbe slightly faster for you to do the duplicate removal in Python\nas it saves the user-kernel-user copy of the file data.  Even so,\nthis new version should save you those 4 CPU minutes as the index\nwill be generated from the in-memory SHA1s rather than needing to\nrecompute them.\n\nI've changed the calling convention:\n\n  - It now takes the pack's base name as its first parameter. It\n    appends \".pack\" and \".idx\" to form the actual file names its\n    writing to.\n\n  - It expects an estimated object count as its second parameter.\n    In your case this would be something around 760000.  This tells\n    it how large of an object table to allocate, with each entry\n    being 24 bytes + 1 pointer (28 or 32 bytes).  Overshooting\n\tthis number will cause it to degrade by allocating one\n\toverflow entry at a time from malloc.\n\nSo the new version should take about 20 MB of memory and should\nproduce a valid pack and index in the same time as it does only\nthe pack now.  Plus it won't generate duplicates.\n \n> So it looks like the first stage code is working. Next I need to\n> modify cvs2svn to keep track of the sha-1 through it's sorting process\n> instead of file:revision.\n\nWhen you get down to tree writing and commit writing we might want\nto do something similiar with the trees and commits.  I can modify\nfast-import to also store those off into a pack.\n\n-- \nShawn.\n\n\n#include \"builtin.h\"\n#include \"cache.h\"\n#include \"object.h\"\n#include \"blob.h\"\n#include \"delta.h\"\n#include \"pack.h\"\n#include \"csum-file.h\"\n\nstatic int max_depth = 10;\nstatic unsigned long object_count;\nstatic unsigned long duplicate_count;\nstatic unsigned long packoff;\nstatic unsigned long overflow_count;\nstatic int packfd;\nstatic int current_depth;\nstatic void *lastdat;\nstatic unsigned long lastdatlen;\nstatic unsigned char lastsha1[20];\nstatic unsigned char packsha1[20];\n\nstruct object_entry\n{\n\tstruct object_entry *next;\n\tunsigned long offset;\n\tunsigned char sha1[20];\n};\n\nstruct overflow_object_entry\n{\n\tstruct overflow_object_entry *next;\n\tstruct object_entry oe;\n};\n\nstruct object_entry *pool_start;\nstruct object_entry *pool_next;\nstruct object_entry *pool_end;\nstruct overflow_object_entry *overflow;\nstruct object_entry *table[1 << 16];\n\nstatic struct object_entry* new_object(unsigned char *sha1)\n{\n\tif (pool_next != pool_end) {\n\t\tstruct object_entry *e = pool_next++;\n\t\tmemcpy(e->sha1, sha1, sizeof(e->sha1));\n\t\treturn e;\n\t} else {\n\t\tstruct overflow_object_entry *e;\n\n\t\te = xmalloc(sizeof(struct overflow_object_entry));\n\t\te->next = overflow;\n\t\tmemcpy(e->oe.sha1, sha1, sizeof(e->oe.sha1));\n\t\toverflow = e;\n\t\toverflow_count++;\n\t\treturn &e->oe;\n\t}\n}\n\nstatic struct object_entry* insert_object(unsigned char *sha1)\n{\n\tunsigned int h = sha1[0] << 8 | sha1[1];\n\tstruct object_entry *e = table[h];\n\tstruct object_entry *p = 0;\n\n\twhile (e) {\n\t\tif (!memcmp(sha1, e->sha1, sizeof(e->sha1)))\n\t\t\treturn e;\n\t\tp = e;\n\t\te = e->next;\n\t}\n\n\te = new_object(sha1);\n\te->next = 0;\n\te->offset = 0;\n\tif (p)\n\t\tp->next = e;\n\telse\n\t\ttable[h] = e;\n\treturn e;\n}\n\nstatic ssize_t yread(int fd, void *buffer, size_t length)\n{\n\tssize_t ret = 0;\n\twhile (ret < length) {\n\t\tssize_t size = xread(fd, (char *) buffer + ret, length - ret);\n\t\tif (size < 0) {\n\t\t\treturn size;\n\t\t}\n\t\tif (size == 0) {\n\t\t\treturn ret;\n\t\t}\n\t\tret += size;\n\t}\n\treturn ret;\n}\n\nstatic ssize_t ywrite(int fd, void *buffer, size_t length)\n{\n\tssize_t ret = 0;\n\twhile (ret < length) {\n\t\tssize_t size = xwrite(fd, (char *) buffer + ret, length - ret);\n\t\tif (size < 0) {\n\t\t\treturn size;\n\t\t}\n\t\tif (size == 0) {\n\t\t\treturn ret;\n\t\t}\n\t\tret += size;\n\t}\n\treturn ret;\n}\n\nstatic unsigned long encode_header(enum object_type type, unsigned long size, unsigned char *hdr)\n{\n\tint n = 1;\n\tunsigned char c;\n\n\tif (type < OBJ_COMMIT || type > OBJ_DELTA)\n\t\tdie(\"bad type %d\", type);\n\n\tc = (type << 4) | (size & 15);\n\tsize >>= 4;\n\twhile (size) {\n\t\t*hdr++ = c | 0x80;\n\t\tc = size & 0x7f;\n\t\tsize >>= 7;\n\t\tn++;\n\t}\n\t*hdr = c;\n\treturn n;\n}\n\nstatic void write_blob(void *dat, unsigned long datlen)\n{\n\tz_stream s;\n\tvoid *out, *delta;\n\tunsigned char hdr[64];\n\tunsigned long hdrlen, deltalen;\n\n\tif (lastdat && current_depth < max_depth) {\n\t\tdelta = diff_delta(lastdat, lastdatlen,\n\t\t\tdat, datlen,\n\t\t\t&deltalen, 0);\n\t} else\n\t\tdelta = 0;\n\n\tmemset(&s, 0, sizeof(s));\n\tdeflateInit(&s, zlib_compression_level);\n\n\tif (delta) {\n\t\tcurrent_depth++;\n\t\ts.next_in = delta;\n\t\ts.avail_in = deltalen;\n\t\thdrlen = encode_header(OBJ_DELTA, deltalen, hdr);\n\t\tif (ywrite(packfd, hdr, hdrlen) != hdrlen)\n\t\t\tdie(\"Can't write object header: %s\", strerror(errno));\n\t\tif (ywrite(packfd, lastsha1, sizeof(lastsha1)) != sizeof(lastsha1))\n\t\t\tdie(\"Can't write object base: %s\", strerror(errno));\n\t\tpackoff += hdrlen + sizeof(lastsha1);\n\t} else {\n\t\tcurrent_depth = 0;\n\t\ts.next_in = dat;\n\t\ts.avail_in = datlen;\n\t\thdrlen = encode_header(OBJ_BLOB, datlen, hdr);\n\t\tif (ywrite(packfd, hdr, hdrlen) != hdrlen)\n\t\t\tdie(\"Can't write object header: %s\", strerror(errno));\n\t\tpackoff += hdrlen;\n\t}\n\n\ts.avail_out = deflateBound(&s, s.avail_in);\n\ts.next_out = out = xmalloc(s.avail_out);\n\twhile (deflate(&s, Z_FINISH) == Z_OK)\n\t\t/* nothing */;\n\tdeflateEnd(&s);\n\n\tif (ywrite(packfd, out, s.total_out) != s.total_out)\n\t\tdie(\"Failed writing compressed data %s\", strerror(errno));\n\tpackoff += s.total_out;\n\n\tfree(out);\n\tif (delta)\n\t\tfree(delta);\n}\n\nstatic void init_pack_header()\n{\n\tconst char* magic = \"PACK\";\n\tunsigned long version = 2;\n\tunsigned long zero = 0;\n\n\tversion = htonl(version);\n\n\tif (ywrite(packfd, (char*)magic, 4) != 4)\n\t\tdie(\"Can't write pack magic: %s\", strerror(errno));\n\tif (ywrite(packfd, &version, 4) != 4)\n\t\tdie(\"Can't write pack version: %s\", strerror(errno));\n\tif (ywrite(packfd, &zero, 4) != 4)\n\t\tdie(\"Can't write 0 object count: %s\", strerror(errno));\n\tpackoff = 4 * 3;\n}\n\nstatic void fixup_header_footer()\n{\n\tSHA_CTX c;\n\tchar hdr[8];\n\tunsigned long cnt;\n\tchar *buf;\n\tsize_t n;\n\n\tif (lseek(packfd, 0, SEEK_SET) != 0)\n\t\tdie(\"Failed seeking to start: %s\", strerror(errno));\n\n\tSHA1_Init(&c);\n\tif (yread(packfd, hdr, 8) != 8)\n\t\tdie(\"Failed reading header: %s\", strerror(errno));\n\tSHA1_Update(&c, hdr, 8);\n\n\tcnt = htonl(object_count);\n\tSHA1_Update(&c, &cnt, 4);\n\tif (ywrite(packfd, &cnt, 4) != 4)\n\t\tdie(\"Failed writing object count: %s\", strerror(errno));\n\n\tbuf = xmalloc(128 * 1024);\n\tfor (;;) {\n\t\tn = xread(packfd, buf, 128 * 1024);\n\t\tif (n <= 0)\n\t\t\tbreak;\n\t\tSHA1_Update(&c, buf, n);\n\t}\n\tfree(buf);\n\n\tSHA1_Final(packsha1, &c);\n\tif (ywrite(packfd, packsha1, sizeof(packsha1)) != sizeof(packsha1))\n\t\tdie(\"Failed writing pack checksum: %s\", strerror(errno));\n}\n\nstatic int oecmp (const void *_a, const void *_b)\n{\n\tstruct object_entry *a = *((struct object_entry**)_a);\n\tstruct object_entry *b = *((struct object_entry**)_b);\n\treturn memcmp(a->sha1, b->sha1, sizeof(a->sha1));\n}\n\nstatic void write_index(const char *idx_name)\n{\n\tstruct sha1file *f;\n\tstruct object_entry **idx, **c, **last;\n\tstruct object_entry *e;\n\tstruct overflow_object_entry *o;\n\tunsigned int array[256];\n\tint i;\n\n\t/* Build the sorted table of object IDs. */\n\tidx = xmalloc(object_count * sizeof(struct object_entry*));\n\tc = idx;\n\tfor (e = pool_start; e != pool_next; e++)\n\t\t*c++ = e;\n\tfor (o = overflow; o; o = o->next)\n\t\t*c++ = &o->oe;\n\tlast = idx + object_count;\n\tqsort(idx, object_count, sizeof(struct object_entry*), oecmp);\n\n\t/* Generate the fan-out array. */\n\tc = idx;\n\tfor (i = 0; i < 256; i++) {\n\t\tstruct object_entry **next = c;;\n\t\twhile (next < last) {\n\t\t\tif ((*next)->sha1[0] != i)\n\t\t\t\tbreak;\n\t\t\tnext++;\n\t\t}\n\t\tarray[i] = htonl(next - idx);\n\t\tc = next;\n\t}\n\n\tf = sha1create(\"%s\", idx_name);\n\tsha1write(f, array, 256 * sizeof(int));\n\tfor (c = idx; c != last; c++) {\n\t\tunsigned int offset = htonl((*c)->offset);\n\t\tsha1write(f, &offset, 4);\n\t\tsha1write(f, (*c)->sha1, sizeof((*c)->sha1));\n\t}\n\tsha1write(f, packsha1, sizeof(packsha1));\n\tsha1close(f, NULL, 1);\n\tfree(idx);\n}\n\nint main(int argc, const char **argv)\n{\n\tconst char *base_name = argv[1];\n\tint est_obj_cnt = atoi(argv[2]);\n\tchar *pack_name;\n\tchar *idx_name;\n\n\tpack_name = xmalloc(strlen(base_name) + 6);\n\tsprintf(pack_name, \"%s.pack\", base_name);\n\tidx_name = xmalloc(strlen(base_name) + 5);\n\tsprintf(idx_name, \"%s.idx\", base_name);\n\n\tpackfd = open(pack_name, O_RDWR|O_CREAT|O_TRUNC, 0666);\n\tif (packfd < 0)\n\t\tdie(\"Can't create pack file %s: %s\", pack_name, strerror(errno));\n\n\tpool_start = xmalloc(est_obj_cnt * sizeof(struct object_entry));\n\tpool_next = pool_start;\n\tpool_end = pool_start + est_obj_cnt;\n\n\tinit_pack_header();\n\tfor (;;) {\n\t\tunsigned long datlen;\n\t\tint hdrlen;\n\t\tvoid *dat;\n\t\tchar hdr[128];\n\t\tunsigned char sha1[20];\n\t\tSHA_CTX c;\n\t\tstruct object_entry *e;\n\n\t\tif (yread(0, &datlen, 4) != 4)\n\n\t\t\tbreak;\n\n\t\tdat = xmalloc(datlen);\n\t\tif (yread(0, dat, datlen) != datlen)\n\t\t\tbreak;\n\n\t\thdrlen = sprintf(hdr, \"blob %lu\", datlen) + 1;\n\t\tSHA1_Init(&c);\n\t\tSHA1_Update(&c, hdr, hdrlen);\n\t\tSHA1_Update(&c, dat, datlen);\n\t\tSHA1_Final(sha1, &c);\n\n\t\te = insert_object(sha1);\n\t\tif (!e->offset) {\n\t\t\te->offset = packoff;\n\t\t\twrite_blob(dat, datlen);\n\t\t\tobject_count++;\n\t\t\tprintf(\"%s\\n\", sha1_to_hex(sha1));\n\t\t\tfflush(stdout);\n\n\t\t\tif (lastdat)\n\t\t\t\tfree(lastdat);\n\t\t\tlastdat = dat;\n\t\t\tlastdatlen = datlen;\n\t\t\tmemcpy(lastsha1, sha1, sizeof(sha1));\n\t\t} else {\n\t\t\tduplicate_count++;\n\t\t\tfree(dat);\n\t\t}\n\t}\n\tfixup_header_footer();\n\tclose(packfd);\n\twrite_index(idx_name);\n\n\tfprintf(stderr, \"%lu objects, %lu duplicates, %lu pool overflow\\n\",\n\t\tobject_count, duplicate_count, overflow_count);\n\n\treturn 0;\n}\n"},{"id":"24907","messageId":"9e4733910608062148u4341dabag451c3f49f1a792a1@mail.gmail.com","threadId":"5153","inReplyTo":"20060806180323.GA19120@spearce.org","subject":"Re: fast-import and unique objects.","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-08-07T04:48:47Z","receivedAt":"2006-08-07T04:48:47Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 8/6/06, Shawn Pearce <spearce@spearce.org> wrote:\n> So the new version should take about 20 MB of memory and should\n> produce a valid pack and index in the same time as it does only\n> the pack now.  Plus it won't generate duplicates.\n\nI did a run with this and it works great.\n\nI'm staring at the cvs2svn code now trying to figure out how to modify\nit without rewriting everything. I may just leave it all alone and\nbuild a table with cvs_file:rev to sha-1 mappings. It would be much\nmore efficient to carry sha-1 throughout the stages but that may\nrequire significant rework.\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"24908","messageId":"20060807050422.GD20514@spearce.org","threadId":"5153","inReplyTo":"9e4733910608062148u4341dabag451c3f49f1a792a1@mail.gmail.com","subject":"Re: fast-import and unique objects.","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-08-07T05:04:23Z","receivedAt":"2006-08-07T05:04:23Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jon Smirl <jonsmirl@gmail.com> wrote:\n> On 8/6/06, Shawn Pearce <spearce@spearce.org> wrote:\n> >So the new version should take about 20 MB of memory and should\n> >produce a valid pack and index in the same time as it does only\n> >the pack now.  Plus it won't generate duplicates.\n> \n> I did a run with this and it works great.\n\nGood.  :-) On my drive in to work this afternoon I realized\nthat making you specify the size of the object table is stupid,\nI could easily allocate a thousand objects at a time rather than\npreallocating the whole thing.  Oh well.  fast-import thus far\nhasn't been meant as production code for inclusion in core GIT,\nbut maybe it will get cleaned up and submitted as such if your\nconversion efforts go well and produce a better CVS importer.\n \n> I'm staring at the cvs2svn code now trying to figure out how to modify\n> it without rewriting everything. I may just leave it all alone and\n> build a table with cvs_file:rev to sha-1 mappings. It would be much\n> more efficient to carry sha-1 throughout the stages but that may\n> require significant rework.\n\nDoes it matter?  How long does the cvs2svn processing take,\nexcluding the GIT blob processing that's now known to take 2 hours?\nWhat's your target for an acceptable conversion time on the system\nyou are working on?\n\n\nAny thoughts yet on how you might want to feed trees and commits\nto a fast pack writer?  I was thinking about doing a stream into\nfast-import such as:\n\n\t<4 byte length of commit><commit><treeent>*<null>\n\nwhere <commit> is the raw commit minus the first \"tree nnn\\n\" line, and\n<treeent> is:\n\n\t<type><sp><sha1><sp><path><null>\n\nwhere <type> is one of 'B' (normal blob), 'L' (symlink), 'X'\n(executable blob), <sha1> is the 40 byte hex, <path> is the file from\nthe root of the repository (\"src/module/foo.c\"), and <sp> and <null>\nare the obvious values.  You would feed all tree entries and the pack\nwriter would split the stream up into the individual tree objects.\n\nfast-import would generate the tree(s) delta'ing them against the\nprior tree of the same path, prefix \"tree nnn\\n\" to the commit\nblob you supplied, generate the commit, and print out its ID.\nBy working from the first commit up to the most recent each tree\ndeltas would be using the older tree as the base which may not be\nideal if a large number of items get added to a tree but should be\neffective enough to generate a reasonably sized initial pack.\n\nIt would however mean you need to monitor the output pipe from\nfast-import to get back the commit id so you can use it to prep\nthe next commit's parent(s) as you can't produce that in Python.\n\n-- \nShawn.\n"},{"id":"24909","messageId":"46a038f90608062210h1c13ccf9p77e19072b68d49d6@mail.gmail.com","threadId":"5153","inReplyTo":"9e4733910608062148u4341dabag451c3f49f1a792a1@mail.gmail.com","subject":"Re: fast-import and unique objects.","fromName":"Martin Langhoff","fromEmail":"martin.langhoff@gmail.com","sentAt":"2006-08-07T05:10:10Z","receivedAt":"2006-08-07T05:10:10Z","isPatch":false,"sender":{"key":"martin.langhoff@gmail.com","avatar":"https://gravatar.com/avatar/1e3f311b6c4c15836501901ca58f8c0b0667246488084ba524d8bc9867e22fd9?d=mp&s=160"},"body":"On 8/7/06, Jon Smirl <jonsmirl@gmail.com> wrote:\n> On 8/6/06, Shawn Pearce <spearce@spearce.org> wrote:\n> > So the new version should take about 20 MB of memory and should\n> > produce a valid pack and index in the same time as it does only\n> > the pack now.  Plus it won't generate duplicates.\n>\n> I did a run with this and it works great.\n\nGreat.\n\n> I'm staring at the cvs2svn code now trying to figure out how to modify\n> it without rewriting everything. I may just leave it all alone and\n> build a table with cvs_file:rev to sha-1 mappings. It would be much\n> more efficient to carry sha-1 throughout the stages but that may\n> require significant rework.\n\nProbably a good thing. If the patch isn't intrusive it has a better\nchance of merging well over time and of being merged upstream -- ;-)\n\n\nm\n"},{"id":"24911","messageId":"20060807075744.GB15477@h4x0r5.com","threadId":"5153","inReplyTo":"20060806180323.GA19120@spearce.org","subject":"Re: fast-import and unique objects.","fromName":"Ryan Anderson","fromEmail":"ryan@michonline.com","sentAt":"2006-08-07T07:57:45Z","receivedAt":"2006-08-07T07:57:45Z","isPatch":false,"sender":{"key":"ryan@michonline.com","avatar":null},"body":"On Sun, Aug 06, 2006 at 02:03:24PM -0400, Shawn Pearce wrote:\n> \n>   - It expects an estimated object count as its second parameter.\n>     In your case this would be something around 760000.  This tells\n>     it how large of an object table to allocate, with each entry\n>     being 24 bytes + 1 pointer (28 or 32 bytes).  Overshooting\n> \tthis number will cause it to degrade by allocating one\n> \toverflow entry at a time from malloc.\n\nHrm, you're allocating a big table and then assigning consecutive\nentries out of it, as pointers.\n\nWhy not just malloc a big block, and assign offsets into it, as if it\nwere a really big array.  Every time it runs out, realloc it to double\nthe current size, and update the base pointer.\n\n-- \n\nRyan Anderson\n  sometimes Pug Majere\n"},{"id":"24924","messageId":"9e4733910608070737k52aaea7clf871d716d16547c2@mail.gmail.com","threadId":"5153","inReplyTo":"20060807050422.GD20514@spearce.org","subject":"Re: fast-import and unique objects.","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-08-07T14:37:30Z","receivedAt":"2006-08-07T14:37:30Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 8/7/06, Shawn Pearce <spearce@spearce.org> wrote:\n> > I'm staring at the cvs2svn code now trying to figure out how to modify\n> > it without rewriting everything. I may just leave it all alone and\n> > build a table with cvs_file:rev to sha-1 mappings. It would be much\n> > more efficient to carry sha-1 throughout the stages but that may\n> > require significant rework.\n>\n> Does it matter?  How long does the cvs2svn processing take,\n> excluding the GIT blob processing that's now known to take 2 hours?\n> What's your target for an acceptable conversion time on the system\n> you are working on?\n\nAs is, it takes the code about a week to import MozCVS into\nSubversion. But I've already addressed the core of why that was taking\nso long. The original code forks off a copy of cvs for each revision\nto exact the text. Doing that 1M times takes about two days. The\nversion with fast-import takes two hours.\n\nAt the end of the process cvs2svn forks off svn 250K times to import\nthe change sets. That takes about four days to finish. Doing a\nfast-import backend should fix that.\n\n> Any thoughts yet on how you might want to feed trees and commits\n> to a fast pack writer?  I was thinking about doing a stream into\n> fast-import such as:\n\nThe data I have generates an output that indicates add/change/delete\nfor each file name. Add/change should have an associated sha-1 for the\nnew revision. cvs/svn have no concept of trees.\n\nHow about sending out a stream of add/change/delete operations\ninterspersed with commits? That would let fast-import track the tree\nand only generate tree nodes when they change.\n\nThe protocol may need some thought. I need to be able to handle\nbranches and labels too.\n\n\n>         <4 byte length of commit><commit><treeent>*<null>\n>\n> where <commit> is the raw commit minus the first \"tree nnn\\n\" line, and\n> <treeent> is:\n>\n>         <type><sp><sha1><sp><path><null>\n>\n> where <type> is one of 'B' (normal blob), 'L' (symlink), 'X'\n> (executable blob), <sha1> is the 40 byte hex, <path> is the file from\n> the root of the repository (\"src/module/foo.c\"), and <sp> and <null>\n> are the obvious values.  You would feed all tree entries and the pack\n> writer would split the stream up into the individual tree objects.\n>\n> fast-import would generate the tree(s) delta'ing them against the\n> prior tree of the same path, prefix \"tree nnn\\n\" to the commit\n> blob you supplied, generate the commit, and print out its ID.\n> By working from the first commit up to the most recent each tree\n> deltas would be using the older tree as the base which may not be\n> ideal if a large number of items get added to a tree but should be\n> effective enough to generate a reasonably sized initial pack.\n>\n> It would however mean you need to monitor the output pipe from\n> fast-import to get back the commit id so you can use it to prep\n> the next commit's parent(s) as you can't produce that in Python.\n>\n> --\n> Shawn.\n>\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"24925","messageId":"eb7jr2$4ar$1@sea.gmane.org","threadId":"5153","inReplyTo":"9e4733910608070737k52aaea7clf871d716d16547c2@mail.gmail.com","subject":"Re: fast-import and unique objects.","fromName":"Jakub Narebski","fromEmail":"jnareb@gmail.com","sentAt":"2006-08-07T14:48:08Z","receivedAt":"2006-08-07T14:48:08Z","isPatch":false,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Jon Smirl wrote:\n\n> How about sending out a stream of add/change/delete operations\n> interspersed with commits? That would let fast-import track the tree\n> and only generate tree nodes when they change.\n\nThe problem with CVS, which cvsps (CVS PatchSet) tries to address is that\nchanges are to a file, and reconstructing which changes go together to make\n_one_ commit...\n\n-- \nJakub Narebski\nWarsaw, Poland\nShadeHawk on #git\n"},{"id":"24936","messageId":"9e4733910608071145i6cbfa9b2ub8ed52020614d9e7@mail.gmail.com","threadId":"5153","inReplyTo":"eb7jr2$4ar$1@sea.gmane.org","subject":"Re: fast-import and unique objects.","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-08-07T18:45:56Z","receivedAt":"2006-08-07T18:45:56Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 8/7/06, Jakub Narebski <jnareb@gmail.com> wrote:\n> Jon Smirl wrote:\n>\n> > How about sending out a stream of add/change/delete operations\n> > interspersed with commits? That would let fast-import track the tree\n> > and only generate tree nodes when they change.\n>\n> The problem with CVS, which cvsps (CVS PatchSet) tries to address is that\n> changes are to a file, and reconstructing which changes go together to make\n> _one_ commit...\n\nThe cvs2svn code is also gathering CVS commits into change sets. The\nadvantage of the cvs2svn code is that it is written by some of the\noriginal CVS developers. Those developers are familiar with the 1,000\ndifferent ways to break a CVS repository and cvs2svn tries to deal\nwith the breakage.\n\ncvsps works for some repositories but it fails on the Mozilla one.\nThere is something about branches in the Mozilla repository that\nconfuses it. cvs2svn handles the Mozilla repository correctly as far\nas I can tell.\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"24963","messageId":"20060807230258.GA19067@spearce.org","threadId":"5153","inReplyTo":"20060807075744.GB15477@h4x0r5.com","subject":"Re: fast-import and unique objects.","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-08-07T23:02:58Z","receivedAt":"2006-08-07T23:02:58Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Ryan Anderson <ryan@michonline.com> wrote:\n> On Sun, Aug 06, 2006 at 02:03:24PM -0400, Shawn Pearce wrote:\n> > \n> >   - It expects an estimated object count as its second parameter.\n> >     In your case this would be something around 760000.  This tells\n> >     it how large of an object table to allocate, with each entry\n> >     being 24 bytes + 1 pointer (28 or 32 bytes).  Overshooting\n> > \tthis number will cause it to degrade by allocating one\n> > \toverflow entry at a time from malloc.\n> \n> Hrm, you're allocating a big table and then assigning consecutive\n> entries out of it, as pointers.\n> \n> Why not just malloc a big block, and assign offsets into it, as if it\n> were a really big array.  Every time it runs out, realloc it to double\n> the current size, and update the base pointer.\n\nBecause I didn't want to move a 24 MB block of memory.  :-)\n\nI'm probably going to clean that section of code up tonight and\nallocate a large block at the beginning then allocate overflow blocks\nat about 5000 entries at a time.  There's no need for the blocks to\nbe contiguous in memory, I just didn't want to have a high overhead\nfrom malloc when there would be a large number of them...\n\n-- \nShawn.\n"},{"id":"24967","messageId":"20060808031240.GB19067@spearce.org","threadId":"5153","inReplyTo":"9e4733910608070737k52aaea7clf871d716d16547c2@mail.gmail.com","subject":"Re: fast-import and unique objects.","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-08-08T03:12:41Z","receivedAt":"2006-08-08T03:12:41Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jon Smirl <jonsmirl@gmail.com> wrote:\n> the change sets. That takes about four days to finish. Doing a\n> fast-import backend should fix that.\n\nShouldn't be a problem.  :-)\n \n> >Any thoughts yet on how you might want to feed trees and commits\n> >to a fast pack writer?  I was thinking about doing a stream into\n> >fast-import such as:\n> \n> The data I have generates an output that indicates add/change/delete\n> for each file name. Add/change should have an associated sha-1 for the\n> new revision. cvs/svn have no concept of trees.\n> \n> How about sending out a stream of add/change/delete operations\n> interspersed with commits? That would let fast-import track the tree\n> and only generate tree nodes when they change.\n> \n> The protocol may need some thought. I need to be able to handle\n> branches and labels too.\n\nKnowing a little bit about SVN I would assume the current cvs2svn\ncode would issue commands like:\n\n    svn copy A B;   # Make branch B starting from where A is now.\n    svn switch B;   # Make branch B the current branch.\n    svn add F;      # Add file F.\n    svn rm Y;       # Delete file Y.\n    svn commit;     # Commit current tree on current branch.\n    svn copy A T;   # Create tag T from where A is now.\n\nBut I don't know how it would issue a merge commit.  Or even if it\ncould find such a thing in the RCS files.\n\nThe above command set would be rather trivial to implement in a\nfast-import backend.  I'm thinking we extend the protocol so it\nlooks something like the following:\n\n  stream ::= cmd*;\n\n  cmd ::= new_blob\n        | new_branch\n        | set_branch\n        | update\n        | commit\n        | tag\n        ;\n\n  new_blob    ::= 'blob' blob_data;\n  new_branch  ::= 'newb' branch_name source_branch;\n  set_branch  ::= 'setb' branch_name;\n  add_file    ::= update_file;\n  update_file ::= 'updf' file_update;\n  delete_file ::= 'delf' file_delete;\n  commit      ::= 'comt' author_committer_msg;\n  tag         ::= 'tagg' branch_name tagger_msg;\n\n  source_branch ::= branch_name\n                  | zero32\n                  ;\n  branch_name ::= len32 branch_name_str;\n  branch_name_str ::= # valid GIT branch name, should be relative\n                      # to .git/ (so include a refs/heads prefix)\n                    ;\n  file_update ::= len32 mode sp hexsha1 sp path;\n  file_delete ::= len32 path;\n\n  blob_data ::= len32 binary_data;\n\n  author_committer_msg ::= len32\n    'author' sp name '<' email '>' ts tz lf\n    'committer' sp name '<' email '>' ts tz lf\n    lf\n    binary_data;\n\n  tagger_msg ::= len32\n    'tagger' sp name '<' email '>' ts tz lf\n    lf\n    binary_data;\n\n  len32 ::= # unsigned 32 bit value, native format;\n  zero32 ::= # 32 bits, unset, aka '0';\n\n  mode ::= 'P'    # plain file\n         | 'X'    # executable file\n         ;\n\n  binary_data ::= # file content, not interpreted;\n  sp ::= # ASCII space character;\n  lf ::= # ASCII newline (LF) character;\n  path ::= # GIT style file path, e.g. \"a/b/c\";\n  hexsha1 ::= # SHA1 in hexadecimal format;\n  nullsha1 ::= # 40 '0's in sequence;\n  name ::= # valid GIT author/committer name;\n  email ::= # valid GIT author/committer email;\n  ts ::= # time since the epoch in seconds, ascii decimal;\n  tz ::= # GIT style timezone;\n\nThis is a change from the current protocol as new blobs need to\nget prefixed by the command 'blob'.  Note that all commands are 4\nbytes wide and that any variable data portion of a command uses a\n\"Pascal style\" leading 32 bit length.  Although ugly it just makes\nit a tiny bit easier to code the stream parser.  :-)\n\nThe commits and tags are half generated in the frontend and half\nin the backend.  The backend is producing and tracking the tree and\nthe current SHA1 of each branch.  Consequently it will generate the\n'tree' and 'parent' lines of a commit or the 'object', 'type' and\n'tag' lines of a tag.  This is limited to only a linear development\npath, no merges...\n\nThe backend would need to run inside of an existing GIT repository\nas it would output all tags and branch heads when it terminates.\n(Right now it runs from anywhere.)\n\nI don't know how many branches you would be asking the backend to\nhold at once, but I was thinking that the backend would just create\na tree structure in memory when it receives a new_branch command,\nand consider one of those to be the current branch when a set_branch\ncommand is issued.  With all branches in memory at once switching\nwould be very cheap, but if the number of branches is high it could\neat memory like a pig...\n\nRight now I'm thinking that a file entry in a tree would cost:\n\n  29 bytes + length_of_name + malloc_overhead\n\nwhile a single tree would cost that plus:\n\n  36 bytes + 4*number_of_entries + 2*malloc_overhead + last_treesz\n\nwhere last_treesz is last content of that tree (uncompressed),\nso we can deltify against it quickly.\n\nSo what's your worst case number of files and directories in a\nsingle commit?  And how many branches are we talking about carrying\naround in the backend?\n\n-- \nShawn.\n"},{"id":"24977","messageId":"9e4733910608080511t5aa96865p41d6bc1b85e236fa@mail.gmail.com","threadId":"5153","inReplyTo":"20060808031240.GB19067@spearce.org","subject":"Re: fast-import and unique objects.","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-08-08T12:11:15Z","receivedAt":"2006-08-08T12:11:15Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 8/7/06, Shawn Pearce <spearce@spearce.org> wrote:\n> Knowing a little bit about SVN I would assume the current cvs2svn\n> code would issue commands like:\n\nWe're designing a dumpfile format for git like the one SVN has.\n\n>     svn copy A B;   # Make branch B starting from where A is now.\n>     svn switch B;   # Make branch B the current branch.\n>     svn add F;      # Add file F.\n>     svn rm Y;       # Delete file Y.\n>     svn commit;     # Commit current tree on current branch.\n>     svn copy A T;   # Create tag T from where A is now.\n>\n> But I don't know how it would issue a merge commit.  Or even if it\n> could find such a thing in the RCS files.\n\nAFAIK the svn code doesn't do merge commits. We probably need a post\nprocessing pass in the git repo that finds the merges and closes off\nthe branches. gitk won't be pretty with 1,500 open branches. This may\nneed some manual clues.\n\nAnother thing missing in cvs2svn is a sane way to handle unnamed\nbranches. Right now it is generating names. It's not real good at\ndetecting everything that belongs together in an unnamed branch so one\nunnamed branches can turn into dozens of branches.\n\nOnce we get everything in a tool where it is possible to visualize the\nrepo errors like this will be more obvious. That should make it easier\nto come up with a strategy to fix the import process.\n\n> The above command set would be rather trivial to implement in a\n> fast-import backend.  I'm thinking we extend the protocol so it\n> looks something like the following:\n>\n>   stream ::= cmd*;\n>\n>   cmd ::= new_blob\n>         | new_branch\n>         | set_branch\n>         | update\n>         | commit\n>         | tag\n>         ;\n>\n>   new_blob    ::= 'blob' blob_data;\n>   new_branch  ::= 'newb' branch_name source_branch;\n>   set_branch  ::= 'setb' branch_name;\n>   add_file    ::= update_file;\n>   update_file ::= 'updf' file_update;\n>   delete_file ::= 'delf' file_delete;\n>   commit      ::= 'comt' author_committer_msg;\n>   tag         ::= 'tagg' branch_name tagger_msg;\n>\n>   source_branch ::= branch_name\n>                   | zero32\n>                   ;\n>   branch_name ::= len32 branch_name_str;\n>   branch_name_str ::= # valid GIT branch name, should be relative\n>                       # to .git/ (so include a refs/heads prefix)\n>                     ;\n>   file_update ::= len32 mode sp hexsha1 sp path;\n>   file_delete ::= len32 path;\n>\n>   blob_data ::= len32 binary_data;\n>\n>   author_committer_msg ::= len32\n>     'author' sp name '<' email '>' ts tz lf\n>     'committer' sp name '<' email '>' ts tz lf\n>     lf\n>     binary_data;\n>\n>   tagger_msg ::= len32\n>     'tagger' sp name '<' email '>' ts tz lf\n>     lf\n>     binary_data;\n>\n>   len32 ::= # unsigned 32 bit value, native format;\n>   zero32 ::= # 32 bits, unset, aka '0';\n>\n>   mode ::= 'P'    # plain file\n>          | 'X'    # executable file\n>          ;\n>\n>   binary_data ::= # file content, not interpreted;\n>   sp ::= # ASCII space character;\n>   lf ::= # ASCII newline (LF) character;\n>   path ::= # GIT style file path, e.g. \"a/b/c\";\n>   hexsha1 ::= # SHA1 in hexadecimal format;\n>   nullsha1 ::= # 40 '0's in sequence;\n>   name ::= # valid GIT author/committer name;\n>   email ::= # valid GIT author/committer email;\n>   ts ::= # time since the epoch in seconds, ascii decimal;\n>   tz ::= # GIT style timezone;\n>\n> This is a change from the current protocol as new blobs need to\n> get prefixed by the command 'blob'.  Note that all commands are 4\n> bytes wide and that any variable data portion of a command uses a\n> \"Pascal style\" leading 32 bit length.  Although ugly it just makes\n> it a tiny bit easier to code the stream parser.  :-)\n>\n> The commits and tags are half generated in the frontend and half\n> in the backend.  The backend is producing and tracking the tree and\n> the current SHA1 of each branch.  Consequently it will generate the\n> 'tree' and 'parent' lines of a commit or the 'object', 'type' and\n> 'tag' lines of a tag.  This is limited to only a linear development\n> path, no merges...\n>\n> The backend would need to run inside of an existing GIT repository\n> as it would output all tags and branch heads when it terminates.\n> (Right now it runs from anywhere.)\n>\n> I don't know how many branches you would be asking the backend to\n> hold at once, but I was thinking that the backend would just create\n> a tree structure in memory when it receives a new_branch command,\n> and consider one of those to be the current branch when a set_branch\n> command is issued.  With all branches in memory at once switching\n> would be very cheap, but if the number of branches is high it could\n> eat memory like a pig...\n>\n> Right now I'm thinking that a file entry in a tree would cost:\n>\n>   29 bytes + length_of_name + malloc_overhead\n>\n> while a single tree would cost that plus:\n>\n>   36 bytes + 4*number_of_entries + 2*malloc_overhead + last_treesz\n>\n> where last_treesz is last content of that tree (uncompressed),\n> so we can deltify against it quickly.\n\nThe file names are used over and over. Alloc a giant chunk of memory\nand keep appending the file name strings to it. Then build a little\ntree so that you can look up existing names. i.e. turn the files names\ninto atoms. Never delete anything.\n\nYou can do something similar with the 29 and 36 byte arrays. Alloc two\ngiant chunks of memory. Append new entries to the end. Maintain a list\nof deleted entries for reuse (store the pointer to the next free entry\ninside the unused blocks, don't use a separate structure). You have\nthree things to track, end of used memory, head of free list, end of\nallocated memory.\n\nNow there is no malloc overhead on millions of items.\n\n> So what's your worst case number of files and directories in a\n> single commit?  And how many branches are we talking about carrying\n> around in the backend?\n\nAbout 100,000 files in the initial change set that builds the repo.\nFInal repo has 120,000 files.\n\nThere are 1,500 branches. I haven't looked at the svn dump file format\nfor branches, but I suspect that it sends everything on a branch out\nat once and doesn't intersperse it with the trunk commits.\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"},{"id":"25004","messageId":"20060808224537.GB18163@spearce.org","threadId":"5153","inReplyTo":"9e4733910608080511t5aa96865p41d6bc1b85e236fa@mail.gmail.com","subject":"Re: fast-import and unique objects.","fromName":"Shawn Pearce","fromEmail":"spearce@spearce.org","sentAt":"2006-08-08T22:45:37Z","receivedAt":"2006-08-08T22:45:37Z","isPatch":false,"sender":{"key":"spearce@spearce.org","avatar":"https://avatars.githubusercontent.com/u/34844?v=4"},"body":"Jon Smirl <jonsmirl@gmail.com> wrote:\n> We're designing a dumpfile format for git like the one SVN has.\n\nI'm not sure I'd call it a dumpfile format.  More like an importfile\nformat.  Reading a GIT pack is really pretty trivial; if someone was\ngoing to write a parser/reader to pull apart a GIT repository and\nuse that information in another way they would just do it against\nthe pack files.  Its really not that much code.  But generating a\npack efficiently for a large volume of data is slightly less trivial;\nthe attempt here is to produce some tool that can take a relatively\ntrivial data stream and produce a reasonable (but not necessarily\nabsolute smallest) pack from it in the least amount of CPU and\ndisk time necessary to do the job.  I would hope that nobody would\nseriously consider dumping a GIT repository back INTO this format!\n\n[snip]\n> AFAIK the svn code doesn't do merge commits. We probably need a post\n> processing pass in the git repo that finds the merges and closes off\n> the branches. gitk won't be pretty with 1,500 open branches. This may\n> need some manual clues.\n\n*wince* 1500 open branches.  Youch.  OK, that answers a lot of\nquestions for me with regards to memory handling in fast-import.\nWhich you provide excellent suggestions for below.  I guess I didn't\nthink you had nearly that many...\n \n[snip]\n> The file names are used over and over. Alloc a giant chunk of memory\n> and keep appending the file name strings to it. Then build a little\n> tree so that you can look up existing names. i.e. turn the files names\n> into atoms. Never delete anything.\n\nAgreed.  For 1500 branches its worth doing.\n \n[snip]\n> About 100,000 files in the initial change set that builds the repo.\n> FInal repo has 120,000 files.\n> \n> There are 1,500 branches. I haven't looked at the svn dump file format\n> for branches, but I suspect that it sends everything on a branch out\n> at once and doesn't intersperse it with the trunk commits.\n\nIf you can tell fast-import your are completely done processing a\nbranch I can recycle the memory I have tied up for that branch; but\nif that's going to be difficult then...  hmm.\n\nRight now I'm looking at around 5 MB/branch, based on implementing\nthe memory handling optimizations you suggested.  That's still *huge*\nfor 1500 branches.  I clearly can't hang onto every branch in memory\nfor the entire life of the import like I was planning on doing.\nI'll kick that around for a couple of hours and see what I come\nup with.\n\n-- \nShawn.\n"},{"id":"25010","messageId":"9e4733910608081656p4bcd68d9xb3e60b58d9afbcc9@mail.gmail.com","threadId":"5153","inReplyTo":"20060808224537.GB18163@spearce.org","subject":"Re: fast-import and unique objects.","fromName":"Jon Smirl","fromEmail":"jonsmirl@gmail.com","sentAt":"2006-08-08T23:56:32Z","receivedAt":"2006-08-08T23:56:32Z","isPatch":false,"sender":{"key":"jonsmirl@gmail.com","avatar":"https://gravatar.com/avatar/cff3bf5bfdfa6708b905712ff91f0f9b8aaca161659f38c02b787920d5d28b7e?d=mp&s=160"},"body":"On 8/8/06, Shawn Pearce <spearce@spearce.org> wrote:\n> Jon Smirl <jonsmirl@gmail.com> wrote:\n> > We're designing a dumpfile format for git like the one SVN has.\n>\n> I'm not sure I'd call it a dumpfile format.  More like an importfile\n> format.  Reading a GIT pack is really pretty trivial; if someone was\n> going to write a parser/reader to pull apart a GIT repository and\n> use that information in another way they would just do it against\n> the pack files.  Its really not that much code.  But generating a\n> pack efficiently for a large volume of data is slightly less trivial;\n> the attempt here is to produce some tool that can take a relatively\n> trivial data stream and produce a reasonable (but not necessarily\n> absolute smallest) pack from it in the least amount of CPU and\n> disk time necessary to do the job.  I would hope that nobody would\n> seriously consider dumping a GIT repository back INTO this format!\n>\n> [snip]\n> > AFAIK the svn code doesn't do merge commits. We probably need a post\n> > processing pass in the git repo that finds the merges and closes off\n> > the branches. gitk won't be pretty with 1,500 open branches. This may\n> > need some manual clues.\n>\n> *wince* 1500 open branches.  Youch.  OK, that answers a lot of\n> questions for me with regards to memory handling in fast-import.\n> Which you provide excellent suggestions for below.  I guess I didn't\n> think you had nearly that many...\n>\n> [snip]\n> > The file names are used over and over. Alloc a giant chunk of memory\n> > and keep appending the file name strings to it. Then build a little\n> > tree so that you can look up existing names. i.e. turn the files names\n> > into atoms. Never delete anything.\n>\n> Agreed.  For 1500 branches its worth doing.\n>\n> [snip]\n> > About 100,000 files in the initial change set that builds the repo.\n> > FInal repo has 120,000 files.\n> >\n> > There are 1,500 branches. I haven't looked at the svn dump file format\n> > for branches, but I suspect that it sends everything on a branch out\n> > at once and doesn't intersperse it with the trunk commits.\n>\n> If you can tell fast-import your are completely done processing a\n> branch I can recycle the memory I have tied up for that branch; but\n> if that's going to be difficult then...  hmm.\n>\n> Right now I'm looking at around 5 MB/branch, based on implementing\n> the memory handling optimizations you suggested.  That's still *huge*\n> for 1500 branches.  I clearly can't hang onto every branch in memory\n> for the entire life of the import like I was planning on doing.\n> I'll kick that around for a couple of hours and see what I come\n> up with.\n\nSome of these branches are what cvs2svn calls unlabeled branches.\ncvs2svn is probably creating more of these than necessary since the\ncode for coalescing them into a single big unlabeled branch is not\nthat good.\n\nI attached the list of branch names being generated.\n\n\n\n>\n> --\n> Shawn.\n>\n\n\n-- \nJon Smirl\njonsmirl@gmail.com\n"}]}