{"thread":{"id":"7172","subject":"Re: Errors cloning large repo","startedAt":"2007-03-10T01:21:25Z","lastAt":"2007-03-10T01:45:22Z","messageCount":2,"participants":["Anton Tropashko","Linus Torvalds"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"36710","messageId":"891197.22028.qm@web52611.mail.yahoo.com","threadId":"7172","inReplyTo":null,"subject":"Re: Errors cloning large repo","fromName":"Anton Tropashko","fromEmail":"atropashko@yahoo.com","sentAt":"2007-03-10T01:21:25Z","receivedAt":"2007-03-10T01:21:25Z","isPatch":false,"sender":{"key":"atropashko@yahoo.com","avatar":null},"body":"> I should try it out with some made-up auto-generated directory setup, but \n> I'm not sure I have the energy to do it ;)\n\nbut your /usr should be large enough if /usr/local and /usr/local/src are not!!!\nI don't think you need to generate anything.\nOr you are saying that the problem is the number of files I have, not the\ntotal size of the files? In any event there should be a plenty of files in /usr\n\n> That said, it might also be a good idea (regardless of anything else) to \n> split things up, if only because it's quite possible that not everybody is \n> interested in having *everything*. Forcing people to work with a 8.5GB \n> repository when they might not care about it all could be a bad idea.\n\n> \"git reset --hard\" will do it for you. As will \"git checkout -f\", for that \n> matter.\n\n> \"git revert\" will just undo an old commit (as you apparently already found \n> out)\n\nYep. I found checkout -f works before I got the rest alternative.\n\nI was pleased that git did not lock me out of committing a few\ndeletions for *.pdf, *.doc and makefiles after repack started.\nrepack -a -d just finished and I started clone again.\nIt's already deltifying at 6%.\n\nThank you.\n\n\n\n\n\n\n \n____________________________________________________________________________________\nNow that's room service!  Choose from over 150,000 hotels\nin 45,000 destinations on Yahoo! Travel to find your fit.\nhttp://farechase.yahoo.com/promo-generic-14795097\n"},{"id":"36713","messageId":"Pine.LNX.4.64.0703091736290.10832@woody.linux-foundation.org","threadId":"7172","inReplyTo":"891197.22028.qm@web52611.mail.yahoo.com","subject":"Re: Errors cloning large repo","fromName":"Linus Torvalds","fromEmail":"torvalds@linux-foundation.org","sentAt":"2007-03-10T01:45:22Z","receivedAt":"2007-03-10T01:45:22Z","isPatch":false,"sender":{"key":"torvalds@linux-foundation.org","avatar":"https://avatars.githubusercontent.com/u/1024025?v=4"},"body":"\n\nOn Fri, 9 Mar 2007, Anton Tropashko wrote:\n> \n> but your /usr should be large enough if /usr/local and /usr/local/src \n> are not!!!\n\nI don't like the size distribution.\n\nMy /usr has 181585 files, but is 4.0G in size, which doesn't match yours. \nAlso, I've wanted to generate bogus data for a while, just for testing, so \nI wrote this silly program that I can tweak the size distribution for.\n\nIt gives me something that approaches your distribution (I ran it a few \ntimes, I now have 110402 files, and 5.7GB of space according to 'du').\n\nIt's totally unrealistic wrt packing, though (no deltas, and no \ncompression, since the data itself is all random), and I don't know how to \napproximate that kind of details samely.\n\nI'll need to call it a day for the kids dinner etc, so I'm probably done \nfor the day. I'll play with this a bit more to see if I can find various \nscalability issues (and just ignore the delta/compression problem - you \nprobably don't have many deltas either, so I'm hoping that the fact \nthat I only have 5.7GB will approximate your data thanks to it not being \ncompressible).\n\n\t\tLinus\n\n---\n#include <time.h>\n#include <stdio.h>\n#include <unistd.h>\n#include <stdlib.h>\n#include <string.h>\n#include <sys/stat.h>\n#include <sys/types.h>\n#include <sys/fcntl.h>\n\n/*\n * Create a file with a random size in the range\n * 0-1MB, but with a \"pink noise\"ish distribution\n * (ie equally many files in the 1-2kB range as in\n * the half-meg to megabyte range).\n */\nstatic void create_file(const char *name)\n{\n\tint i;\n\tint fd = open(name, O_CREAT | O_WRONLY | O_TRUNC, 0666);\n\tstatic char buffer[1000];\n\tunsigned long size = random() % (1 << (10+(random() % 10)));\n\n\tif (fd < 0)\n\t\treturn;\n\tfor (i = 0; i < sizeof(buffer); i++)\n\t\tbuffer[i] = random();\n\twhile (size) {\n\t\tint len = sizeof(buffer);\n\t\tif (len > size)\n\t\t\tlen = size;\n\t\twrite(fd, buffer, len);\n\t\tsize -= len;\n\t}\n\tclose(fd);\n}\n\nstatic void start(const char *base,\n\tfloat dir_likely, float dir_expand,\n\tfloat end_likely, float end_expand)\n{\n\tint len = strlen(base);\n\tchar *name = malloc(len + 10);\n\n\tmkdir(base, 0777);\n\n\tmemcpy(name, base, len);\n\tname[len++] = '/';\n\n\tdir_likely *= dir_expand;\n\tend_likely *= end_expand;\n\n\tfor (;;) {\n\t\tfloat rand = (random() & 65535) / 65536.0;\n\n\t\tsprintf(name + len, \"%ld\", random() % 1000000);\n\t\trand -= dir_likely;\n\t\tif (rand < 0) {\n\t\t\tstart(name, dir_likely, dir_expand, end_likely, end_expand);\n\t\t\tcontinue;\n\t\t}\n\t\trand -= end_likely;\n\t\tif (rand < 0)\n\t\t\tbreak;\n\t\tcreate_file(name);\n\t}\n}\n\nint main(int argc, char **argv)\n{\n\t/*\n\t * Tune the numbers to your liking..\n\t *\n\t * The floats are:\n\t *  - dir_likely (likelihood of creating a recursive directory)\n\t *  - dir_expand (how dir_likely behaves as we move down recursively)\n\t *  - end_likely (likelihood of ending file creation in a directory)\n\t *  - end_expand (how end_likely behaves as we move down recursively)\n\t *\n\t * The numbers 0.3/0.6 0.03/1.1 are totally made up, and for me\n\t * generate a tree of between a few hundred files and a few tens \n\t * of thousands of files.\n\t *\n\t * Re-run several times to generate more files in the tree.\n\t */\n\tsrandom(time(NULL));\n\tstart(\"tree\",\n\t\t0.3, 0.6,\n\t\t0.02, 1.1);\n\treturn 0;\n}\n"}]}