# Re: Errors cloning large repo 2 messages from 2007-03-10 to 2007-03-10. Participants: Anton Tropashko, Linus Torvalds. Thread: https://gitlist.dev/t/7172 ## Anton Tropashko, 2007-03-10 01:21 Subject: Re: Errors cloning large repo Message-ID: <891197.22028.qm@web52611.mail.yahoo.com> URL: https://gitlist.dev/e/891197.22028.qm%40web52611.mail.yahoo.com ``` > I should try it out with some made-up auto-generated directory setup, but > I'm not sure I have the energy to do it ;) but your /usr should be large enough if /usr/local and /usr/local/src are not!!! I don't think you need to generate anything. Or you are saying that the problem is the number of files I have, not the total size of the files? In any event there should be a plenty of files in /usr > That said, it might also be a good idea (regardless of anything else) to > split things up, if only because it's quite possible that not everybody is > interested in having *everything*. Forcing people to work with a 8.5GB > repository when they might not care about it all could be a bad idea. > "git reset --hard" will do it for you. As will "git checkout -f", for that > matter. > "git revert" will just undo an old commit (as you apparently already found > out) Yep. I found checkout -f works before I got the rest alternative. I was pleased that git did not lock me out of committing a few deletions for *.pdf, *.doc and makefiles after repack started. repack -a -d just finished and I started clone again. It's already deltifying at 6%. Thank you. ____________________________________________________________________________________ Now that's room service! Choose from over 150,000 hotels in 45,000 destinations on Yahoo! Travel to find your fit. http://farechase.yahoo.com/promo-generic-14795097 ``` ## Linus Torvalds, 2007-03-10 01:45 Subject: Re: Errors cloning large repo Message-ID: URL: https://gitlist.dev/e/Pine.LNX.4.64.0703091736290.10832%40woody.linux-foundation.org In-Reply-To: <891197.22028.qm@web52611.mail.yahoo.com> ``` On Fri, 9 Mar 2007, Anton Tropashko wrote: > > but your /usr should be large enough if /usr/local and /usr/local/src > are not!!! I don't like the size distribution. My /usr has 181585 files, but is 4.0G in size, which doesn't match yours. Also, I've wanted to generate bogus data for a while, just for testing, so I wrote this silly program that I can tweak the size distribution for. It gives me something that approaches your distribution (I ran it a few times, I now have 110402 files, and 5.7GB of space according to 'du'). It's totally unrealistic wrt packing, though (no deltas, and no compression, since the data itself is all random), and I don't know how to approximate that kind of details samely. I'll need to call it a day for the kids dinner etc, so I'm probably done for the day. I'll play with this a bit more to see if I can find various scalability issues (and just ignore the delta/compression problem - you probably don't have many deltas either, so I'm hoping that the fact that I only have 5.7GB will approximate your data thanks to it not being compressible). Linus --- #include #include #include #include #include #include #include #include /* * Create a file with a random size in the range * 0-1MB, but with a "pink noise"ish distribution * (ie equally many files in the 1-2kB range as in * the half-meg to megabyte range). */ static void create_file(const char *name) { int i; int fd = open(name, O_CREAT | O_WRONLY | O_TRUNC, 0666); static char buffer[1000]; unsigned long size = random() % (1 << (10+(random() % 10))); if (fd < 0) return; for (i = 0; i < sizeof(buffer); i++) buffer[i] = random(); while (size) { int len = sizeof(buffer); if (len > size) len = size; write(fd, buffer, len); size -= len; } close(fd); } static void start(const char *base, float dir_likely, float dir_expand, float end_likely, float end_expand) { int len = strlen(base); char *name = malloc(len + 10); mkdir(base, 0777); memcpy(name, base, len); name[len++] = '/'; dir_likely *= dir_expand; end_likely *= end_expand; for (;;) { float rand = (random() & 65535) / 65536.0; sprintf(name + len, "%ld", random() % 1000000); rand -= dir_likely; if (rand < 0) { start(name, dir_likely, dir_expand, end_likely, end_expand); continue; } rand -= end_likely; if (rand < 0) break; create_file(name); } } int main(int argc, char **argv) { /* * Tune the numbers to your liking.. * * The floats are: * - dir_likely (likelihood of creating a recursive directory) * - dir_expand (how dir_likely behaves as we move down recursively) * - end_likely (likelihood of ending file creation in a directory) * - end_expand (how end_likely behaves as we move down recursively) * * The numbers 0.3/0.6 0.03/1.1 are totally made up, and for me * generate a tree of between a few hundred files and a few tens * of thousands of files. * * Re-run several times to generate more files in the tree. */ srandom(time(NULL)); start("tree", 0.3, 0.6, 0.02, 1.1); return 0; } ```