{"thread":{"id":"30019","subject":"GSOC on designing a faster index format","startedAt":"2012-03-21T21:39:45Z","lastAt":"2012-03-21T22:20:58Z","messageCount":2,"participants":["Calvin Deutschbein","Thomas Rast"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"187437","messageId":"CAAQUJYbr+RbcpRzKqK+s+9sXezQuFR-S=OdSdnTYTtURDqKRhQ@mail.gmail.com","threadId":"30019","inReplyTo":null,"subject":"GSOC on designing a faster index format","fromName":"Calvin Deutschbein","fromEmail":"deutschbeinc@gmail.com","sentAt":"2012-03-21T21:39:45Z","receivedAt":"2012-03-21T21:39:45Z","isPatch":false,"sender":{"key":"deutschbeinc@gmail.com","avatar":null},"body":"Hi!\n\nI'm a first year CS major (mostly proficient in C and Racket) at\nUChicago looking at GSOC and GIT caught my interest, especially on the\nidea of designing a faster index format.  As the GSOC page recommended\nstarting a discussion in the mailing thread, I thought I would do so\non this particular topic.\n\n\"Git is pretty slow when managing huge repositories in terms of files\nin any given tree, as it needs to rewrite the index (in full) on\npretty much every operation. For example, even though logically git\nadd already_tracked_file only changes a single blob SHA-1 in the\nindex, Git will verify index correctness during loading and recompute\nthe new hash during writing over the whole index. It thus ends up\nspending a large amount of time simply on hashing the index.\"\n\nDoing anything slowly is never fun, especially so as scale increases.\nDealing with version control for some of my course work, I appreciate\nthe speed possible with smaller projects and imagine on larger\nprojects the time saved would be quite lucrative.  That was what\ninitially attracted me to this idea, beyond the general desire to\nmaximize efficiency.  However, writing over the entire index with\nevery operation seems like a strange way spend resources.  Verifying\ncorrectness, it would seem, can generally be done without a rewrite\nand isn't necessarily exhaustively necessary with every index edit if\nefficiency is becoming of utmost concern.  This then, would seem to\nindicate that hashing the index wouldn't always be necessary.  Is it\ndone then just for 100% security in index correctness?\n\nI also appreciate the later mention of the importance of being as easy\nto parse as possible, as I tend to find such coding an intellectually\nengaging exercise and this represents a valid reason to invest\nresources in such a pursuit.  Unfortunately, I'm not entirely familiar\nwith other git-reading programs.  Is there a specific set of such\nprograms I should look into?\n\nThank you,\nCalvin Deutschbein\n"},{"id":"187439","messageId":"87y5qtmw8l.fsf@thomas.inf.ethz.ch","threadId":"30019","inReplyTo":"CAAQUJYbr+RbcpRzKqK+s+9sXezQuFR-S=OdSdnTYTtURDqKRhQ@mail.gmail.com","subject":"Re: GSOC on designing a faster index format","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2012-03-21T22:20:58Z","receivedAt":"2012-03-21T22:20:58Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Hi Calvin,\n\nPlease also read/follow the other threads on the same topic\n\n  http://thread.gmane.org/gmane.comp.version-control.git/193541\n  http://thread.gmane.org/gmane.comp.version-control.git/193550\n\nCalvin Deutschbein <deutschbeinc@gmail.com> writes:\n\n> Verifying\n> correctness, it would seem, can generally be done without a rewrite\n> and isn't necessarily exhaustively necessary with every index edit if\n> efficiency is becoming of utmost concern.  This then, would seem to\n> indicate that hashing the index wouldn't always be necessary.  Is it\n> done then just for 100% security in index correctness?\n\nIt's basically the *easiest* way to make sure the index was not\ncorrupted.  It also happens to be plenty fast enough for not-so-big\nprojects.\n\n> I also appreciate the later mention of the importance of being as easy\n> to parse as possible, as I tend to find such coding an intellectually\n> engaging exercise and this represents a valid reason to invest\n> resources in such a pursuit.  Unfortunately, I'm not entirely familiar\n> with other git-reading programs.  Is there a specific set of such\n> programs I should look into?\n\nAt least jgit and libgit2, the former because it represents a large user\nbase (more or less everybody using Eclipse and/or Gerrit) and the latter\nbecause their life is already hard enough with the current git internals\n;-)\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"}]}