# GSOC on designing a faster index format 2 messages from 2012-03-21 to 2012-03-21. Participants: Calvin Deutschbein, Thomas Rast. Thread: https://gitlist.dev/t/30019 ## Calvin Deutschbein, 2012-03-21 21:39 Subject: GSOC on designing a faster index format Message-ID: URL: https://gitlist.dev/e/CAAQUJYbr%2BRbcpRzKqK%2Bs%2B9sXezQuFR-S%3DOdSdnTYTtURDqKRhQ%40mail.gmail.com ``` Hi! I'm a first year CS major (mostly proficient in C and Racket) at UChicago looking at GSOC and GIT caught my interest, especially on the idea of designing a faster index format.  As the GSOC page recommended starting a discussion in the mailing thread, I thought I would do so on this particular topic. "Git is pretty slow when managing huge repositories in terms of files in any given tree, as it needs to rewrite the index (in full) on pretty much every operation. For example, even though logically git add already_tracked_file only changes a single blob SHA-1 in the index, Git will verify index correctness during loading and recompute the new hash during writing over the whole index. It thus ends up spending a large amount of time simply on hashing the index." Doing anything slowly is never fun, especially so as scale increases. Dealing with version control for some of my course work, I appreciate the speed possible with smaller projects and imagine on larger projects the time saved would be quite lucrative.  That was what initially attracted me to this idea, beyond the general desire to maximize efficiency.  However, writing over the entire index with every operation seems like a strange way spend resources.  Verifying correctness, it would seem, can generally be done without a rewrite and isn't necessarily exhaustively necessary with every index edit if efficiency is becoming of utmost concern.  This then, would seem to indicate that hashing the index wouldn't always be necessary.  Is it done then just for 100% security in index correctness? I also appreciate the later mention of the importance of being as easy to parse as possible, as I tend to find such coding an intellectually engaging exercise and this represents a valid reason to invest resources in such a pursuit. Unfortunately, I'm not entirely familiar with other git-reading programs. Is there a specific set of such programs I should look into? Thank you, Calvin Deutschbein ``` ## Thomas Rast, 2012-03-21 22:20 Subject: Re: GSOC on designing a faster index format Message-ID: <87y5qtmw8l.fsf@thomas.inf.ethz.ch> URL: https://gitlist.dev/e/87y5qtmw8l.fsf%40thomas.inf.ethz.ch In-Reply-To: ``` Hi Calvin, Please also read/follow the other threads on the same topic http://thread.gmane.org/gmane.comp.version-control.git/193541 http://thread.gmane.org/gmane.comp.version-control.git/193550 Calvin Deutschbein writes: > Verifying > correctness, it would seem, can generally be done without a rewrite > and isn't necessarily exhaustively necessary with every index edit if > efficiency is becoming of utmost concern.  This then, would seem to > indicate that hashing the index wouldn't always be necessary.  Is it > done then just for 100% security in index correctness? It's basically the *easiest* way to make sure the index was not corrupted. It also happens to be plenty fast enough for not-so-big projects. > I also appreciate the later mention of the importance of being as easy > to parse as possible, as I tend to find such coding an intellectually > engaging exercise and this represents a valid reason to invest > resources in such a pursuit. Unfortunately, I'm not entirely familiar > with other git-reading programs. Is there a specific set of such > programs I should look into? At least jgit and libgit2, the former because it represents a large user base (more or less everybody using Eclipse and/or Gerrit) and the latter because their life is already hard enough with the current git internals ;-) -- Thomas Rast trast@{inf,student}.ethz.ch ```