{"thread":{"id":"31038","subject":"[GSoC] Designing a faster index format - Progress report week 13","startedAt":"2012-07-16T20:33:00Z","lastAt":"2012-07-25T06:44:10Z","messageCount":12,"participants":["Thomas Gummerer","Junio C Hamano","Robin Rosenberg","Thomas Rast"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"195166","messageId":"20120716203300.GA1849@tgummerer.surfnet.iacbox","threadId":"31038","inReplyTo":null,"subject":"[GSoC] Designing a faster index format - Progress report week 13","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-07-16T20:33:00Z","receivedAt":"2012-07-16T20:33:00Z","isPatch":false,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"== Work done in the previous 12 weeks ==\n\n- Definition of a tentative index file v5 format [1]. This differs\n from the proposal in making it possible to bisect the directory\n entries and file entries, to do a binary search. The exact bits\n for each section were also defined. To further compress the index,\n along with prefix compression, the stat data is hashed, since\n it's only used for comparison, but the plain data is never used.\n Thanks to Michael Haggerty, Nguyen Thai Ngoc Duy, Thomas Rast\n and Robin Rosenberg for feedback.\n- Prototype of a converter from the index format v2/v3 to the index\n format v5. [2] The converter reads the index from a git repository,\n can output parts of the index (header, index entries as in\n git ls-files --debug, cache tree as in test-dump-cache-tree, or\n the reuc data). Then it writes the v5 index file format to\n .git/index-v5. Thanks to Michael Haggerty for the code review.\n- Prototype of a reader for the new index file format. [3] The\n reader has mainly the purpose to show the algorithm used to read\n the index lexicographically sorted after the full name which is\n required by the current internal memory format. Big thanks for\n reviewing this code and giving me advice on refactoring goes\n to Michael Haggerty.\n- Read the index format format and translate it to the current in\n memory format. This doesn't include reading any of the current\n extensions, which are now part of the main index. The code again\n is on github. [4] Thanks for reviewing the first steps to Thomas\n Rast.\n- Read the cache-tree data (formerly an extension, now it's integrated\n with the rest of the directory data) from the new ondisk format.\n There are still a few optimizations to do in this algorithm.\n- Started implementing the API (suggested by Duy), but it's still\n in the very early stages. There is one commit for this on GitHub [1],\n but it's a very early work in progress.\n- Started implementing the writer, which extracts the directories from\n the in-memory format, and writes the header and the directories to\n disk.\n- I found a few bugs in the algorithm for extracting the directories\n and decided to completely rewrite it, using a hash table instead of\n simple lists, since the old one would have to many corner cases to\n handle.\n- Implemented writing the file block to disk, and basic tests from the\n test suite are running fine, not including tests that require\n conflicted data or the cache-tree to work, which both are not\n implemented yet.\n- Started implementing a patch to introduce a ce_namelen field in\n struct cache_entry and drop the name length from the flags. [5]\n\n== Work done int the last week ==\n\n- Polished the patch for the ce_namelen field. The thread for the\n patch can be found at [5]. Again thanks to Junio, Duy and Thomas\n for reviewing it and giving me suggestions for improving it.\n- Implemented the cache-tree and conflict data writing to the\n index-v5 file.\n \n== Outlook for the next week ==\n\n- There are still a few bugs in the conflict writing, which will\n be fixed, to make the test suite pass with index-v5.\n- Once the test suite passes, the code still needs to be refactored\n and optimized.\n- If the two points above go well, I'll continue working on the api\n that Duy suggested.\n\n[1] https://github.com/tgummerer/git/wiki/Index-file-format-v5\n[2] https://github.com/tgummerer/git/blob/pythonprototype/git-convert-index.py\n[3] https://github.com/tgummerer/git/blob/pythonprototype/git-read-index-v5.py\n[4] https://github.com/tgummerer/git/tree/index-v5\n[5] http://thread.gmane.org/gmane.comp.version-control.git/200997\n"},{"id":"195173","messageId":"7vwr23zb65.fsf@alter.siamese.dyndns.org","threadId":"31038","inReplyTo":"20120716203300.GA1849@tgummerer.surfnet.iacbox","subject":"Re: [GSoC] Designing a faster index format - Progress report week 13","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-07-16T20:46:42Z","receivedAt":"2012-07-16T20:46:42Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Gummerer writes:\n\n> == Work done in the previous 12 weeks ==\n>\n> - Definition of a tentative index file v5 format [1]. This differs\n> from the proposal in making it possible to bisect the directory\n> entries and file entries, to do a binary search. The exact bits\n> for each section were also defined. To further compress the index,\n> along with prefix compression, the stat data is hashed, since\n> it's only used for comparison, but the plain data is never used.\n\ns/comparison/equality comparison/ perhaps?\n\n> Thanks to Michael Haggerty, Nguyen Thai Ngoc Duy, Thomas Rast\n> and Robin Rosenberg for feedback.\n\n> - Read the index format format and translate it to the current in\n\ns/format format/on-disk file format/ or something?\n\n> memory format. This doesn't include reading any of the current\n> extensions, which are now part of the main index. The code again\n> is on github. [4] Thanks for reviewing the first steps to Thomas\n> Rast.\n\n> - Started implementing the writer, which extracts the directories from\n> the in-memory format, and writes the header and the directories to\n> disk.\n> - I found a few bugs in the algorithm for extracting the directories\n> and decided to completely rewrite it, using a hash table instead of\n> simple lists, since the old one would have to many corner cases to\n> handle.\n\nWhat does \"the algorithm\" refer to? Is it the one described in the\nprevious bullet point, or is it the code in production? If latter,\nit would help to separate out the task to fix the breakage, as\npeople with the current or previous versions of Git will be\nnegatively affected until that bug is fixed. If former, I am not\nsure if this task needs to be described in two bullet points (\"I did\nX, X had bug so I redid X in a different way\" is still a single task\nto do X).\n\n> == Work done int the last week ==\n>\n> - Polished the patch for the ce_namelen field. The thread for the\n> patch can be found at [5].\n\nThanks for this one; I think it is ready for 'next', but if you are\nstill not satisfied I do not mind waiting for further perfection.\n"},{"id":"195191","messageId":"20120717082130.GB1849@tgummerer.surfnet.iacbox","threadId":"31038","inReplyTo":"7vwr23zb65.fsf@alter.siamese.dyndns.org","subject":"Re: [GSoC] Designing a faster index format - Progress report week 13","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-07-17T08:21:30Z","receivedAt":"2012-07-17T08:21:30Z","isPatch":false,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"\n\nOn 07/16, Junio C Hamano wrote:\n> Thomas Gummerer writes:\n> \n> > == Work done in the previous 12 weeks ==\n> >\n> > - Definition of a tentative index file v5 format [1]. This differs\n> > from the proposal in making it possible to bisect the directory\n> > entries and file entries, to do a binary search. The exact bits\n> > for each section were also defined. To further compress the index,\n> > along with prefix compression, the stat data is hashed, since\n> > it's only used for comparison, but the plain data is never used.\n> \n> s/comparison/equality comparison/ perhaps?\n> \n\nExactly, thanks.\n\n> > Thanks to Michael Haggerty, Nguyen Thai Ngoc Duy, Thomas Rast\n> > and Robin Rosenberg for feedback.\n> \n> > - Read the index format format and translate it to the current in\n> \n> s/format format/on-disk file format/ or something?\n>\n\nYes, thanks.\n\n> > memory format. This doesn't include reading any of the current\n> > extensions, which are now part of the main index. The code again\n> > is on github. [4] Thanks for reviewing the first steps to Thomas\n> > Rast.\n> \n> > - Started implementing the writer, which extracts the directories from\n> > the in-memory format, and writes the header and the directories to\n> > disk.\n> > - I found a few bugs in the algorithm for extracting the directories\n> > and decided to completely rewrite it, using a hash table instead of\n> > simple lists, since the old one would have to many corner cases to\n> > handle.\n> \n> What does \"the algorithm\" refer to? Is it the one described in the\n> previous bullet point, or is it the code in production? If latter,\n> it would help to separate out the task to fix the breakage, as\n> people with the current or previous versions of Git will be\n> negatively affected until that bug is fixed. If former, I am not\n> sure if this task needs to be described in two bullet points (\"I did\n> X, X had bug so I redid X in a different way\" is still a single task\n> to do X).\n\nIt refers to the algorithm in the previous bullet point, which\nextracts the directories, and can be included in the above bullet\npoint. Sorry for the confusion.\n\n> > == Work done int the last week ==\n> >\n> > - Polished the patch for the ce_namelen field. The thread for the\n> > patch can be found at [5].\n> \n> Thanks for this one; I think it is ready for 'next', but if you are\n> still not satisfied I do not mind waiting for further perfection.\n\nThanks, I'm satisfied with it, for me it can be merged to 'next'.\n"},{"id":"195192","messageId":"20120717082452.GC1849@tgummerer.surfnet.iacbox","threadId":"31038","inReplyTo":"7vwr23zb65.fsf@alter.siamese.dyndns.org","subject":"Re: [GSoC] Designing a faster index format - Progress report week 13","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2012-07-17T08:24:52Z","receivedAt":"2012-07-17T08:24:52Z","isPatch":false,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"Thanks Junio for reading the progress report, this is just\ncorrected version without the errors that he pointed out.\n\n== Work done in the previous 12 weeks ==\n\n- Definition of a tentative index file v5 format [1]. This differs\n from the proposal in making it possible to bisect the directory\n entries and file entries, to do a binary search. The exact bits\n for each section were also defined. To further compress the index,\n along with prefix compression, the stat data is hashed, since\n it's only used for equality comparison, but the plain data is\n never used.\n Thanks to Michael Haggerty, Nguyen Thai Ngoc Duy, Thomas Rast\n and Robin Rosenberg for feedback.\n- Prototype of a converter from the index format v2/v3 to the index\n format v5. [2] The converter reads the index from a git repository,\n can output parts of the index (header, index entries as in\n git ls-files --debug, cache tree as in test-dump-cache-tree, or\n the reuc data). Then it writes the v5 index file format to\n .git/index-v5. Thanks to Michael Haggerty for the code review.\n- Prototype of a reader for the new index file format. [3] The\n reader has mainly the purpose to show the algorithm used to read\n the index lexicographically sorted after the full name which is\n required by the current internal memory format. Big thanks for\n reviewing this code and giving me advice on refactoring goes\n to Michael Haggerty.\n- Read the on-disk index file format and translate it to the current\n in memory format. This doesn't include reading any of the current\n extensions, which are now part of the main index. The code again\n is on github. [4] Thanks for reviewing the first steps to Thomas\n Rast.\n- Read the cache-tree data (formerly an extension, now it's integrated\n with the rest of the directory data) from the new ondisk format.\n There are still a few optimizations to do in this algorithm.\n- Started implementing the API (suggested by Duy), but it's still\n in the very early stages. There is one commit for this on GitHub [1],\n but it's a very early work in progress.\n- Started implementing the writer, which extracts the directories from\n the in-memory format, and writes the header and the directories to\n disk. The algorithm uses a hash-table instead of a simple list,\n to avoid many corner cases.\n- Implemented writing the file block to disk, and basic tests from the\n test suite are running fine, not including tests that require\n conflicted data or the cache-tree to work, which both are not\n implemented yet.\n- Started implementing a patch to introduce a ce_namelen field in\n struct cache_entry and drop the name length from the flags. [5]\n\n== Work done int the last week ==\n\n- Polished the patch for the ce_namelen field. The thread for the\n patch can be found at [5]. Again thanks to Junio, Duy and Thomas\n for reviewing it and giving me suggestions for improving it.\n- Implemented the cache-tree and conflict data writing to the\n index-v5 file.\n \n== Outlook for the next week ==\n\n- There are still a few bugs in the conflict writing, which will\n be fixed, to make the test suite pass with index-v5.\n- Once the test suite passes, the code still needs to be refactored\n and optimized.\n- If the two points above go well, I'll continue working on the api\n that Duy suggested.\n\n[1] https://github.com/tgummerer/git/wiki/Index-file-format-v5\n[2] https://github.com/tgummerer/git/blob/pythonprototype/git-convert-index.py\n[3] https://github.com/tgummerer/git/blob/pythonprototype/git-read-index-v5.py\n[4] https://github.com/tgummerer/git/tree/index-v5\n[5] http://thread.gmane.org/gmane.comp.version-control.git/200997\n"},{"id":"195445","messageId":"500C1AA9.4000306@dewire.com","threadId":"31038","inReplyTo":"20120717082452.GC1849@tgummerer.surfnet.iacbox","subject":"Re: [GSoC] Designing a faster index format - Progress report week 13","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2012-07-22T15:22:17Z","receivedAt":"2012-07-22T15:22:17Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"\nA note on how JGit would work here. Java has none of the fields\nthat constitute statcrc. I guess we would write zero here when\ncreating new entries. Git could recognize that when checking status and \nsimply assume \"clean\" unless mtime or st_size says otherwise.\n\nFor existing entries JGit could either keep the old value (which is\nprobably a lie since) or zero it.\n\nA modification to the spec would be that 0 == not set.\n\n-- robin\n"},{"id":"195449","messageId":"7vfw8jsk5o.fsf@alter.siamese.dyndns.org","threadId":"31038","inReplyTo":"500C1AA9.4000306@dewire.com","subject":"Re: [GSoC] Designing a faster index format - Progress report week 13","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-07-22T18:52:35Z","receivedAt":"2012-07-22T18:52:35Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Robin Rosenberg writes:\n\n> A note on how JGit would work here. Java has none of the fields\n> that constitute statcrc. I guess we would write zero here when\n> creating new entries. Git could recognize that when checking status\n> and simply assume \"clean\" unless mtime or st_size says otherwise.\n\nEven though it may not be the end of the world, that is certainly\nbad. Recording the constituent fields separately without the statcrc\nmicrooptimization, thereby not shaving a handful of bytes per the\nindex entry, is not the end of the world either in the same sense,\nwhich leads us to question the benefit we would be getting from such\na change.\n"},{"id":"195450","messageId":"87629fvaxz.fsf@thomas.inf.ethz.ch","threadId":"31038","inReplyTo":"7vfw8jsk5o.fsf@alter.siamese.dyndns.org","subject":"Re: [GSoC] Designing a faster index format - Progress report week 13","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2012-07-22T19:43:20Z","receivedAt":"2012-07-22T19:43:20Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Junio C Hamano writes:\n\n> Robin Rosenberg writes:\n>\n>> A note on how JGit would work here. Java has none of the fields\n>> that constitute statcrc. I guess we would write zero here when\n>> creating new entries. Git could recognize that when checking status\n>> and simply assume \"clean\" unless mtime or st_size says otherwise.\n>\n> Even though it may not be the end of the world, that is certainly\n> bad. Recording the constituent fields separately without the statcrc\n> microoptimization, thereby not shaving a handful of bytes per the\n> index entry, is not the end of the world either in the same sense,\n> which leads us to question the benefit we would be getting from such\n> a change.\n\nHum, I'm a bit lost now.\n\nWhat is the status quo? I take it JGit does not have any of ctime, dev,\nino etc., and either leaves the existing value or puts a 0. Which is\nnot different from either leaving the stat crc in place, or putting a 0.\nExcept that IIUC, putting a 0 in both cases means forcing a refresh once\nC git comes along (or some other reader that knows about the fields).\n\nSo if we want to keep the safety net, a magic \"I don't know\" value would\nindeed be a good idea. But I don't see how what Robin said constitutes\nan argument in favor of splitting stat_crc into its fields again?\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"195459","messageId":"7vr4s3pkqr.fsf@alter.siamese.dyndns.org","threadId":"31038","inReplyTo":"87629fvaxz.fsf@thomas.inf.ethz.ch","subject":"Re: [GSoC] Designing a faster index format - Progress report week 13","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-07-22T21:08:12Z","receivedAt":"2012-07-22T21:08:12Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Rast writes:\n\n> Hum, I'm a bit lost now.\n>\n> What is the status quo? I take it JGit does not have any of ctime, dev,\n> ino etc., and either leaves the existing value or puts a 0....\n> an argument in favor of splitting stat_crc into its fields again?\n\nA difference is that JGit already has such code, and we would be\nadding a burden to do so yet again. It also may not just be JGit,\nbut anything that wants to be \"compatible\" with systems whose\nfilesystem interface does not give enough data by omitting fields\nthe current index pays attention to.\n\nIt isn't really a discussion about splitting again, but more about\nnot squishing them into a new field in the first place---IIUC, even\noutside Windows, ctime is already problematic on some systems where\nbackground processes muck with extended attributes Git does not pay\nattention to. If the patch makes us lose the ability to selectively\nignore changes to certain fields (e.g. changes to dev and ino are\nnoticed but ctime are ignored) by squishing them into one new field,\nwouldn't removing them without adding such a useless field a simpler\nway to go?\n"},{"id":"195587","messageId":"500DD05D.8030708@dewire.com","threadId":"31038","inReplyTo":"7vr4s3pkqr.fsf@alter.siamese.dyndns.org","subject":"Re: [GSoC] Designing a faster index format - Progress report week 13","fromName":"Robin Rosenberg","fromEmail":"robin.rosenberg@dewire.com","sentAt":"2012-07-23T22:29:49Z","receivedAt":"2012-07-23T22:29:49Z","isPatch":false,"sender":{"key":"robin.rosenberg@dewire.com","avatar":"https://avatars.githubusercontent.com/u/46357?v=4"},"body":"Junio C Hamano skrev 2012-07-22 23.08:\n> Thomas Rast writes:\n>\n>> Hum, I'm a bit lost now.\n>>\n>> What is the status quo? I take it JGit does not have any of ctime, dev,\n>> ino etc., and either leaves the existing value or puts a 0....\n>> an argument in favor of splitting stat_crc into its fields again?\n>\n> A difference is that JGit already has such code, and we would be\n> adding a burden to do so yet again. It also may not just be JGit,\n> but anything that wants to be \"compatible\" with systems whose\n> filesystem interface does not give enough data by omitting fields\n> the current index pays attention to.\n>\n> It isn't really a discussion about splitting again, but more about\n> not squishing them into a new field in the first place---IIUC, even\n> outside Windows, ctime is already problematic on some systems where\n> background processes muck with extended attributes Git does not pay\n> attention to. If the patch makes us lose the ability to selectively\n> ignore changes to certain fields (e.g. changes to dev and ino are\n> noticed but ctime are ignored) by squishing them into one new field,\n> wouldn't removing them without adding such a useless field a simpler\n> way to go?\n\nI wasnt't thinking of splitting, but now I read it again, I do think\nit should split. Having size accessible is a good thing, and even better \nif it a 64-bit value so we don't have the modulo-4G problem when looking \nat it. Current size is 4G + 33 bytes, index says 33. Did the\nfile change or not?\n\nHaving access to size make the need for actually\ninvoking the racy git logic and comparing file content less likely.\n\nAs for ctime it is accessible in Java7, though everyone aren't using it \nand JGit code has to run on Java5. An idea is to make an optional \ncomponent, but that doesn't make ctime available everywhere.\n\n-- robin\n"},{"id":"195620","messageId":"87d33lqsrp.fsf@thomas.inf.ethz.ch","threadId":"31038","inReplyTo":"500DD05D.8030708@dewire.com","subject":"Re: [GSoC] Designing a faster index format - Progress report week 13","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2012-07-24T11:54:02Z","receivedAt":"2012-07-24T11:54:02Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Robin Rosenberg writes:\n\n> Junio C Hamano skrev 2012-07-22 23.08:\n>> Thomas Rast writes:\n>>\n>>> What is the status quo? I take it JGit does not have any of ctime, dev,\n>>> ino etc., and either leaves the existing value or puts a 0....\n>>> an argument in favor of splitting stat_crc into its fields again?\n>>\n>> A difference is that JGit already has such code, and we would be\n>> adding a burden to do so yet again. It also may not just be JGit,\n>> but anything that wants to be \"compatible\" with systems whose\n>> filesystem interface does not give enough data by omitting fields\n>> the current index pays attention to.\n>>\n>> It isn't really a discussion about splitting again, but more about\n>> not squishing them into a new field in the first place---IIUC, even\n>> outside Windows, ctime is already problematic on some systems where\n>> background processes muck with extended attributes Git does not pay\n>> attention to. If the patch makes us lose the ability to selectively\n>> ignore changes to certain fields (e.g. changes to dev and ino are\n>> noticed but ctime are ignored) by squishing them into one new field,\n>> wouldn't removing them without adding such a useless field a simpler\n>> way to go?\n> \n> I wasnt't thinking of splitting, but now I read it again, I do think\n> it should split.\n\nAren't you two going off in different directions? I read Junio as\nimplying that if size/ctime/dev/ino are a pain to deal with, we should\njust drop them altogether. You seem to be saying the opposite:\n\n> Having size accessible is a good thing, and even\n> better if it a 64-bit value so we don't have the modulo-4G problem\n> when looking at it. Current size is 4G + 33 bytes, index says 33. Did\n> the\n> file change or not?\n>\n> Having access to size make the need for actually\n> invoking the racy git logic and comparing file content less likely.\n\nI don't think this is true. Racy git logic is needed every time that\nthe file *looks* unchanged, but isn't. In the case where the file is\ncertified (by mtime) unchanged, we don't go checking. But in the case\nwhere it *looks* changed, we still have to go and read it to know if,\nperhaps, the only thing the user did was hit \"save\" again.\n\nNot to mention that this really hurts in terms of index size. Our\nbenchmark for index-v5 is the Webkit project, which stands at 180k\nfiles. So every 6B/entry is about an MB of final size, which needs to\nbe loaded, hashed (or crc'd), then hashed/crc'd again and written.\nJunio's index-v4 was a speed boost mainly because it cuts down on the\nsize of the index. Do we want to throw that out?\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"},{"id":"195636","messageId":"7vy5m9f65b.fsf@alter.siamese.dyndns.org","threadId":"31038","inReplyTo":"87d33lqsrp.fsf@thomas.inf.ethz.ch","subject":"Re: [GSoC] Designing a faster index format - Progress report week 13","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2012-07-24T16:58:08Z","receivedAt":"2012-07-24T16:58:08Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thomas Rast writes:\n\n> Junio's index-v4 was a speed boost mainly because it cuts down on the\n> size of the index. Do we want to throw that out?\n\nThat's pretty much orthogonal, isn't it?\n\nThe index-v4 is merely to show how a stupid prefix compression of\npathnames without nothing else would reduce the file size and I/O\ncost, in order to set the bar for anything more clever than that.\n\nI thought that this discussion is about keeping, squishing, or\ndiscarding part of the cached stat info, and nobody is suggesting\nwhat to do with the prefix compression of pathnames.\n"},{"id":"195722","messageId":"87mx2omjb9.fsf@thomas.inf.ethz.ch","threadId":"31038","inReplyTo":"7vy5m9f65b.fsf@alter.siamese.dyndns.org","subject":"Re: [GSoC] Designing a faster index format - Progress report week 13","fromName":"Thomas Rast","fromEmail":"trast@student.ethz.ch","sentAt":"2012-07-25T06:44:10Z","receivedAt":"2012-07-25T06:44:10Z","isPatch":false,"sender":{"key":"tr@thomasrast.ch","avatar":"https://avatars.githubusercontent.com/u/153510?v=4"},"body":"Junio C Hamano writes:\n\n> Thomas Rast writes:\n>\n>> Junio's index-v4 was a speed boost mainly because it cuts down on the\n>> size of the index. Do we want to throw that out?\n>\n> That's pretty much orthogonal, isn't it?\n>\n> The index-v4 is merely to show how a stupid prefix compression of\n> pathnames without nothing else would reduce the file size and I/O\n> cost, in order to set the bar for anything more clever than that.\n>\n> I thought that this discussion is about keeping, squishing, or\n> discarding part of the cached stat info, and nobody is suggesting\n> what to do with the prefix compression of pathnames.\n\nTrue, sorry for being so confusing. 'Throw \"that\" out' was meant to\nrefer to the observation that smaller indexes are generally faster.\n\nWhether this matters in the long run is another question. Perhaps\npartial loading combined with something like inotify can avoid full\nreads in most operations.\n\n-- \nThomas Rast\ntrast@{inf,student}.ethz.ch\n"}]}