{"thread":{"id":"8326","subject":"Re: [PATCH] improve delta long block matching with big files","startedAt":"2007-05-26T15:19:09Z","lastAt":"2007-05-26T15:19:09Z","messageCount":1,"participants":["Martin Koegler"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"43351","messageId":"20070526151909.GA9429@auto.tuwien.ac.at","threadId":"8326","inReplyTo":null,"subject":"Re: [PATCH] improve delta long block matching with big files","fromName":"Martin Koegler","fromEmail":"mkoegler@auto.tuwien.ac.at","sentAt":"2007-05-26T15:19:09Z","receivedAt":"2007-05-26T15:19:09Z","isPatch":true,"sender":{"key":"mkoegler@auto.tuwien.ac.at","avatar":null},"body":"Nicolas Pitre wrote:\n> Martin Koegler noted that create_delta() performs a new hash lookup\n> after every block copy encoding which are currently limited to 64KB.\n> \n> In case of larger identical blocks, the next hash lookup would normally\n> point to the next 64KB block in the reference buffer and multiple block\n> copy operations will be consecutively encoded.\n> \n> It is however possible that the reference buffer be sparsely indexed if\n> hash buckets have been trimmed down in create_delta_index() when hashing\n> of the reference buffer isn't well balanced.  In that case the hash\n> lookup following a block copy might fail to match anything and the fact\n> that the reference buffer still matches beyond the previous 64KB block\n> will be missed.\n> \n> Let's rework the code so that buffer comparison isn't bounded to 64KB\n> anymore.  The match size should be as large as possible up front and\n> only then should multiple block copy be encoded to cover it all.\n> Also, fewer hash lookups will be performed in the end.\n> \n> According to Martin, this patch should reduce his 92MB pack down to 75MB\n> with the dataset he has.\n> \n> Tests performed on the Linux kernel repo show a slightly smaller pack and\n> a slightly faster repack.\n>\nAcked-by: Martin Koegler <mkoegler@auto.tuwien.ac.at>\n> Signed-off-by: Nicolas Pitre <nico@cam.org>\n---\n\nThe patch results in a 75 MB pack file for my repository and is\nfaster:\n\nTotal 6452 (delta 4581), reused 1522 (delta 0)\n10073.11user 5200.33system 4:14:36elapsed 99%CPU (0avgtext+0avgdata 0maxresident)k\n0inputs+0outputs (0major+1371504760minor)pagefaults 0swaps\n\nmfg Martin Kögler\n"}]}