{"thread":{"id":"46702","subject":"[PATCH] read-cache: fix index corruption with index v4","startedAt":"2017-09-04T22:57:55Z","lastAt":"2017-09-07T19:23:57Z","messageCount":4,"participants":["Thomas Gummerer","Kevin Willford"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"327537","messageId":"20170904225807.22139-1-t.gummerer@gmail.com","threadId":"46702","inReplyTo":null,"subject":"[PATCH] read-cache: fix index corruption with index v4","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2017-09-04T22:58:06Z","receivedAt":"2017-09-04T22:57:55Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"ce012deb98 (\"read-cache: avoid allocating every ondisk entry when\nwriting\", 2017-08-21) changed the way cache entries are written to the\nindex file.  While previously it wrote the name to an struct that was\nallocated using xcalloc(), it now uses ce_write() directly.  Previously\nce_namelen - common bytes were written to the cache entry, which would\nautomatically make it nul terminated, as it was allocated using calloc.\n\nNow we are writing ce_namelen - common + 1 bytes directly from the\nce->name to the index.  As ce->name is however not nul terminated, and\nindex-v4 needs the nul terminator to split between one index entry and\nthe next, this would end up in a corrupted index.\n\nFix that by only writing ce_namelen - common bytes directly from\nce->name to the index, and adding the nul terminator in an extra call to\nce_write.\n\nThis bug was turned up by setting TEST_GIT_INDEX_VERSION = 4 in\nconfig.mak and running the test suite (t1700 specifically broke).\n\nSigned-off-by: Thomas Gummerer <t.gummerer@gmail.com>\n---\n\nI unfortunately didn't have more time to dig so\n\n> As ce->name is however not nul terminated\n\njust comes from my memory and from the patch below actually fixing the\ncorruption, so it's really the most likely cause.  Would be great if\nsomeone who can remember more about the index could confirm that this\nis indeed the case.\n\n read-cache.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 40da87ea71..80830ddcfc 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -2103,7 +2103,9 @@ static int ce_write_entry(git_SHA_CTX *c, int fd, struct cache_entry *ce,\n \t\tif (!result)\n \t\t\tresult = ce_write(c, fd, to_remove_vi, prefix_size);\n \t\tif (!result)\n-\t\t\tresult = ce_write(c, fd, ce->name + common, ce_namelen(ce) - common + 1);\n+\t\t\tresult = ce_write(c, fd, ce->name + common, ce_namelen(ce) - common);\n+\t\tif (!result)\n+\t\t\tresult = ce_write(c, fd, \"\\0\", 1);\n \n \t\tstrbuf_splice(previous_name, common, to_remove,\n \t\t\t      ce->name + common, ce_namelen(ce) - common);\n-- \n2.14.1.480.gb18f417b89\n\n"},{"id":"327599","messageId":"SN1PR21MB001457E4DD1AE17D041C6C77B7960@SN1PR21MB0014.namprd21.prod.outlook.com","threadId":"46702","inReplyTo":"20170904225807.22139-1-t.gummerer@gmail.com","subject":"RE: [PATCH] read-cache: fix index corruption with index v4","fromName":"Kevin Willford","fromEmail":"kewillf@microsoft.com","sentAt":"2017-09-05T19:37:28Z","receivedAt":"2017-09-05T19:37:39Z","isPatch":true,"sender":{"key":"kewillf@microsoft.com","avatar":null},"body":"> From: Thomas Gummerer [mailto:t.gummerer@gmail.com]\n> Sent: Monday, September 4, 2017 4:58 PM\n> \n> ce012deb98 (\"read-cache: avoid allocating every ondisk entry when\n> writing\", 2017-08-21) changed the way cache entries are written to the\n> index file.  While previously it wrote the name to an struct that was\n> allocated using xcalloc(), it now uses ce_write() directly.  Previously\n> ce_namelen - common bytes were written to the cache entry, which would\n> automatically make it nul terminated, as it was allocated using calloc.\n> \n> Now we are writing ce_namelen - common + 1 bytes directly from the\n> ce->name to the index.  As ce->name is however not nul terminated, and\n> index-v4 needs the nul terminator to split between one index entry and\n> the next, this would end up in a corrupted index.\n> \n> Fix that by only writing ce_namelen - common bytes directly from\n> ce->name to the index, and adding the nul terminator in an extra call to\n> ce_write.\n> \n> This bug was turned up by setting TEST_GIT_INDEX_VERSION = 4 in\n> config.mak and running the test suite (t1700 specifically broke).\n> \n> Signed-off-by: Thomas Gummerer <t.gummerer@gmail.com>\n> ---\n> \n> I unfortunately didn't have more time to dig so\n> \n> > As ce->name is however not nul terminated\n> \n> just comes from my memory and from the patch below actually fixing the\n> corruption, so it's really the most likely cause.  Would be great if\n> someone who can remember more about the index could confirm that this\n> is indeed the case.\n> \n\nDigging into this and ce->name IS nul terminated.  The issue comes in when\nthe CE_STRIP_NAME is set, which is only set when using a split index. \nThis sets the ce->ce_namelen = 0 without changing the actual ce->name buffer.\nWhen writing the entry for the split index version 4 it was using the first character\nin the ce->name buffer because of the + 1, which obviously isn't correct.    Before\nit was using a newly allocated name buffer from the ondisk struct which was\nallocated based on the ce_namelen of zero.\n\n>  read-cache.c | 4 +++-\n>  1 file changed, 3 insertions(+), 1 deletion(-)\n> \n> diff --git a/read-cache.c b/read-cache.c\n> index 40da87ea71..80830ddcfc 100644\n> --- a/read-cache.c\n> +++ b/read-cache.c\n> @@ -2103,7 +2103,9 @@ static int ce_write_entry(git_SHA_CTX *c, int fd, struct\n> cache_entry *ce,\n>  \t\tif (!result)\n>  \t\t\tresult = ce_write(c, fd, to_remove_vi, prefix_size);\n>  \t\tif (!result)\n> -\t\t\tresult = ce_write(c, fd, ce->name + common,\n> ce_namelen(ce) - common + 1);\n> +\t\t\tresult = ce_write(c, fd, ce->name + common,\n> ce_namelen(ce) - common);\n> +\t\tif (!result)\n> +\t\t\tresult = ce_write(c, fd, \"\\0\", 1);\n\nYou could use the padding variable here as well which is used in the < version 4\nce_write.\n\n> \n>  \t\tstrbuf_splice(previous_name, common, to_remove,\n>  \t\t\t      ce->name + common, ce_namelen(ce) - common);\n> --\n> 2.14.1.480.gb18f417b89\n\nWhile looking at the code I was wondering if we could get around the\nwhole setting ce->ce_namelen to zero bit but that would be much bigger\npatch and possibly introduce other bugs so this seems the appropriate\nfix for now.\n\nThanks for finding this!\nKevin \n"},{"id":"327672","messageId":"CALgYhfNYgmCJqptNQLKaQpCs9mAgqZHUrDS3BVEqCv_f+WX-qg@mail.gmail.com","threadId":"46702","inReplyTo":"SN1PR21MB001457E4DD1AE17D041C6C77B7960@SN1PR21MB0014.namprd21.prod.outlook.com","subject":"Re: [PATCH] read-cache: fix index corruption with index v4","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2017-09-06T20:06:16Z","receivedAt":"2017-09-06T20:06:23Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"On Tue, Sep 5, 2017 at 8:37 PM, Kevin Willford <kewillf@microsoft.com> wrote:\n>> From: Thomas Gummerer [mailto:t.gummerer@gmail.com]\n>> Sent: Monday, September 4, 2017 4:58 PM\n\n[..]\n\n>> I unfortunately didn't have more time to dig so\n>>\n>> > As ce->name is however not nul terminated\n>>\n>> just comes from my memory and from the patch below actually fixing the\n>> corruption, so it's really the most likely cause.  Would be great if\n>> someone who can remember more about the index could confirm that this\n>> is indeed the case.\n>>\n>\n> Digging into this and ce->name IS nul terminated.  The issue comes in when\n> the CE_STRIP_NAME is set, which is only set when using a split index.\n> This sets the ce->ce_namelen = 0 without changing the actual ce->name buffer.\n> When writing the entry for the split index version 4 it was using the first character\n> in the ce->name buffer because of the + 1, which obviously isn't correct.    Before\n> it was using a newly allocated name buffer from the ondisk struct which was\n> allocated based on the ce_namelen of zero.\n\nThank you very much for digging into this.  That also explains why\nonly t1700 was\naffected, but none of the other tests.  Will update the commit message.\n\n>>  read-cache.c | 4 +++-\n>>  1 file changed, 3 insertions(+), 1 deletion(-)\n>>\n>> diff --git a/read-cache.c b/read-cache.c\n>> index 40da87ea71..80830ddcfc 100644\n>> --- a/read-cache.c\n>> +++ b/read-cache.c\n>> @@ -2103,7 +2103,9 @@ static int ce_write_entry(git_SHA_CTX *c, int fd, struct\n>> cache_entry *ce,\n>>               if (!result)\n>>                       result = ce_write(c, fd, to_remove_vi, prefix_size);\n>>               if (!result)\n>> -                     result = ce_write(c, fd, ce->name + common,\n>> ce_namelen(ce) - common + 1);\n>> +                     result = ce_write(c, fd, ce->name + common,\n>> ce_namelen(ce) - common);\n>> +             if (!result)\n>> +                     result = ce_write(c, fd, \"\\0\", 1);\n>\n> You could use the padding variable here as well which is used in the < version 4\n> ce_write.\n\nThanks, will do that.\n\n>>\n>>               strbuf_splice(previous_name, common, to_remove,\n>>                             ce->name + common, ce_namelen(ce) - common);\n>> --\n>> 2.14.1.480.gb18f417b89\n>\n> While looking at the code I was wondering if we could get around the\n> whole setting ce->ce_namelen to zero bit but that would be much bigger\n> patch and possibly introduce other bugs so this seems the appropriate\n> fix for now.\n>\n> Thanks for finding this!\n\nThanks for the review! Will send an updated patch in a bit.\n\n> Kevin\n"},{"id":"327726","messageId":"20170907192412.8085-1-t.gummerer@gmail.com","threadId":"46702","inReplyTo":"CALgYhfNYgmCJqptNQLKaQpCs9mAgqZHUrDS3BVEqCv_f+WX-qg@mail.gmail.com","subject":"[PATCH v2] read-cache: fix index corruption with index v4","fromName":"Thomas Gummerer","fromEmail":"t.gummerer@gmail.com","sentAt":"2017-09-07T19:24:12Z","receivedAt":"2017-09-07T19:23:57Z","isPatch":true,"sender":{"key":"t.gummerer@gmail.com","avatar":"https://avatars.githubusercontent.com/u/191004?v=4"},"body":"ce012deb98 (\"read-cache: avoid allocating every ondisk entry when\nwriting\", 2017-08-21) changed the way cache entries are written to the\nindex file.  While previously it wrote the name to an struct that was\nallocated using xcalloc(), it now uses ce_write() directly.  Previously\nce_namelen - common bytes were written to the cache entry, which would\nautomatically make it nul terminated, as it was allocated using calloc.\n\nNow we are writing ce_namelen - common + 1 bytes directly from the\nce->name to the index.  If CE_STRIP_NAME however gets set in the split\nindex case ce->ce_namelen is set to 0 without changing the actual\nce->name buffer.  When index-v4, this results in the first character of\nce->name being written out instead of just a terminating nul charcter.\n\nAs index-v4 requires the terminating nul character as terminator of\nthe name when reading it back, this results in a corrupted index.\n\nFix that by only writing ce_namelen - common bytes directly from\nce->name to the index, and adding the nul terminator in an extra call to\nce_write.\n\nThis bug was turned up by setting TEST_GIT_INDEX_VERSION = 4 in\nconfig.mak and running the test suite (t1700 specifically broke).\n\nSigned-off-by: Thomas Gummerer <t.gummerer@gmail.com>\n---\n\n> Will send an updated patch in a bit.\n\nIn a bit was a lie, I didn't get to it anymore yesterday, but here it is :)\n\n read-cache.c | 4 +++-\n 1 file changed, 3 insertions(+), 1 deletion(-)\n\ndiff --git a/read-cache.c b/read-cache.c\nindex 40da87ea71..c6c69cf027 100644\n--- a/read-cache.c\n+++ b/read-cache.c\n@@ -2103,7 +2103,9 @@ static int ce_write_entry(git_SHA_CTX *c, int fd, struct cache_entry *ce,\n \t\tif (!result)\n \t\t\tresult = ce_write(c, fd, to_remove_vi, prefix_size);\n \t\tif (!result)\n-\t\t\tresult = ce_write(c, fd, ce->name + common, ce_namelen(ce) - common + 1);\n+\t\t\tresult = ce_write(c, fd, ce->name + common, ce_namelen(ce) - common);\n+\t\tif (!result)\n+\t\t\tresult = ce_write(c, fd, padding, 1);\n \n \t\tstrbuf_splice(previous_name, common, to_remove,\n \t\t\t      ce->name + common, ce_namelen(ce) - common);\n-- \n2.14.1.480.gb18f417b89\n\n"}]}