{"thread":{"id":"30057","subject":"[GSOC 2012] Some questions regarding a possible project to improve big file support","startedAt":"2012-03-25T20:48:09Z","lastAt":"2012-03-26T01:21:53Z","messageCount":2,"participants":["Peter C.","Nguyen Thai Ngoc Duy"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"187692","messageId":"4F6F8489.20108@gmail.com","threadId":"30057","inReplyTo":null,"subject":"[GSOC 2012] Some questions regarding a possible project to improve big file support","fromName":"Peter C.","fromEmail":"th3flyboy@gmail.com","sentAt":"2012-03-25T20:48:09Z","receivedAt":"2012-03-25T20:48:09Z","isPatch":false,"sender":{"key":"th3flyboy@gmail.com","avatar":null},"body":"Hello, I'm considering working on Git for GSOC 2012, specifically in\nimproving big file support, however I wanted to ask a few questions\nfirst, some about the low level operations of how Git handles diffs\nbetween files, and also a question or two regarding implementation.\n\nMy first question is more of a question regarding low level\nfunctionality of how Git diffs files. The question is, in the diff\nprocess, does git just parse the file and see if there are diffs, or\ndoes it use something like hashing to first tell if the file has been\nmodified at all, and then go to the diff process if the hash is\ndifferent. An extension to this question is, in Git's internal database,\ndoes it set any kind of flag to say that a file is a binary if it is one.\n\nMy thought process in implementation involves checking the hash, and if\nthe hash is the same, skip it, if the hash is different, check the MIME\ntype possibly using libmagic, and if it matches a known binary format,\nthen just commit the new version, rather than trying to run a whole diff\nand load the whole file in the process.\n\nThe thing I'm worried about is, would anything involved in this break\nexisting Git functionality, or backward compatibility. I'd also greatly\nappreciate any feedback on my ideas.\n\nThanks,\nPeter\n"},{"id":"187698","messageId":"CACsJy8CQHbwCQGypVRgKhD9Wb4O6kXGoDLD9EUZgnBP-6ZivMw@mail.gmail.com","threadId":"30057","inReplyTo":"4F6F8489.20108@gmail.com","subject":"Re: [GSOC 2012] Some questions regarding a possible project to improve big file support","fromName":"Nguyen Thai Ngoc Duy","fromEmail":"pclouds@gmail.com","sentAt":"2012-03-26T01:21:53Z","receivedAt":"2012-03-26T01:21:53Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Mar 26, 2012 at 3:48 AM, Peter C. <th3flyboy@gmail.com> wrote:\n> My first question is more of a question regarding low level\n> functionality of how Git diffs files. The question is, in the diff\n> process, does git just parse the file and see if there are diffs, or\n> does it use something like hashing to first tell if the file has been\n> modified at all, and then go to the diff process if the hash is\n> different. An extension to this question is, in Git's internal database,\n> does it set any kind of flag to say that a file is a binary if it is one.\n\nIf hashes are available, we compare hash first (e.g. diff-tree). We\ncan mark a file binary with gitattributes. I think the binary\ndetection code, buffer_is_binary, could be just moved up a little bit\nbefore we unpack file contents. But I'm not really familiar with this\narea.\n\n> My thought process in implementation involves checking the hash, and if\n> the hash is the same, skip it, if the hash is different, check the MIME\n> type possibly using libmagic, and if it matches a known binary format,\n> then just commit the new version, rather than trying to run a whole diff\n> and load the whole file in the process.\n\nOverkill, compared to how binary is detected today :)\n\n#define FIRST_FEW_BYTES 8000\nint buffer_is_binary(const char *ptr, unsigned long size)\n{\n\tif (FIRST_FEW_BYTES < size)\n\t\tsize = FIRST_FEW_BYTES;\n\treturn !!memchr(ptr, 0, size);\n}\n\nIf you are interested in this big file support, I think you should\nfocus on the \"Many large files do not delta well...\" item in the wiki\npage. The framework has already been done by Junio. That can make git\nmanage gigabyte files just fine (aka \"bup\").\n-- \nDuy\n"}]}