{"thread":{"id":"18865","subject":"On Tracking Binary Files","startedAt":"2009-04-14T14:42:22Z","lastAt":"2009-04-14T20:27:10Z","messageCount":6,"participants":["Patrick Berkeley","Johannes Schindelin","Nicolas Pitre","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"111289","messageId":"7efce40a0904140742i48aad41ds66de5dfe368f3b16@mail.gmail.com","threadId":"18865","inReplyTo":"7efce40a0904140741w28da9b54ucfe4b54bf48b0844@mail.gmail.com","subject":"On Tracking Binary Files","fromName":"Patrick Berkeley","fromEmail":"patrickberkeley@gmail.com","sentAt":"2009-04-14T14:42:22Z","receivedAt":"2009-04-14T14:42:22Z","isPatch":false,"sender":{"key":"patrickberkeley@gmail.com","avatar":"https://gravatar.com/avatar/5d040bdc3120f328fe443af2a77d209de2fe2ce6379254860ed253f21f835b9d?d=mp&s=160"},"body":"Does Git track the deltas on binary files?\n\nSomeone in #git mentioned that if the binaries change too much Git no\nlonger just stores the changes. If this is the case, what is the\nbreaking point where Git goes from storing the deltas to the entire\nnew file?\n\nThanks, Patrick\n"},{"id":"111294","messageId":"alpine.DEB.1.00.0904141853040.10279@pacific.mpi-cbg.de","threadId":"18865","inReplyTo":"7efce40a0904140742i48aad41ds66de5dfe368f3b16@mail.gmail.com","subject":"Re: On Tracking Binary Files","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2009-04-14T16:54:06Z","receivedAt":"2009-04-14T16:54:06Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi,\n\nOn Tue, 14 Apr 2009, Patrick Berkeley wrote:\n\n> Does Git track the deltas on binary files?\n> \n> Someone in #git mentioned that if the binaries change too much Git no\n> longer just stores the changes. If this is the case, what is the\n> breaking point where Git goes from storing the deltas to the entire\n> new file?\n\nGit does not store the deltas as you think it does. The deltification of \nthe objects is almost independent from the commmit history, i.e. we \n_always_ store snapshots for most practical matters.\n\nCiao,\nDscho\n"},{"id":"111308","messageId":"alpine.LFD.2.00.0904141532160.6741@xanadu.home","threadId":"18865","inReplyTo":"7efce40a0904140742i48aad41ds66de5dfe368f3b16@mail.gmail.com","subject":"Re: On Tracking Binary Files","fromName":"Nicolas Pitre","fromEmail":"nico@cam.org","sentAt":"2009-04-14T19:42:07Z","receivedAt":"2009-04-14T19:42:07Z","isPatch":false,"sender":{"key":"nico@fluxnic.net","avatar":"https://avatars.githubusercontent.com/u/702790?v=4"},"body":"On Tue, 14 Apr 2009, Patrick Berkeley wrote:\n\n> Does Git track the deltas on binary files?\n\nYes. And actually git's delta storage doesn't care at all whether a \nfile is text or binary.\n\n> Someone in #git mentioned that if the binaries change too much Git no \n> longer just stores the changes. If this is the case, what is the \n> breaking point where Git goes from storing the deltas to the entire \n> new file?\n\nIf two versions of the same file are simply too different to make delta \ncompression worth it, then no deltas are used. It is still possible \nthat a third version of the same file would produce a nice delta against \neither the first or second version though, in which case that third \nversion will be stored as a delta. And so on.\n\nA sophisticated set of euristics is applied to the list of objects as a \nwhole to determine the best delta arrangement possible. So there is no \nsuch thing as a simple \"breaking point\".\n\n\nNicolas\n"},{"id":"111309","messageId":"7efce40a0904141244n281353f3q716071d4fff5a7a0@mail.gmail.com","threadId":"18865","inReplyTo":"alpine.LFD.2.00.0904141532160.6741@xanadu.home","subject":"Re: On Tracking Binary Files","fromName":"Patrick Berkeley","fromEmail":"patrickberkeley@gmail.com","sentAt":"2009-04-14T19:44:26Z","receivedAt":"2009-04-14T19:44:26Z","isPatch":false,"sender":{"key":"patrickberkeley@gmail.com","avatar":"https://gravatar.com/avatar/5d040bdc3120f328fe443af2a77d209de2fe2ce6379254860ed253f21f835b9d?d=mp&s=160"},"body":"Thanks very much for the explanation .\n\n\nOn Tue, Apr 14, 2009 at 15:42, Nicolas Pitre wrote:\n> On Tue, 14 Apr 2009, Patrick Berkeley wrote:\n>\n>> Does Git track the deltas on binary files?\n>\n> Yes.  And actually git's delta storage doesn't care at all whether a\n> file is text or binary.\n>\n>> Someone in #git mentioned that if the binaries change too much Git no\n>> longer just stores the changes. If this is the case, what is the\n>> breaking point where Git goes from storing the deltas to the entire\n>> new file?\n>\n> If two versions of the same file are simply too different to make delta\n> compression worth it, then no deltas are used.  It is still possible\n> that a third version of the same file would produce a nice delta against\n> either the first or second version though, in which case that third\n> version will be stored as a delta.  And so on.\n>\n> A sophisticated set of euristics is applied to the list of objects as a\n> whole to determine the best delta arrangement possible.  So there is no\n> such thing as a simple \"breaking point\".\n>\n>\n> Nicolas\n>\n"},{"id":"111310","messageId":"7vws9n2e7p.fsf@gitster.siamese.dyndns.org","threadId":"18865","inReplyTo":"alpine.DEB.1.00.0904141853040.10279@pacific.mpi-cbg.de","subject":"Re: On Tracking Binary Files","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2009-04-14T20:05:30Z","receivedAt":"2009-04-14T20:05:30Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> On Tue, 14 Apr 2009, Patrick Berkeley wrote:\n>\n>> Does Git track the deltas on binary files?\n>> \n>> Someone in #git mentioned that if the binaries change too much Git no\n>> longer just stores the changes. If this is the case, what is the\n>> breaking point where Git goes from storing the deltas to the entire\n>> new file?\n>\n> Git does not store the deltas as you think it does. The deltification of \n> the objects is almost independent from the commmit history, i.e. we \n> _always_ store snapshots for most practical matters.\n\nAlways store snapshots sounds as if you are not storing delta at all. I\nthink I know what you meant to say, but the way you phrased it is\nmisleading.\n\nDocumentation/technical/pack-heuristics.txt talks about this in some\ndetail. A short version is:\n\n - It does not make a difference if you are dealing with binary or text;\n\n - The delta is not necessarily against the same path in the previous\n revision, so even a new file added to the history can be stored in a\n delitified form;\n\n - When an object stored in the deltified representation is used, it would\n incur more cost than using the same object in the compressed base\n representation. The deltification mechanism makes a trade-off taking\n this cost into account, as well as the space efficiency.\n\nThe last point may probably be not covered by pack-heuristics IRC talk\nLinus had in the documentation. Basically:\n\n - A deltified object is stored as an (compressed) xdelta against some\n base object. If the best deltified representation we come up with is\n larger than the result of just compressing the object without\n deltification, it is not worth storing it from the space comsumption\n point of view. Thus, we originally said something like \"if an\n attempted delta is larger than half of the object size (assuming\n average 50% of compression ratio), do not use the deltified\n representation, it is not worth it\". We attempt to delta against many\n base objects to pick the best possible delta; the number of attempt is\n called the delta window.\n\n - The base object of a deltified object could also be deltified, and you\n may need to repeatedly apply delta on top of some object that is not a\n delta to get to the final object. The length of this chain is called\n delta depth, and obviously you would want to keep the delta depth short\n to gain a reasonable runtime performance. Thus, when delitifying one\n object A, we make a weighted comparison between the size of the delta\n to build it out of an object of depth N and the size of the delta to\n build it out of an object of depth M. A slightly larger delta that is\n based on an object with a shallower delta depth is favored over a\n smaller delta based on an object with a much deeper delta depth.\n"},{"id":"111314","messageId":"7efce40a0904141327w3cbfbfecwbe7d5d9125fe8d4a@mail.gmail.com","threadId":"18865","inReplyTo":"7vws9n2e7p.fsf@gitster.siamese.dyndns.org","subject":"Re: On Tracking Binary Files","fromName":"Patrick Berkeley","fromEmail":"patrickberkeley@gmail.com","sentAt":"2009-04-14T20:27:10Z","receivedAt":"2009-04-14T20:27:10Z","isPatch":false,"sender":{"key":"patrickberkeley@gmail.com","avatar":"https://gravatar.com/avatar/5d040bdc3120f328fe443af2a77d209de2fe2ce6379254860ed253f21f835b9d?d=mp&s=160"},"body":"Junio,\n\nThanks a lot for your thorough explanation..\n\nPatrick\n\nOn Tue, Apr 14, 2009 at 16:05, Junio C Hamano wrote:\n> Johannes Schindelin writes:\n>\n>> On Tue, 14 Apr 2009, Patrick Berkeley wrote:\n>>\n>>> Does Git track the deltas on binary files?\n>>>\n>>> Someone in #git mentioned that if the binaries change too much Git no\n>>> longer just stores the changes. If this is the case, what is the\n>>> breaking point where Git goes from storing the deltas to the entire\n>>> new file?\n>>\n>> Git does not store the deltas as you think it does.  The deltification of\n>> the objects is almost independent from the commmit history, i.e. we\n>> _always_ store snapshots for most practical matters.\n>\n> Always store snapshots sounds as if you are not storing delta at all.  I\n> think I know what you meant to say, but the way you phrased it is\n> misleading.\n>\n> Documentation/technical/pack-heuristics.txt talks about this in some\n> detail.  A short version is:\n>\n>  - It does not make a difference if you are dealing with binary or text;\n>\n>  - The delta is not necessarily against the same path in the previous\n>   revision, so even a new file added to the history can be stored in a\n>   delitified form;\n>\n>  - When an object stored in the deltified representation is used, it would\n>   incur more cost than using the same object in the compressed base\n>   representation.  The deltification mechanism makes a trade-off taking\n>   this cost into account, as well as the space efficiency.\n>\n> The last point may probably be not covered by pack-heuristics IRC talk\n> Linus had in the documentation.  Basically:\n>\n>  - A deltified object is stored as an (compressed) xdelta against some\n>   base object.  If the best deltified representation we come up with is\n>   larger than the result of just compressing the object without\n>   deltification, it is not worth storing it from the space comsumption\n>   point of view.  Thus, we originally said something like \"if an\n>   attempted delta is larger than half of the object size (assuming\n>   average 50% of compression ratio), do not use the deltified\n>   representation, it is not worth it\".  We attempt to delta against many\n>   base objects to pick the best possible delta; the number of attempt is\n>   called the delta window.\n>\n>  - The base object of a deltified object could also be deltified, and you\n>   may need to repeatedly apply delta on top of some object that is not a\n>   delta to get to the final object.  The length of this chain is called\n>   delta depth, and obviously you would want to keep the delta depth short\n>   to gain a reasonable runtime performance.  Thus, when delitifying one\n>   object A, we make a weighted comparison between the size of the delta\n>   to build it out of an object of depth N and the size of the delta to\n>   build it out of an object of depth M.  A slightly larger delta that is\n>   based on an object with a shallower delta depth is favored over a\n>   smaller delta based on an object with a much deeper delta depth.\n>\n>\n"}]}