Re: Fw: Curiosity
- From
- Martin Fick <mfick@codeaurora.org>
- Date
- Dec 18, 2021, 01:06 UTC
- Message-ID
- <df4a5ac37e8d703fa54af91269a9a736@codeaurora.org>
- In-Reply-To
- <1X3gQ48NK5aBDHcpYMlxESRjqubcCBKJUQu2K0dBOnTyvsXCXXoGDBg2Ff4KarK6WsZnzN3HgqHGOlCKKdF-wtZQ5tHsoAcfit2CTXMWqh4=@protonmail.com>
On 2021-12-17 17:15, João Victor Bonfim wrote:
Show 9 quoted lines
>> I suspect that for most algorithms and their implementations, this >> would >> >> not result in repeatable "recompressed" results. Thus the checked-out >> >> files might be different every time you checked them out. :( > > How or why? >
Here are some reasons I can think of (I am no expert):
1) Most compression formats are file formats, not exact algorithms, thus different program implementations of similar algorithms can create vastly different outputs.
2) The same program will evolve over time, get improvements, bug fixes, etc. so each version of the same program could vary over time even with the same settings. The same program version on different platforms could have different output.
3) Settings, compression programs have compression levels, perhaps memory utilization parameters... The way the program measures these may not be deterministic and non-repeatable.
4) Threading. Some compressions algorithms, such as git repack itself, can use several threads to analyze the input data. And since the timing between different threads is not deterministic, when cooperating, they can have different results.
Much of this has to do with the idea that there is usually no such thing as "done" when it comes to compression. You can probably search infinitely to try and find more data patterns to compress the data more. Thus compression programs have to have limits based on heuristics (how far to look ahead/behind, how many patterns to remember...) programmed into them to come to an end somehow. How these limits are determined can sometimes be non deterministic, it may even involve system resources (how much RAM the machine has, how long it has run...) or system config.
I hope that helps,
-Martin
Show 63 quoted lines
> ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐ > > Em quinta-feira, 16 de dezembro de 2021 às 18:33, Martin Fick > <mfick@codeaurora.org> escreveu: > >> On 2021-12-16 14:20, João Victor Bonfim wrote: >> >> > > To expand on this, if what you're storing is already compressed, like >> > > >> > > Ogg Vorbis files or PNGs, like are found in that repository, then >> > > >> > > generally they will not delta well. This is also true of things like >> > > >> > > Microsoft Office or OpenOffice documents, because they're essentially >> > > >> > > Zip files. >> > > >> > > The delta algorithm looks for similarities between files to compress >> > > >> > > them. If a file is already compressed using something like Deflate, >> > > >> > > used in PNGs and Zip files, then even very similar files will >> > > >> > > generally >> > > >> > > look very different, so deltification will generally be ineffective. >> >> ... >> >> > Maybe I am thinking too outside the box, but wouldn't it be quite more >> > >> > effective for git to identify compressed files, specially on edge cases >> > >> > where the compression doesn't have a good chemistry with delta >> > >> > compression, >> > >> > decompress them for repo storage while also storing the compression >> > >> > algorithm as some metadata tag (like a text string or an ID code >> > >> > decided >> > >> > beforehand), and, when creating the work mirrors, return the >> > >> > compression >> > >> > to its default state before checkout? >> >> I suspect that for most algorithms and their implementations, this >> would >> >> not result in repeatable "recompressed" results. Thus the checked-out >> >> files might be different every time you checked them out. :( >> >> -Martin >> >> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- >> >> The Qualcomm Innovation Center, Inc. is a member of Code >> >> Aurora Forum, hosted by The Linux Foundation
-- The Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation