git/list[1] front-page[2] threads[3] people[4] search[5] about
 

on when to checksum

From
TLTom Lord <lord@emf.net>
Date
Apr 20, 2005, 22:25 UTC
Message-ID
<200504202225.PAA15992@emf.net>
Linus, 

I think you have made a mistake by moving the sha1 checksum from the zipped form to the inflated form. Here is why:

What you have set in motion with `git' is an ad-hoc p2p network for sharing filesystem trees -- a global distributed filesystem. I believe your starter here has a good chance of taking off to be much, much larger than just a tool for the kernel.

A subset of your work: blobs and blob databaes, has much wider application than just sharing trees: Those parts of `git' can form a very solid foundation for many other applications as well. To the extent `git' succeeds in the context of the kernel, it will be invested in and extended and generalized --- and the kernel project will benefit. So don't ignore those wider applications even though they are not your focus today: they will generate investment that feeds back to your project.

Your `git' is silent on transports and mirroring of blob databases -- tasks for scripting, sure -- but those elements won't be far behind.

Eventually, slinging around blobs as atomic elements of payloads will become very common.

The blob handle (aka "address")/payload model of a blob db is very clean and simple. In a network of nodes speaking to one and other by exchanging blobs, I forsee a prominent need for intermediate nodes that process blobs "blindly" and as quickly as possible.

Blob compression is mostly goofy if regarded just as a way to save on (diminishingly cheap) disk space but it is mostly sane if regarded as a way to cut the cost of network bandwidth roughly in half.

Must intermediate nodes inflate the payloads passing through them or which they cache just to validate them? That's not a desirable otucome for many obvious reasonhs.

There *are* concerns about checksumming zips: it is necessary to nail down the zip process and make sure it is absolutely and permanently deterministic for this application. But *that* is the problem to solve, not avoid by moving what the checksum refers to.

Thanks, -t

Next: Linus Torvalds
Message 1 of 8 in “on when to checksum”
  1. Tom LordApr 20, 2005
  2. Linus TorvaldsApr 20, 2005
  3. Tom LordApr 20, 2005
  4. Linus TorvaldsApr 20, 2005
  5. Tom LordApr 20, 2005
  6. Tom LordMay 2, 2005
  7. Linus TorvaldsMay 2, 2005
  8. Andrew Timberlake-NewellApr 21, 2005

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.