{"thread":{"id":"60272","subject":"[silly] loose, pack, and another thing?","startedAt":"2023-09-27T04:52:56Z","lastAt":"2023-10-04T13:11:15Z","messageCount":7,"participants":["Junio C Hamano","Christian Couder","Jonathan Tan","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"482365","messageId":"xmqqbkdometi.fsf@gitster.g","threadId":"60272","inReplyTo":null,"subject":"[silly] loose, pack, and another thing?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-09-27T04:15:53Z","receivedAt":"2023-09-27T04:52:56Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Just wondering if it would help to have the third kind of object\nrepresentation in the object database, sitting next to loose objects\nand packed objects, say .git/objects/verbatim/ for\nthe contents and .git/objects/verbatim/.type that\nrecords \"blob\", \"tree\", \"commit\", or \"tag\" (in practice, I would\nexpect huge \"blob\" objects would be the only ones that use this\nmechanism).\n\nThe contents will be stored verbatim without compression and without\nany object header (i.e., the usual \" \\0\") and the file\ncould be \"ln\"ed (or \"cow\"ed if the underlying filesystem allows it)\nto materialize it in the working tree if needed.\n\n\"fsck\" needs to be told about how to verify them. Create the object\nheader in-core and hash that, followed by the contents of that file,\nand make sure the result matches the part of the\nfilename, or something like that.\n"},{"id":"482374","messageId":"CAP8UFD0FLsEV4y4W8Vkr5PP_7F4x-nLdHUVx5WRP8BuJnW869A@mail.gmail.com","threadId":"60272","inReplyTo":"xmqqbkdometi.fsf@gitster.g","subject":"Re: [silly] loose, pack, and another thing?","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2023-09-27T13:46:06Z","receivedAt":"2023-09-27T13:46:29Z","isPatch":false,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Wed, Sep 27, 2023 at 11:47 AM Junio C Hamano wrote:\n>\n> Just wondering if it would help to have the third kind of object\n> representation in the object database, sitting next to loose objects\n> and packed objects, say .git/objects/verbatim/ for\n> the contents and .git/objects/verbatim/.type that\n> records \"blob\", \"tree\", \"commit\", or \"tag\" (in practice, I would\n> expect huge \"blob\" objects would be the only ones that use this\n> mechanism).\n\nYeah, I think it could help handle large blobs. I guess it would rely\non the underlying filesystem to store the object size.\n\n> The contents will be stored verbatim without compression and without\n> any object header (i.e., the usual \" \\0\") and the file\n> could be \"ln\"ed (or \"cow\"ed if the underlying filesystem allows it)\n> to materialize it in the working tree if needed.\n>\n> \"fsck\" needs to be told about how to verify them. Create the object\n> header in-core and hash that, followed by the contents of that file,\n> and make sure the result matches the part of the\n> filename, or something like that.\n\nWhat happens when they are transferred? Should the remote unpack them\ninto the same kind of verbatim object?\n"},{"id":"482442","messageId":"20230928214010.3502838-1-jonathantanmy@google.com","threadId":"60272","inReplyTo":"xmqqbkdometi.fsf@gitster.g","subject":"Re: [silly] loose, pack, and another thing?","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-09-28T21:40:10Z","receivedAt":"2023-09-28T21:40:17Z","isPatch":false,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Junio C Hamano writes:\n> Just wondering if it would help to have the third kind of object\n> representation in the object database, sitting next to loose objects\n> and packed objects, say .git/objects/verbatim/ for\n> the contents and .git/objects/verbatim/.type that\n> records \"blob\", \"tree\", \"commit\", or \"tag\" (in practice, I would\n> expect huge \"blob\" objects would be the only ones that use this\n> mechanism).\n> \n> The contents will be stored verbatim without compression and without\n> any object header (i.e., the usual \" \\0\") and the file\n> could be \"ln\"ed (or \"cow\"ed if the underlying filesystem allows it)\n> to materialize it in the working tree if needed.\n\nThis sounds like a useful feature. We probably would want to use the\n\"ln\" or \"cow\" every time we use streaming (stream_blob_to_fd() in\nstreaming.h) currently, so hopefully we won't need to increase the\nnumber of ways in which we can write an object to the worktree (just\nchange the streaming to write to a filename instead of an fd).\n\n> \"fsck\" needs to be told about how to verify them. Create the object\n> header in-core and hash that, followed by the contents of that file,\n> and make sure the result matches the part of the\n> filename, or something like that.\n\nYeah, this sounds like what index-pack is doing - the hash algo can take\nthe contents of one buffer (a header that we synthesize ourselves), and\nthen take the contents of another buffer (the file contents).\n"},{"id":"482443","messageId":"20230928214713.3504919-1-jonathantanmy@google.com","threadId":"60272","inReplyTo":"CAP8UFD0FLsEV4y4W8Vkr5PP_7F4x-nLdHUVx5WRP8BuJnW869A@mail.gmail.com","subject":"Re: [silly] loose, pack, and another thing?","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-09-28T21:47:13Z","receivedAt":"2023-09-28T21:47:20Z","isPatch":false,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Christian Couder writes:\n> > The contents will be stored verbatim without compression and without\n> > any object header (i.e., the usual \" \\0\") and the file\n> > could be \"ln\"ed (or \"cow\"ed if the underlying filesystem allows it)\n> > to materialize it in the working tree if needed.\n> >\n> > \"fsck\" needs to be told about how to verify them. Create the object\n> > header in-core and hash that, followed by the contents of that file,\n> > and make sure the result matches the part of the\n> > filename, or something like that.\n> \n> What happens when they are transferred? Should the remote unpack them\n> into the same kind of verbatim object?\n\nI think that the design space is vast and needs to be discussed, perhaps\nindependently of the local repo case (in which for a start, we could\njust detect large blobs being added to the index and put them in our\nnew object store instead of loose/packed storage, and make sure that we\nnever repack them). Some concerns during fetch:\n\n- Servers would probably want to serve the large blobs via CDN, so we\nprobably need something similar to packfile-uris. Would servers also\nwant to inline these blobs? (If not, we don't need to design this part.)\n\n- Would servers be willing to zlib-compress large blobs (into packfile\nformat) if the client doesn't support verbatim objects?\n\nAnd during push:\n\n- Clients probably want to be able to inline large blobs when pushing.\nShould it also be possible to specify the large blob via URI, and if\nyes, how does the server tell the client what URIs are acceptable?\n"},{"id":"482594","messageId":"20231003190955.GA1562@coredump.intra.peff.net","threadId":"60272","inReplyTo":"20230928214010.3502838-1-jonathantanmy@google.com","subject":"Re: [silly] loose, pack, and another thing?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-10-03T19:09:55Z","receivedAt":"2023-10-03T19:10:03Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Sep 28, 2023 at 02:40:10PM -0700, Jonathan Tan wrote:\n\n> Junio C Hamano writes:\n> > Just wondering if it would help to have the third kind of object\n> > representation in the object database, sitting next to loose objects\n> > and packed objects, say .git/objects/verbatim/ for\n> > the contents and .git/objects/verbatim/.type that\n> > records \"blob\", \"tree\", \"commit\", or \"tag\" (in practice, I would\n> > expect huge \"blob\" objects would be the only ones that use this\n> > mechanism).\n> > \n> > The contents will be stored verbatim without compression and without\n> > any object header (i.e., the usual \" \\0\") and the file\n> > could be \"ln\"ed (or \"cow\"ed if the underlying filesystem allows it)\n> > to materialize it in the working tree if needed.\n> \n> This sounds like a useful feature. We probably would want to use the\n> \"ln\" or \"cow\" every time we use streaming (stream_blob_to_fd() in\n> streaming.h) currently, so hopefully we won't need to increase the\n> number of ways in which we can write an object to the worktree (just\n> change the streaming to write to a filename instead of an fd).\n\nOne thing that scares me about a regular \"ln\" between the worktree and\nodb is that you are very susceptible to corrupting the repository by\nmodifying the worktree file with regular tools. If they do a complete\nrewrite and atomic rename (or link) to put the new file in place, that\nis OK. But opening the file for appending, or general writing, is bad.\n\nYou can get some safety with the immutable attribute (which applies to\nthe inode itself, and thus any path that hardlinks to it). But setting\nthat usually requires being root. And it creates other irritations for\nnormal use (you have to unset it before even removing the hardlink).\n\nIt would be nice if there was some portable copy-on-write abstraction we\ncould rely on, but I don't think there is one.\n\n-Peff\n"},{"id":"482622","messageId":"xmqqo7hfwg6m.fsf@gitster.g","threadId":"60272","inReplyTo":"20231003190955.GA1562@coredump.intra.peff.net","subject":"Re: [silly] loose, pack, and another thing?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-10-03T21:26:41Z","receivedAt":"2023-10-03T21:26:48Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> One thing that scares me about a regular \"ln\" between the worktree and\n> odb is that you are very susceptible to corrupting the repository by\n> modifying the worktree file with regular tools. If they do a complete\n> rewrite and atomic rename (or link) to put the new file in place, that\n> is OK. But opening the file for appending, or general writing, is bad.\n\nVery true.\n\n> You can get some safety with the immutable attribute (which applies to\n> the inode itself, and thus any path that hardlinks to it). But setting\n> that usually requires being root. And it creates other irritations for\n> normal use (you have to unset it before even removing the hardlink).\n\nAs a regular user, \"chmod a-w\" has the same characteristics (works\nat the inode level) but without \"cannot remove it\" downside. It\nused to be sufficient in RCS and CVS days, though, as a signal that\nyou are only to look at it without touching it, to \"chmod a-w\" a\npath that is checked out but not for modifying. Some editors even\noffer to do chmod u+w for you when saving, so if we want absolute\nsafety, it may not be enough.\n\n> It would be nice if there was some portable copy-on-write abstraction we\n> could rely on, but I don't think there is one.\n\n;-)\n"},{"id":"482641","messageId":"20231004131110.GA607079@coredump.intra.peff.net","threadId":"60272","inReplyTo":"xmqqo7hfwg6m.fsf@gitster.g","subject":"Re: [silly] loose, pack, and another thing?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2023-10-04T13:11:10Z","receivedAt":"2023-10-04T13:11:15Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Oct 03, 2023 at 02:26:41PM -0700, Junio C Hamano wrote:\n\n> > You can get some safety with the immutable attribute (which applies to\n> > the inode itself, and thus any path that hardlinks to it). But setting\n> > that usually requires being root. And it creates other irritations for\n> > normal use (you have to unset it before even removing the hardlink).\n> \n> As a regular user, \"chmod a-w\" has the same characteristics (works\n> at the inode level) but without \"cannot remove it\" downside. It\n> used to be sufficient in RCS and CVS days, though, as a signal that\n> you are only to look at it without touching it, to \"chmod a-w\" a\n> path that is checked out but not for modifying. Some editors even\n> offer to do chmod u+w for you when saving, so if we want absolute\n> safety, it may not be enough.\n\nAh, right. For some reason I was thinking that only affected the link\nentry, but of course the mode bits are on the linked inode itself. So\nthat does easily give some protection, though I agree that many programs\nare happy to circumvent it for you.\n\nIt has been a long time since I've used it, but I think there may be\nsome prior at in git-annex:\n\n https://git-annex.branchable.com/\n\nIIRC it can work in a \"copy\" mode where contents are copied into the\nworking tree. But since the point is to deal with large data sets, it\nalso has a linking mode (maybe even symlinks?) that point directly from\nthe working tree into the annex storage. If we are considering a similar\nfeature, we might be able to learn from their experience.\n\n-Peff\n"}]}