{"thread":{"id":"62549","subject":"git support for \"xattrs\" (extended filesystem attributes)?","startedAt":"2024-11-26T19:40:33Z","lastAt":"2024-11-27T20:57:35Z","messageCount":7,"participants":["Peter B.","Junio C Hamano","Jeff King","brian m. carlson"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"508193","messageId":"5b4c09a9-64bb-e672-e604-120563fc1ad6@das-werkstatt.com","threadId":"62549","inReplyTo":null,"subject":"git support for \"xattrs\" (extended filesystem attributes)?","fromName":"Peter B.","fromEmail":"pb@das-werkstatt.com","sentAt":"2024-11-26T19:40:32Z","receivedAt":"2024-11-26T19:40:33Z","isPatch":false,"sender":{"key":"pb@das-werkstatt.com","avatar":"https://gravatar.com/avatar/d8bf08d3a6e738366f791f1341ccdba6bf2c0d7d4b710b78633b7835aa46c9fc?d=mp&s=160"},"body":"Hi everyone :)\n\nI'm evaluating and testing extended attributes (xattrs) for professional \narchival collection use.\nI couldn't find any proper documentation/information on how (well) git \nsupports including xattrs in version control.\n\nI assume behavior may be similar to any other property-change of \nfiles/folders?\nWould it be possible to have bit-proof preservation of all xattrs? (even \nlarger ones).\n\nI'm grateful for any insights, pointers or comments. :D\n\n\nThanks in advance,\nPeter B.\n\n"},{"id":"508194","messageId":"xmqqr06xwzdc.fsf@gitster.g","threadId":"62549","inReplyTo":"5b4c09a9-64bb-e672-e604-120563fc1ad6@das-werkstatt.com","subject":"Re: git support for \"xattrs\" (extended filesystem attributes)?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-26T19:47:59Z","receivedAt":"2024-11-26T19:48:02Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Peter B.\" <pb@das-werkstatt.com> writes:\n\n> I'm evaluating and testing extended attributes (xattrs) for\n> professional archival collection use.\n> I couldn't find any proper documentation/information on how (well) git\n> supports including xattrs in version control.\n\nGit tracks contents, pathnames where these contents are stored, and\nthe executable bit.  Nothing else.  Not even the owner/group of each\ncontents, or time of creation or the last modification.\n\nI do not think Git is even aware of xattr at all.\n"},{"id":"508199","messageId":"20241126224124.GB77402@coredump.intra.peff.net","threadId":"62549","inReplyTo":"5b4c09a9-64bb-e672-e604-120563fc1ad6@das-werkstatt.com","subject":"Re: git support for \"xattrs\" (extended filesystem attributes)?","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2024-11-26T22:41:24Z","receivedAt":"2024-11-26T22:41:32Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Tue, Nov 26, 2024 at 08:40:32PM +0100, Peter B. wrote:\n\n> I'm evaluating and testing extended attributes (xattrs) for professional\n> archival collection use.\n> I couldn't find any proper documentation/information on how (well) git\n> supports including xattrs in version control.\n\nAs Junio noted, Git does not store most metadata at all (and most other\nversion control systems are the same).\n\nThere are some third party tools to save the information in a separate\nfile within the history and restore it on checkout. I haven't used them\nmyself and some of them are quite old, so YMMV:\n\n  - https://etckeeper.branchable.com/\n\n  - https://github.com/przemoc/metastore\n\n-Peff\n"},{"id":"508200","messageId":"xmqq4j3twnof.fsf@gitster.g","threadId":"62549","inReplyTo":"20241126224124.GB77402@coredump.intra.peff.net","subject":"Re: git support for \"xattrs\" (extended filesystem attributes)?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-11-27T00:00:32Z","receivedAt":"2024-11-27T00:00:35Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King <peff@peff.net> writes:\n\n> There are some third party tools to save the information in a separate\n> file within the history and restore it on checkout.\n\nAh, you're right of course.  We are extensible that way.\n\n> I haven't used them\n> myself and some of them are quite old, so YMMV:\n>\n>   - https://etckeeper.branchable.com/\n>\n>   - https://github.com/przemoc/metastore\n"},{"id":"508208","messageId":"Z0Z2DVuR2PiN-oxy@tapette.crustytoothpaste.net","threadId":"62549","inReplyTo":"5b4c09a9-64bb-e672-e604-120563fc1ad6@das-werkstatt.com","subject":"Re: git support for \"xattrs\" (extended filesystem attributes)?","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2024-11-27T01:29:49Z","receivedAt":"2024-11-27T01:29:59Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2024-11-26 at 19:40:32, Peter B. wrote:\n> Hi everyone :)\n> \n> I'm evaluating and testing extended attributes (xattrs) for professional\n> archival collection use.\n> I couldn't find any proper documentation/information on how (well) git\n> supports including xattrs in version control.\n> \n> I assume behavior may be similar to any other property-change of\n> files/folders?\n> Would it be possible to have bit-proof preservation of all xattrs? (even\n> larger ones).\n\nAs others have mentioned, Git doesn't store this.  There are a couple\nways that one could store this information, depending on what you want\nto store.\n\nOne way is to store the metadata in the `.gitattributes` file if you're\nstoring one or two well-known xattrs and the values are well stored as\ntext in the file.  You can then use `git ls-attr` and a `post-checkout`\nhook[0] to set those attributes appropriately.\n\nAnother way is to store the data as part of an mtree(1) file in the\nrepository.  mtree is a file format that comes from the BSDs that can\ncontain a wide variety of data, including file type, uid and gid, text\nuser and group, permissions, size, mtime, cryptographic hashes, and a\nwide variety of other data.  mtree utilities can usually generate a\nmanifest based on the state of the file system, validate that files\nmatch a manifest, and usually update existing files such that they match\na manifest.  There's a Go-based version[1] which does support storing\nxattrs.  Again, you can use a `post-checkout` hook here.\n\nSince mtree is a key-value format, you can also extend the value with\nother data.  I use a similar format in my dotfiles to specify install\nlocation, for instance, so I recommend this as a good way to store this\ndata.\n\n[0] Git LFS uses post-checkout hooks to adjust the read-only bit for\nfiles which are lockable but not locked, so this is generally a robust\nmethod.\n[1] https://github.com/vbatts/go-mtree\n-- \nbrian m. carlson (they/them or he/him)\nToronto, Ontario, CA\n"},{"id":"508214","messageId":"7746de08-2c4e-48bb-0a5f-4e6412a1f643@das-werkstatt.com","threadId":"62549","inReplyTo":"Z0Z2DVuR2PiN-oxy@tapette.crustytoothpaste.net","subject":"Re: git support for \"xattrs\" (extended filesystem attributes)?","fromName":"Peter B.","fromEmail":"pb@das-werkstatt.com","sentAt":"2024-11-27T07:59:30Z","receivedAt":"2024-11-27T08:04:34Z","isPatch":false,"sender":{"key":"pb@das-werkstatt.com","avatar":"https://gravatar.com/avatar/d8bf08d3a6e738366f791f1341ccdba6bf2c0d7d4b710b78633b7835aa46c9fc?d=mp&s=160"},"body":"Thank you everyone for your input!\nThis is very good now know.\n\nI'll take a look at your suggestions, especially mtree and metastore, as \nthey seem very useful in general.\n\n\nThanks and have a great day,\nPeter\n"},{"id":"508245","messageId":"f58d0ca9-305f-50e5-83f7-7939d310b8fc@das-werkstatt.com","threadId":"62549","inReplyTo":"Z0Z2DVuR2PiN-oxy@tapette.crustytoothpaste.net","subject":"Re: git support for \"xattrs\" (extended filesystem attributes)?","fromName":"Peter B.","fromEmail":"pb@das-werkstatt.com","sentAt":"2024-11-27T20:52:26Z","receivedAt":"2024-11-27T20:57:35Z","isPatch":false,"sender":{"key":"pb@das-werkstatt.com","avatar":"https://gravatar.com/avatar/d8bf08d3a6e738366f791f1341ccdba6bf2c0d7d4b710b78633b7835aa46c9fc?d=mp&s=160"},"body":"On 11/27/24 02:29, brian m. carlson wrote:\n> [...] There's a Go-based version[1] which does support storing\n> xattrs.  Again, you can use a `post-checkout` hook here.\n>\n> Since mtree is a key-value format, you can also extend the value with\n> other data.  I use a similar format in my dotfiles to specify install\n> location, for instance, so I recommend this as a good way to store this\n> data.\n\nI just installed gomtree, and read my xattrs: looks very promising!\nThank you for the suggestion. :)\n\nI'll read up on \"post-checkout hooks\" to see how I can integrate this in \nmy git setup.\n\n\nIn case anyone's curious, here's an example of gomtree's output text for \na file annotated using `yt-dlp --xattr`:\n\n```\nUniversal\\040Sequence\\040\\13320vPbH6UWIc].mp4 \nxattr.user.xdg.referrer.url=aHR0cHM6Ly93d3cueW91dHViZS5jb20vd2F0Y2g/dj0yMHZQYkg2VVdJYw== \nxattr.user.dublincore.title=VW5pdmVyc2FsIFNlcXVlbmNl \nxattr.user.dublincore.date=MjAxNi0wOS0xNA== \nxattr.user.dublincore.contributor=Q29uc3BpcmFjeQ== \nxattr.user.dublincore.format=Mjk4IC0gMTI4MHg1NDAgKDcyMHA2MCkrMTQwIC0gYXVkaW8gb25seSAobWVkaXVtKQ== \nxattr.user.dublincore.description=VmlkZW8gcmVjb3JkaW5nIG9mIHRoZSBmaW5hbCB2ZXJzaW9uIG9mICJVbml2ZXJzYWwgU2VxdWVuY2UiLCBhIDY0IEtCeXRlIHJlYWx0aW1lIGRlbW8gYnkgQ29uc3BpcmFjeS4gRmlyc3QgcHJlc2VudGVkIGF0IHRoZSBGdW5jdGlvbiAyMDE2IGRlbW9wYXJ0eSwgd2hlcmUgaXQgcmVhY2hlZCAxc3QgcGxhY2UuCgpEb3dubG9hZCB0aGUgcmVhbHRpbWUgdmVyc2lvbiBhbmQgc291bmR0cmFjayBhdApodHRwOi8vY29uc3BpcmFjeS5odS9yZWxlYXNlLzY0ay91bml2ZXJzYWxfc2VxdWVuY2UvCgpFbmdpbmUgY29kZSBhbmQgZWZmZWN0cyBieQpCT1lDCgpWaXN1YWxzIGFuZCBkaXJlY3Rpb24gYnkKWk9PTQoKU3ludGggYW5kIG11c2ljIGJ5CkdBUkdBSgoKaHR0cDovL3d3dy5jb25zcGlyYWN5Lmh1Cmh0dHA6Ly9mYWNlYm9vay5jb20vY29uc3BpcmFjeS5odQpodHRwOi8vdHdpdHRlci5jb20vY29uc3BpcmFjeWh1CgojY29uc3BpcmFjeSAjZGVtb3NjZW5lICM2NGsgI2ludHJvICNyZWFsdGltZSAjaHVuZ2FyeQ==\n```\n\nValues are base64-encoded.\nNice.\n\n\n\nRegards,\nPeter B.\n"}]}