{"thread":{"id":"61981","subject":"Representing Debian Metadata in Git","startedAt":"2024-08-20T07:10:41Z","lastAt":"2024-08-23T02:20:45Z","messageCount":6,"participants":["Simon Richter","Chris Hofstaedtler","rsbecker@nexbridge.com","Blair Noctis","Sean Whitton"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"501323","messageId":"eb963843-9e2c-49f2-911f-fa36f33f9bfd@debian.org","threadId":"61981","inReplyTo":null,"subject":"Representing Debian Metadata in Git","fromName":"Simon Richter","fromEmail":"sjr@debian.org","sentAt":"2024-08-20T07:10:34Z","receivedAt":"2024-08-20T07:10:41Z","isPatch":false,"sender":{"key":"sjr@debian.org","avatar":null},"body":"Hi,\n\nthere's a bit of a discussion within Debian on collaborating using Git.\n\nOne of the long-standing issues is that there are multiple ways Debian \npackaging can be represented in a git tree, and none of them are optimal.\n\nThe problem at hand is that the packaging workflow consists of\n\n1. importing an upstream release\n2. optionally stripping out undistributable parts\n3. adding packaging metadata\n4. optionally adding a patch stack\n\nThe workflow for upgrading a package is\n\n1. import a new upstream release\n2. apply and possibly modify the exclusion list\n3. apply the packaging metadata, updating it in the process\n4. rebase the patch stack\n\nRight now, git is used mainly as a network file system, and only tagged \nreleases are expected to be consistent enough to compile, because often \ngoing from one consistent state to another as an atomic operation would \nrequire multiple changes to be applied in the same commit.\n\nThe imported archive is represented either directly as a tree (which may \nbe imported from the upstream project if no files are undistributable \nfor Debian), or via a mechanism that can reproduce a compressed archive \nthat is bitwise identical to the upstream release, from a tree and some \nadditional patch data.\n\nThe patch stack is stored as a set of patches inside a directory, and \nrebased using quilt.\n\nAn alternate representation stores the patch stack as a branch that is \nrebased using git, and then exported to single files.\n\nThe Debian changelog is stored as a file inside Git, but some automation \nexists to update this from Git commit messages.\n\nDebian changelog entries refer to bugs in the Debian Bug Tracking \nsystem. There is a desire to also incorporate forges (currently, GitLab) \nand refer to the forges' issue tracker from commit messages (where the \nissue tracker is used for team collaboration, while the Debian BTS is \nused for user-visible bugs).\n\nAll of this is very silly, because we're essentially storing metadata as \ndata because we cannot express in Git what we're actually doing, and the \nconflicting priorities people have have led to conflicting solutions.\n\nI'd like to xkcd 927 this now, and find a common mapping.\n\n From a requirements perspective, I'd like to be able to\n\n  - express patches as commits:\n    - allow cherry-picking upstream commits as Debian patches\n    - allow cherry-picking Debian patches for upstream submission\n  - generate the Debian changelog from changes committed to Git\n  - express filter steps for generating the upstream archive(s) from a \ntree‑ish and some metadata\n  - store upstream signatures inside Git\n  - keep a history of patches, including patches applied to previously \nreleased packages\n\nA possible implementation would be a type of Git \"user extension\" object \nthat contains\n\n  - an extension name\n  - an object type (interpreted by the extension)\n  - type-tagged references to other objects\n  - other type-tagged data\n\nValidity of the object would be determined by the extension, and git \nwould treat this object as mostly opaque (i.e. whenever one is \nencountered, the extension needs to be called). The only exception would \nbe references, because we need to be able to transfer these objects and \nall their dependencies efficiently (so the extension would generate a \nlist of references that should be recursively packed or omitted).\n\nOn top of that, we could represent a Debian package through special \nobjects, such as\n\n  - debian::debian-dir (a tree-like object referenced from the root \ntree, contains a tree for plain files plus links to special objects for \ngenerated items, such as patch stacks)\n  - debian::upstream-archive (a tree-like object that marks the boundary \nbetween objects imported from upstream, and objects that are part of \npackaging, and gives instructions for regenerating the upstream archives \nwithout storing them as blobs)\n  - debian::update-upstream (a commit-like object to move to a new \nupstream-archive object, this contains the upstream version number that \nthe following upload object must use)\n  - debian::changelog-entry (a commit-like object that adds an item to \nthe Debian changelog)\n  - debian::upload (a commit-like object that adds a version to the \nDebian changelog)\n  - debian::rebase-patches (a commit-like object that links the patch \nstacks before and after a rebase)\n  - ...\n\nChanges to packaging would still be represented as commit objects \ncontaining a tree, but that tree would contain a special entry for the \n\"debian\" subdirectory that points to the last packaging change.\n\nThis is very high-level so far, because I'd like to get some feedback \nfirst on whether it makes sense to pursue this further. This would use \nup the last unused three-bit object type in Git, so it will have to be \nvery generic on this side to not block future development -- and it \nwould require a lot of design effort on the Debian side as well to \nhammer out the details.\n\nAny feelings/objections/missed requirements?\n\n    Simon\n"},{"id":"501464","messageId":"gbgpxxpikhkykvhug4ugbufqm6bdh44ygiknpzc3khalibwutk@jlebwg4vsvjt","threadId":"61981","inReplyTo":"eb963843-9e2c-49f2-911f-fa36f33f9bfd@debian.org","subject":"Re: Representing Debian Metadata in Git","fromName":"Chris Hofstaedtler","fromEmail":"zeha@debian.org","sentAt":"2024-08-21T21:37:38Z","receivedAt":"2024-08-21T22:12:12Z","isPatch":false,"sender":{"key":"zeha@debian.org","avatar":"https://gravatar.com/avatar/7c1b1c4f260e576d7a60fca4bbc82fb1d78904167c48e3df3138e3eea4085260?d=mp&s=160"},"body":"Hi Simon,\n\n* Simon Richter <sjr@debian.org> [240820 09:11]:\n> One of the long-standing issues is that there are multiple ways Debian\n> packaging can be represented in a git tree, and none of them are optimal.\n[..]\n> A possible implementation would be a type of Git \"user extension\" object\n> that contains\n> \n>  - an extension name\n>  - an object type (interpreted by the extension)\n>  - type-tagged references to other objects\n>  - other type-tagged data\n[..]\n\n> Any feelings/objections/missed requirements?\n\nIn the current DEP14/DEP18 discussions a lot of discussion was had\nabout how we should represent Debian things in git; your mail also\ngoes into this direction.\n\nMy *feeling* is we should do the opposite - that is, represent less\nDebian stuff in git, and especially do it in less Debian-specific\nways. IOW, no git extensions, no setup with multiple branches that\ncontain more or less unrelated things, etc.\n\nI think we should move more towards a setup that is easily\nunderstood by people not closely following our Debian-specific\nthings. We should avoid surprising things, again that would include\nthe multiple branches and any git extensions.\n\nBefore pushing for new ways of representing Debian stuff in git, I\nthink it would be a good idea to learn from all the other distros\nand distro-like systems successfully using git [1]. Debian is not\nthe only distro that wants to use git to capture changes and\nencourage contributions to its packages.\n\nChris\n\n[1] alpine, homebrew, freebsd ports come to mind immediately. nixos\nand others too.\n\n"},{"id":"501465","messageId":"029801daf41f$7e994b70$7bcbe250$@nexbridge.com","threadId":"61981","inReplyTo":"gbgpxxpikhkykvhug4ugbufqm6bdh44ygiknpzc3khalibwutk@jlebwg4vsvjt","subject":"RE: Representing Debian Metadata in Git","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2024-08-21T23:11:40Z","receivedAt":"2024-08-21T23:12:33Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On Wednesday, August 21, 2024 5:38 PM, Chris Hofstaedtler wrote:\n>* Simon Richter <sjr@debian.org> [240820 09:11]:\n>> One of the long-standing issues is that there are multiple ways Debian\n>> packaging can be represented in a git tree, and none of them are optimal.\n>[..]\n>> A possible implementation would be a type of Git \"user extension\"\n>> object that contains\n>>\n>>  - an extension name\n>>  - an object type (interpreted by the extension)\n>>  - type-tagged references to other objects\n>>  - other type-tagged data\n>[..]\n>\n>> Any feelings/objections/missed requirements?\n>\n>In the current DEP14/DEP18 discussions a lot of discussion was had about how we\n>should represent Debian things in git; your mail also goes into this direction.\n>\n>My *feeling* is we should do the opposite - that is, represent less Debian stuff in git,\n>and especially do it in less Debian-specific ways. IOW, no git extensions, no setup\n>with multiple branches that contain more or less unrelated things, etc.\n>\n>I think we should move more towards a setup that is easily understood by people\n>not closely following our Debian-specific things. We should avoid surprising things,\n>again that would include the multiple branches and any git extensions.\n>\n>Before pushing for new ways of representing Debian stuff in git, I think it would be a\n>good idea to learn from all the other distros and distro-like systems successfully\n>using git [1]. Debian is not the only distro that wants to use git to capture changes\n>and encourage contributions to its packages.\n\nOn the other side (perhaps), git is increasingly being used in the Ops setting for\nDevOps and DevSecOps. Production configurations for high-value applications are\nmoving to storing those configurations into git for tracing and audit. Git is an\nenabler for good production operations practices. My $0.02 (and my customers')\n\n--Randall\n\n"},{"id":"501467","messageId":"z574b3k5mld3oshnlwho5jlpvgagkra6p2kmy2fe6urfhvtjeq@fivivkehco4o","threadId":"61981","inReplyTo":"029801daf41f$7e994b70$7bcbe250$@nexbridge.com","subject":"Re: Representing Debian Metadata in Git","fromName":"Chris Hofstaedtler","fromEmail":"zeha@debian.org","sentAt":"2024-08-21T23:51:23Z","receivedAt":"2024-08-21T23:51:30Z","isPatch":false,"sender":{"key":"zeha@debian.org","avatar":"https://gravatar.com/avatar/7c1b1c4f260e576d7a60fca4bbc82fb1d78904167c48e3df3138e3eea4085260?d=mp&s=160"},"body":"* rsbecker@nexbridge.com <rsbecker@nexbridge.com> [240822 01:21]:\n> >> Any feelings/objections/missed requirements?\n> >\n> >In the current DEP14/DEP18 discussions a lot of discussion was had about how we\n> >should represent Debian things in git; your mail also goes into this direction.\n> >\n> >My *feeling* is we should do the opposite - that is, represent less Debian stuff in git,\n> >and especially do it in less Debian-specific ways. IOW, no git extensions, no setup\n> >with multiple branches that contain more or less unrelated things, etc.\n>[..]\n\n> On the other side (perhaps), git is increasingly being used in the Ops setting for\n> DevOps and DevSecOps. Production configurations for high-value applications are\n> moving to storing those configurations into git for tracing and audit. Git is an\n> enabler for good production operations practices.\n\nDon't get me wrong. Yes, we should use git to do what git is good\nfor (tracking changes, etc).\n\nWe should not invent new ways of using git that no one else uses.\nI'd like to reduce the delta of \"how Debian uses git\" to \"how\neveryone else uses git\" to, hopefully, zero.\n\nChris\n\n"},{"id":"501559","messageId":"5b59d9f0-74a2-418c-acd7-ff1ecbfea467@sail.ng","threadId":"61981","inReplyTo":"eb963843-9e2c-49f2-911f-fa36f33f9bfd@debian.org","subject":"Re: Representing Debian Metadata in Git","fromName":"Blair Noctis","fromEmail":"ncts@debian.org","sentAt":"2024-08-22T19:27:31Z","receivedAt":"2024-08-22T19:28:14Z","isPatch":false,"sender":{"key":"ncts@debian.org","avatar":null},"body":"On 2024-08-20 15:10, Simon Richter wrote:\n(...)\n> Right now, git is used mainly as a network file system, and only tagged \n> releases are expected to be consistent enough to compile, because often \n> going from one consistent state to another as an atomic operation would \n> require multiple changes to be applied in the same commit.\n> \n> The imported archive is represented either directly as a tree (which may \n> be imported from the upstream project if no files are undistributable \n> for Debian), or via a mechanism that can reproduce a compressed archive \n> that is bitwise identical to the upstream release, from a tree and some \n> additional patch data.\n> \n> The patch stack is stored as a set of patches inside a directory, and \n> rebased using quilt.\n> \n> An alternate representation stores the patch stack as a branch that is \n> rebased using git, and then exported to single files.\n> \n> The Debian changelog is stored as a file inside Git, but some automation \n> exists to update this from Git commit messages.\n> \n> Debian changelog entries refer to bugs in the Debian Bug Tracking \n> system. There is a desire to also incorporate forges (currently, GitLab) \n> and refer to the forges' issue tracker from commit messages (where the \n> issue tracker is used for team collaboration, while the Debian BTS is \n> used for user-visible bugs).\n> \n> All of this is very silly, because we're essentially storing metadata as \n> data because we cannot express in Git what we're actually doing, and the \n> conflicting priorities people have have led to conflicting solutions.\n> \n> I'd like to xkcd 927 this now, and find a common mapping.\n\nHere's my very likely very naive 2 cents: we are basically maintaining a \nfork for each non-native package.\n\nBeing a fork, a \"Debianized\" package can also live like other \"upstream\" \nforks: with its own branch based on the original, make necessary changes\nand record them as commits; merge original onto its own branch, dealing\nwith conflicts; maintain its own changelog; rinse and repeat.\n\nDebian-specific metadata can be represented structurally in commit \nmessages, or if necessary, (still) in a plain debian/ subdirectory that\nwon't conflict with upstream.\n\nThen,\n\n>  From a requirements perspective, I'd like to be able to\n> \n>   - express patches as commits:\n>     - allow cherry-picking upstream commits as Debian patches\n>     - allow cherry-picking Debian patches for upstream submission\n>   - generate the Debian changelog from changes committed to Git\n>   - express filter steps for generating the upstream archive(s) from a \n> tree‑ish and some metadata\n>   - store upstream signatures inside Git\n>   - keep a history of patches, including patches applied to previously \n> released packages\n\nthese are naturally met; and\n\n(...)\n> Changes to packaging would still be represented as commit objects \n> containing a tree, but that tree would contain a special entry for the \n> \"debian\" subdirectory that points to the last packaging change.\n\nno more needed.\n\n> This is very high-level so far, because I'd like to get some feedback \n> first on whether it makes sense to pursue this further.This would use \n> up the last unused three-bit object type in Git, so it will have to be \n> very generic on this side to not block future development -- and it \n> would require a lot of design effort on the Debian side as well to \n> hammer out the details.\n\nEven less thought out, but probably easier to implement once the design \nis finished. ;)\n\n-- \nSdrager,\nBlair Noctis\n\n"},{"id":"501569","messageId":"874j7cou8s.fsf@melete.silentflame.com","threadId":"61981","inReplyTo":"eb963843-9e2c-49f2-911f-fa36f33f9bfd@debian.org","subject":"Re: Representing Debian Metadata in Git","fromName":"Sean Whitton","fromEmail":"spwhitton@spwhitton.name","sentAt":"2024-08-23T02:20:19Z","receivedAt":"2024-08-23T02:20:45Z","isPatch":false,"sender":{"key":"spwhitton@spwhitton.name","avatar":null},"body":"Hello,\n\nI think that more than you realise of this already exists :)\n\nOn Tue 20 Aug 2024 at 04:10pm +09, Simon Richter wrote:\n\n> From a requirements perspective, I'd like to be able to\n>\n>  - express patches as commits:\n>    - allow cherry-picking upstream commits as Debian patches\n>    - allow cherry-picking Debian patches for upstream submission\n\ngit-debrebase and git-dpm already achieve this.\n\n>  - express filter steps for generating the upstream archive(s) from a tree‑ish\n>   and some metadata\n\nExcluded-Files in d/copyright is for this.\nI guess that you disprefer that because it's part of the tree, though.\n\n>  - store upstream signatures inside Git\n\nWell, there's signatures on their tags.\n\n>  - keep a history of patches, including patches applied to previously released\n>   packages\n\nThis is already there with git-debrebase and git-dpm, though it is a bit\nfiddly to dig it out.\n\n-- \nSean Whitton\n"}]}