{"thread":{"id":"54268","subject":"grokmirror-2.0 is available","startedAt":"2020-09-21T17:06:59Z","lastAt":"2020-09-21T17:06:59Z","messageCount":1,"participants":["Konstantin Ryabitsev"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"406057","messageId":"20200921170651.aszbydzvnj7l4y2w@chatter.i7.local","threadId":"54268","inReplyTo":null,"subject":"grokmirror-2.0 is available","fromName":"Konstantin Ryabitsev","fromEmail":"konstantin@linuxfoundation.org","sentAt":"2020-09-21T17:06:51Z","receivedAt":"2020-09-21T17:06:59Z","isPatch":false,"sender":{"key":"konstantin@linuxfoundation.org","avatar":"https://gravatar.com/avatar/7cb8827c6de56e1bd2dea16508c6708aa43feed3bf3813bcdacecdf96ceadd79?d=mp&s=160"},"body":"Hello:\n\nI am pleased to announce version 2.0 of kernel.org's git mirroring \nsoftware, grokmirror. This is a major rewrite that intentionally breaks \nthe upgrade path from grokmirror-1.x due to significant backend changes \nrequiring replica administrator's thoughtful consideration -- please see \nthe UPGRADING.rst document provided with this release.\n\n## New in grokmirror-2.0\n\n- Drop support for python < 3.6\n- Introduce \"object storage\" repositories that benefit from git-pack\n  delta islands and improve overall disk storage footprint (results will \n  directly depend on the number of forks).\n- Drop dependency on GitPython: use git calls directly for all operations\n- Remove progress bars to slim down dependencies (drops enlighten)\n- Make grok-pull operate in daemon mode (with -o) (see contrib for\n  systemd unit files). This is more efficient than the cron mode when\n  run very frequently.\n- Provide a socket listener for pubsub push updates (see contrib for\n  Google pubsubv1.py).\n- Merge fsck.conf and repos.conf into a single config file. This\n  requires creating a new configuration file after the upgrade. See\n  UPGRADING.rst for details.\n- Record and propagate HEAD position using the manifest file.\n- Add grok-bundle command to create clone.bundle files for CDN-offloaded\n  cloning (mostly used by Android's repo command).\n- Add SELinux policy for EL7 (see contrib).\n\n## Object Storage Repositories\n\nGrokmirror 2.0 introduces the concept of \"object storage repositories\", which\naims to optimize how repository forks are stored on disk and served to the\ncloning clients.\n\nWhen grok-fsck runs, it will automatically recognize related repositories by\nanalyzing their root commits. If it finds two or more related repositories, it\nwill set up a unified \"object storage\" repo and fetch all refs from each\nrelated repository into it.\n\nFor example, you can have two forks of linux.git:\n  torvalds/linux.git:\n    refs/heads/master\n    refs/tags/v5.0-rc3\n    ...\n\nand its fork:\n\n  maintainer/linux.git:\n    refs/heads/master\n    refs/heads/devbranch\n    refs/tags/v5.0-rc3\n    ...\n\nGrok-fsck will set up an object storage repository and fetch all refs from both\nrepositories:\n\n  objstore/[random-guid-name].git\n     refs/virtual/[sha1-of-torvalds/linux.git:12]/heads/master\n     refs/virtual/[sha1-of-torvalds/linux.git:12]/tags/v5.0-rc3\n     ...\n     refs/virtual/[sha1-of-maintainer/linux.git:12]/heads/master\n     refs/virtual/[sha1-of-maintainer/linux.git:12]/heads/devbranch\n     refs/virtual/[sha1-of-maintainer/linux.git:12]/tags/v5.0-rc3\n     ...\n\nThen both torvalds/linux.git and maintainer/linux.git with be configured to use\nobjstore/[random-guid-name].git via objects/info/alternates and repacked to\njust contain metadata and no objects.\n\nThe alternates repository will be repacked with \"delta islands\" enabled,\nwhich should help optimize clone operations for each \"sibling\"\nrepository.\n\nPlease see the example grokmirror.conf for more details about configuring\nobjstore repositories.\n\n## Space savings using object storage repositories\n\nAny disk space savings will depend on how many repositories are forks of \neach other. For git.kernel.org, which already aggressively used \nalternates for all linux.git forks, we saw reduction from 60GB to 20GB \nfor the entirety of git.kernel.org content. On some of the \ncodeaurora.org systems, especially those containing a lot of pre-release \nforks of entire AOSP repo collections, we saw space usage go from 3TB to \nunder 1TB.\n\n## Stability\n\nThis release has proven pretty stable and has been operating on \ngit.kernel.org and a subset of codeaurora.org systems for over the past \nmonth. However, since the trickiest part is initial repository \nconversion towards the use of object storage repos, we urge proceeding \nwith caution. Please study the UPGRADING.rst document before making any \nchanges to your infrastructure.\n\nWith all support questions, please email tools@linux.kernel.org.\n\nBest regards,\nKonstantin\n"}]}