{"thread":{"id":"58569","subject":"[filter-repo PATCH] contrib: new tool to convert svn:externals to Git submodules","startedAt":"2022-10-05T21:37:22Z","lastAt":"2022-10-06T08:40:46Z","messageCount":3,"participants":["Markus Heidelberg","Elijah Newren"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"464263","messageId":"20221005213633.7070-1-markus.heidelberg@web.de","threadId":"58569","inReplyTo":null,"subject":"[filter-repo PATCH] contrib: new tool to convert svn:externals to Git submodules","fromName":"Markus Heidelberg","fromEmail":"markus.heidelberg@web.de","sentAt":"2022-10-05T21:36:33Z","receivedAt":"2022-10-05T21:37:22Z","isPatch":true,"sender":{"key":"markus.heidelberg@web.de","avatar":"https://avatars.githubusercontent.com/u/6334512?v=4"},"body":"This is meant to be a post-processing step after SVN-to-Git conversion\nby SubGit (https://subgit.com/), which creates a \".gitsvnextmodules\"\nfile that we will use for svn:externals conversion.\n\nSigned-off-by: Markus Heidelberg <markus.heidelberg@web.de>\n---\n\nThe SVN-to-Git mapping file format might have to be changed in the\nfuture when implementing other features, don't know yet. At least I\nwanted to publish this script now that the basics work. Maybe there are\neven some other late-adopters for whose this script might still be\nuseful.\n\nI'm eager for hints where descriptions/comments are confusing or\nincomplete.\n\nThis script was already used for migration of an SVN repository in our\ncompany.\n\nRelates to https://github.com/newren/git-filter-repo/issues/14\n\n contrib/filter-repo-demos/README.md           |   1 +\n .../filter-repo-demos/convert-svnexternals    | 587 ++++++++++++++++++\n 2 files changed, 588 insertions(+)\n create mode 100644 contrib/filter-repo-demos/convert-svnexternals\n\ndiff --git a/contrib/filter-repo-demos/README.md b/contrib/filter-repo-demos/README.md\nindex ef021b1..9612428 100644\n--- a/contrib/filter-repo-demos/README.md\n+++ b/contrib/filter-repo-demos/README.md\n@@ -16,6 +16,7 @@ lint-history         |Run some lint command on all non-binary files in history.\n clean-ignore         |Delete files from history which match current gitignore rules.\n filter-lamely (or filter&#8209;branch&#8209;ish) |A nearly bug compatible re-implementation of filter-branch (the git testsuite passes using it instead of filter-branch), with some performance tricks to make it several times faster (though it's still glacially slow compared to filter-repo).\n bfg-ish              |A re-implementation of most of BFG Repo Cleaner, with new features and bug fixes.\n+convert-svnexternals |Insert Git submodules according to SVN externals.\n\n ## Purpose\n\ndiff --git a/contrib/filter-repo-demos/convert-svnexternals b/contrib/filter-repo-demos/convert-svnexternals\nnew file mode 100644\nindex 0000000..0c81507\n--- /dev/null\n+++ b/contrib/filter-repo-demos/convert-svnexternals\n@@ -0,0 +1,587 @@\n+#!/usr/bin/env python3\n+\n+\"\"\"\n+This is a program that will insert Git submodules according to SVN externals\n+definitions (svn:externals properties) from the original Subversion repository\n+throughout the history.\n+\n+Information about the externals is obtained from the \".gitsvnextmodules\" file\n+created during SVN-to-Git conversion by SubGit (https://subgit.com/). Its\n+config option \"translate.externals=true\" had to be used therefore.\n+\n+Actual modifications:\n+- Insert gitlinks (mode 160000) into the tree.\n+- Add .gitmodules file with relevant sections.\n+- Remove sections converted to submodules from .gitsvnextmodules file\n+  and delete it if empty.\n+\n+.gitsvnextmodules example:\n+[submodule \"somedir/extdir\"]\n+\tpath = somedir/extdir\n+\towner = somedir\n+\turl = https://svn.example.com/somesvnrepo/trunk\n+\trevision = 1234\n+\tbranch = /\n+\tfetch = :refs/remotes/git-svn\n+\tremote = svn\n+\ttype = dir\n+\n+Resulting addition in \"somedir\" tree (cat-file pretty-print format):\n+160000 commit 1234123412341234123412341234123412341234\textdir\n+\n+Resulting .gitmodules entry:\n+[submodule \"somedir/extdir\"]\n+\tpath = somedir/extdir\n+\turl = https://git.example.com/somegitrepo.git\n+\n+SVN-to-Git mapping file:\n+Can be created from SubGit's \"refs/svn/map\".\n+One line per mapping in following format:\n+<svn url> TAB <svn rev> TAB <git url> TAB <git commit> TAB <state>\n+- Leading '#' can be used for comments.\n+- <svn url> must not contain a trailing slash.\n+- <state> has to be \"commit\" to be usable, but can be \"missing\" if <git commit>\n+  does not exist in the repository anymore. Adopted from git-cat-file output.\n+Example:\n+https://svn.example.com/somesvnrepo/trunk\t1234\thttps://git.example.com/somegitrepo.git\t1234123412341234123412341234123412341234\tcommit\n+\n+Features:\n+- Repeatedly added/removed externals will be handled properly.\n+- Externals replaced by directly added files and vice versa will be handled\n+  properly.\n+\n+Caveats:\n+- This script must NOT be run repeatedly. A second invocation would lead to a\n+  different result in case the externals could only be converted partially.\n+- Inconsistent SVN repositories (with failing checkout) not handled, i.e.\n+  - normal directory and external with the same path\n+  - external path not existing for the given revision\n+- No attention was paid to non-ASCII and special characters in gitlink paths,\n+  might cause problems.\n+- There is no error handling for mandatory options missing in .gitsvnextmodules\n+  file. The script would crash in case of such buggy files, but that shouldn't\n+  happen in practice.\n+\n+TODO:\n+- Add external files directly.\n+- Alternatively add external directories directly instead of using a submodule.\n+\"\"\"\n+\n+\"\"\"\n+Please see the\n+  ***** API BACKWARD COMPATIBILITY CAVEAT *****\n+near the top of git-filter-repo.\n+\"\"\"\n+\n+import argparse\n+import os\n+import sys\n+import shutil\n+import subprocess\n+import configparser\n+from urllib.parse import urlsplit\n+\n+try:\n+  import git_filter_repo as fr\n+except ImportError:\n+  raise SystemExit(\"Error: Couldn't find git_filter_repo.py.  Did you forget to make a symlink to git-filter-repo named git_filter_repo.py or did you forget to put the latter in your PYTHONPATH?\")\n+\n+svn_root_url = \"\"\n+svn_git_mappings = []\n+\n+def parse_args():\n+  \"\"\"\n+  Parse and return arguments for this script.\n+\n+  Also do some argument sanity checks and adaptions.\n+  \"\"\"\n+  parser = argparse.ArgumentParser(\n+      description=\"Add Git submodules according to svn:externals from .gitsvnextmodules. \"\n+                  \"As preparation for this conversion process, an analysis can be performed.\")\n+\n+  parser.add_argument('--force', '-f', action='store_true',\n+      help=\"Rewrite repository history even if the current repo does not \"\n+           \"look like a fresh clone.\")\n+  parser.add_argument('--refs', nargs='+',\n+      help=\"Limit history rewriting to the specified refs. Option is directly \"\n+           \"forwarded to git-filter-repo, see there for details and caveats. \"\n+           \"Use for debugging purposes only!\")\n+  parser.add_argument('--svn-root-url',\n+      help=\"Root URL of the corresponding SVN repository, \"\n+           \"needed for conversion of relative to absolute external URLs.\")\n+\n+  analysis = parser.add_argument_group(title=\"Analysis\")\n+  analysis.add_argument('--analyze', action='store_true',\n+      help=\"Analyze repository history and create auxiliary files for conversion process.\")\n+  analysis.add_argument('--report-dir', type=os.fsencode,\n+      help=\"Directory to write report, defaults to GIT_DIR/filter-repo/svnexternals, \"\n+           \"refuses to run if exists, --force delete existing dir first.\")\n+\n+  conversion = parser.add_argument_group(title=\"Conversion\")\n+  conversion.add_argument('--svn-git-mapfiles', type=os.fsencode, nargs='+', metavar='MAPFILE',\n+      help=\"Files with SVN-to-Git revision mappings for SVN externals conversion.\")\n+\n+  args = parser.parse_args()\n+\n+  if args.analyze and args.svn_git_mapfiles:\n+    raise SystemExit(\"Error: --svn-git-mapfiles makes no sense with --analyze.\")\n+\n+  if not args.analyze and not args.svn_git_mapfiles:\n+    raise SystemExit(\"Error: --svn-git-mapfiles is required for the conversion process.\")\n+\n+  return args\n+\n+def read_mappings(mapfiles):\n+  \"\"\"\n+  Read files with SVN-to-Git mappings and return a list of mappings from it.\n+  \"\"\"\n+  mappings = []\n+  for mapfile in mapfiles:\n+    with open(mapfile, \"rb\") as f:\n+      for line in f:\n+        line = line.rstrip(b'\\r\\n')\n+\n+        # Skip blank and comment lines\n+        if not line or line.startswith(b'#'):\n+          continue\n+\n+        # Convert to string for use with configparser later\n+        line = line.decode()\n+\n+        # Parse the line\n+        fields = line.split('\\t', 4)\n+        mapping = {'svn_url': fields[0],\n+                   'svn_rev': int(fields[1]),\n+                   'git_url': fields[2],\n+                   'git_commit': fields[3],\n+                   'state': fields[4]}\n+\n+        mappings.append(mapping)\n+  return mappings\n+\n+cat_file_process = None\n+def parse_config(blob_id):\n+  \"\"\"\n+  Create a configparser object for a .gitsvnextmodules/.gitmodules file from\n+  its blob ID.\n+  \"\"\"\n+  parsed_config = configparser.ConfigParser()\n+\n+  if blob_id is not None:\n+    # Get the blob contents\n+    cat_file_process.stdin.write(blob_id + b'\\n')\n+    cat_file_process.stdin.flush()\n+    objhash, objtype, objsize = cat_file_process.stdout.readline().split()\n+    contents_plus_newline = cat_file_process.stdout.read(int(objsize)+1)\n+\n+    # Parse it\n+    parsed_config.read_string(contents_plus_newline.decode())\n+\n+  return parsed_config\n+\n+def create_blob(parsed_config):\n+  \"\"\"\n+  Create a filter-repo blob object from a .gitsvnextmodules/.gitmodules\n+  configparser object according to Git config style.\n+  \"\"\"\n+  lines = []\n+  for sec in parsed_config.sections():\n+    lines.append(\"[\" + sec + \"]\\n\")\n+    for opt in parsed_config.options(sec):\n+      lines.append(\"\\t\" + opt + \" = \" + parsed_config[sec][opt] + \"\\n\")\n+\n+  return fr.Blob(''.join(lines).encode())\n+\n+def get_git_url(svn_url):\n+  \"\"\"\n+  Get the Git URL for a corresponding SVN URL.\n+  \"\"\"\n+  for entry in svn_git_mappings:\n+    if entry['svn_url'] == svn_url:\n+      return entry['git_url']\n+  else:\n+    return None\n+\n+def get_git_commit_hash(svn_url, svn_rev):\n+  \"\"\"\n+  Get the Git commit hash for its corresponding SVN URL+revision.\n+\n+  The mapping is not restricted to the exact revision, but also uses the next\n+  lower revision found. Needed when the revision was set to that of the root\n+  URL instead of to that of the specific subdirectory (e.g. trunk). TortoiseSVN\n+  behaves so when setting the external to HEAD.\n+  \"\"\"\n+  ent = None\n+  rev = 0\n+\n+  for entry in svn_git_mappings:\n+    if (entry['svn_url'] == svn_url\n+          and entry['svn_rev'] <= svn_rev\n+          and entry['svn_rev'] > rev):\n+      ent = entry\n+      rev = entry['svn_rev']\n+\n+  if ent is not None and ent['state'] == \"commit\":\n+    return ent['git_commit']\n+  else:\n+    return None\n+\n+def get_absolute_svn_url(svnext_url, svn_root_url):\n+  \"\"\"\n+  Convert a relative svn:externals URL to an absolute one.\n+\n+  If the format is unsupported, return the URL unchanged with success=False.\n+  If no root URL is given or the URL is absolute already, return it unchanged.\n+\n+  In all cases, even if returned \"unchanged\", trailing slashes are removed.\n+  \"\"\"\n+  # Remove trailing slash(es)\n+  svnext_url = svnext_url.rstrip(\"/\")\n+  svn_root_url = svn_root_url.rstrip(\"/\")\n+\n+  # Normalize URLs in relative format\n+  svn_root_parsed = urlsplit(svn_root_url)\n+  if svnext_url.startswith((\"../\", \"^/../\")): # unsupported\n+    return (False, svnext_url)\n+  elif not svn_root_url:\n+    pass # unchanged\n+  elif svnext_url.startswith(\"^/\"):\n+    svnext_url = svn_root_url + svnext_url[1:]\n+  elif svnext_url.startswith(\"//\"):\n+    svnext_url = svn_root_parsed.scheme + \":\" + svnext_url\n+  elif svnext_url.startswith(\"/\"):\n+    svnext_url = svn_root_parsed.scheme + \"://\" + svn_root_parsed.netloc + svnext_url\n+\n+  return True, svnext_url\n+\n+def add_submodule_tree_entry(commit, parsed_config, section):\n+  \"\"\"\n+  Add a submodule entry to the tree of a Git commit.\n+\n+  SVN externals information obtained from parsed .gitsvnextmodules file.\n+  \"\"\"\n+  # Skip type=file (SVN file external), not possible as submodule\n+  if parsed_config[section]['type'] != 'dir':\n+    return False\n+\n+  success, svn_url = get_absolute_svn_url(parsed_config[section]['url'], svn_root_url)\n+  # Skip unsupported URL format\n+  if not success:\n+    return False\n+\n+  # Get SVN revision\n+  if parsed_config.has_option(section, 'revision'):\n+    svn_rev = int(parsed_config[section]['revision'])\n+  else:\n+    # TODO: revision has to be guessed according to commit timestamp, skip for now\n+    return False\n+\n+  # SVN url+revision mapping to Git commit\n+  git_hash = get_git_commit_hash(svn_url, svn_rev)\n+  # Skip missing or unusable mapping\n+  if git_hash is None:\n+    return False\n+  git_hash = git_hash.encode()\n+\n+  dirname = parsed_config[section]['path'].encode()\n+\n+  # Add gitlink to tree\n+  commit.file_changes.append(fr.FileChange(b'M', dirname, git_hash, b'160000'))\n+\n+  return True\n+\n+def get_commit_map_path():\n+  \"\"\"\n+  Return path to commit-map file.\n+  \"\"\"\n+  git_dir = fr.GitUtils.determine_git_dir(b'.')\n+  return os.path.join(git_dir, b'filter-repo', b'commit-map')\n+\n+def parse_commit_map(commit_map_file):\n+  \"\"\"\n+  Parse commit-map file and return a dictionary.\n+  \"\"\"\n+  parsed_map = {}\n+  with open(commit_map_file, \"rb\") as f:\n+    for line in f:\n+      line = line.rstrip(b'\\r\\n')\n+\n+      # Skip blank lines\n+      if not line:\n+        continue\n+\n+      # Store old/new commits, also the \"old\"/\"new\" header in the first line\n+      old, new = line.split()\n+      parsed_map[old] = new\n+  return parsed_map\n+\n+def merge_commit_maps(old_commit_map, new_commit_map):\n+  \"\"\"\n+  Merge old and new commit-map by omitting intermediate commits.\n+\n+  Return the merged dictionary.\n+  \"\"\"\n+  merged_map = {}\n+  for (key, old_val) in old_commit_map.items():\n+    new_val = new_commit_map[old_val] if old_val in new_commit_map else old_val\n+    merged_map[key] = new_val\n+  return merged_map\n+\n+def write_commit_map(commit_map, commit_map_file):\n+  \"\"\"\n+  Write commit-map dictionary to file.\n+  \"\"\"\n+  with open(commit_map_file, 'wb') as f:\n+    for (old, new) in commit_map.items():\n+      f.write(b'%-40s %s\\n' % (old, new))\n+\n+def create_report_dir(args):\n+  \"\"\"\n+  Create the directory for analysis report.\n+  \"\"\"\n+  if args.report_dir:\n+    reportdir = args.report_dir\n+  else:\n+    git_dir = fr.GitUtils.determine_git_dir(b'.')\n+\n+  # Create the report directory as necessary\n+    results_tmp_dir = os.path.join(git_dir, b'filter-repo')\n+    if not os.path.isdir(results_tmp_dir):\n+      os.mkdir(results_tmp_dir)\n+    reportdir = os.path.join(results_tmp_dir, b'svnexternals')\n+\n+  if os.path.isdir(reportdir):\n+    if args.force:\n+      sys.stdout.write(\"Warning: Removing recursively: \\\"%s\\\"\" % fr.decode(reportdir))\n+      shutil.rmtree(reportdir)\n+    else:\n+      sys.stdout.write(\"Error: dir already exists (use --force to delete): \\\"%s\\\"\\n\" % fr.decode(reportdir))\n+      sys.exit(1)\n+\n+  os.mkdir(reportdir)\n+\n+  return reportdir\n+\n+analysis = {'dir_ext_orig': [],\n+            'dir_ext_abs': [],\n+            'file_ext_orig': [],\n+            'file_ext_abs': []}\n+def write_analysis(reportdir):\n+  \"\"\"\n+  Prepare analysis and write it to files in report directory.\n+  \"\"\"\n+  analysis['dir_ext_orig'].sort()\n+  analysis['dir_ext_abs'].sort()\n+  analysis['file_ext_orig'].sort()\n+  analysis['file_ext_abs'].sort()\n+\n+  sys.stdout.write(\"Writing reports to %s...\" % fr.decode(reportdir))\n+  sys.stdout.flush()\n+\n+  with open(os.path.join(reportdir, b\"dir-externals-original.txt\"), 'wb') as f:\n+    for url in analysis['dir_ext_orig']:\n+      f.write((\"%s\\n\" % url).encode())\n+\n+  with open(os.path.join(reportdir, b\"dir-externals-absolute.txt\"), 'wb') as f:\n+    for url in analysis['dir_ext_abs']:\n+      f.write((\"%s\\n\" % url).encode())\n+\n+  with open(os.path.join(reportdir, b\"file-externals-original.txt\"), 'wb') as f:\n+    for url in analysis['file_ext_orig']:\n+      f.write((\"%s\\n\" % url).encode())\n+\n+  with open(os.path.join(reportdir, b\"file-externals-absolute.txt\"), 'wb') as f:\n+    for url in analysis['file_ext_abs']:\n+      f.write((\"%s\\n\" % url).encode())\n+\n+  sys.stdout.write(\"done.\\n\")\n+\n+def analyze_externals(commit, metadata):\n+  \"\"\"\n+  Generate/extend analysis of SVN externals for a Git commit.\n+\n+  Used as filter-repo commit callback.\n+  \"\"\"\n+  for change in commit.file_changes:\n+    if change.filename == b'.gitsvnextmodules' and change.type == b'M':\n+      gitsvnextmodules = parse_config(change.blob_id)\n+\n+      for sec in gitsvnextmodules.sections():\n+        url = gitsvnextmodules[sec]['url']\n+        success, abs_url = get_absolute_svn_url(url, svn_root_url)\n+\n+        # List of svn:externals URLs, also add the URL to the absolute list if\n+        # conversion was not successful\n+        if gitsvnextmodules[sec]['type'] == 'dir':\n+          if url not in analysis['dir_ext_orig']:\n+            analysis['dir_ext_orig'].append(url)\n+          if abs_url not in analysis['dir_ext_abs']:\n+            analysis['dir_ext_abs'].append(abs_url)\n+        else:\n+          if url not in analysis['file_ext_orig']:\n+            analysis['file_ext_orig'].append(url)\n+          if abs_url not in analysis['file_ext_abs']:\n+            analysis['file_ext_abs'].append(abs_url)\n+\n+def insert_submodules(commit, metadata):\n+  \"\"\"\n+  Insert submodules for a Git commit.\n+\n+  Used as filter-repo commit callback.\n+\n+  Since .gitsvnextmodules just contains the svn:externals state for the given\n+  commit, we cannot derive specific changes from that file.\n+  So we can only add/modify the gitlinks according to .gitsvnextmodules\n+  (without knowing whether adding a new or modifying an existing or even\n+  \"modifying\" an unchanged submodule, but none of that really matters).\n+  We do not have information about deleted externals, those will be handled in\n+  a separate filter run afterwards.\n+\n+  The .gitmodules file however will already be correct in this function because\n+  we don't need to know about specific changes to add, modify or delete it.\n+  \"\"\"\n+  for change in commit.file_changes:\n+    if change.filename == b'.gitsvnextmodules' and change.type in (b'M', b'D'):\n+      gitsvnextmodules = parse_config(change.blob_id)\n+      gitmodules = configparser.ConfigParser()\n+\n+      # Add gitlinks to the tree and prepare .gitmodules file content\n+      for sec in gitsvnextmodules.sections():\n+        if add_submodule_tree_entry(commit, gitsvnextmodules, sec):\n+          # Gitlink added\n+          # -> Add this entry to .gitmodules as well\n+\n+          # Create the section name string manually, do not rely on\n+          # .gitsvnextmodules to always use the proper section name.\n+          sec_name = 'submodule \"' + gitsvnextmodules[sec]['path'] + '\"'\n+          gitmodules[sec_name] = {}\n+\n+          # submodule.<name>.path\n+          gitmodules[sec_name]['path'] = gitsvnextmodules[sec]['path']\n+\n+          # submodule.<name>.url\n+          success, svn_url = get_absolute_svn_url(gitsvnextmodules[sec]['url'], svn_root_url)\n+          git_url = get_git_url(svn_url)\n+          if git_url is not None:\n+            gitmodules[sec_name]['url'] = git_url\n+          else:\n+            # Abort, but this will not happen in practice, catched in\n+            # add_submodule_tree_entry() via get_git_commit_hash() already.\n+            raise SystemExit(\"Error: No Git URL found in mapping although a commit hash could be found.\")\n+\n+      # Write blob and adapt tree for .gitmodules\n+      if gitmodules.sections():\n+        # Create a blob object from the content and add it to the tree.\n+        blob = create_blob(gitmodules)\n+        filter.insert(blob)\n+        commit.file_changes.append(fr.FileChange(b'M', b'.gitmodules', blob.id, b'100644'))\n+      else:\n+        # Delete the file, even if a \"git rm\" of all submodules keeps it empty.\n+        commit.file_changes.append(fr.FileChange(b'D', b'.gitmodules'))\n+\n+def delete_submodules(commit, metadata):\n+  \"\"\"\n+  Delete submodules from a Git commit.\n+\n+  Used as filter-repo commit callback.\n+\n+  Delete all submodules (inserted in the previous filter run) without an entry\n+  in .gitsvnextmodules, these were real deletions of externals, which couldn't\n+  be detected before.\n+  Only the tree entries have to be removed because the .gitmodules file is\n+  already in correct state from previous filter run.\n+  \"\"\"\n+  for change in commit.file_changes:\n+    if change.filename == b'.gitsvnextmodules' and change.type in (b'M', b'D'):\n+      gitsvnextmodules = parse_config(change.blob_id)\n+\n+      # Search for all submodules in the tree\n+      output = subprocess.check_output('git ls-tree -d -r -z'.split() + [commit.original_id])\n+      for line in output.split(b'\\x00'):\n+        if not line:\n+          continue\n+        mode_objtype_objid, dirname = line.split(b'\\t', 1)\n+        mode, objtype, objid = mode_objtype_objid.split(b' ')\n+        if mode == b'160000' and objtype == b'commit':\n+          # Submodule found\n+          # -> Delete it if there is no corresponding entry in\n+          #    .gitsvnextmodules, keep/reinsert it otherwise\n+          for sec in gitsvnextmodules.sections():\n+            if gitsvnextmodules[sec]['path'].encode() == dirname:\n+              # Reinsert it, might have been deleted in previous commits\n+              if add_submodule_tree_entry(commit, gitsvnextmodules, sec):\n+                # And remove the config section because this external has been\n+                # converted\n+                gitsvnextmodules.remove_section(sec)\n+                break\n+          else:\n+            # Delete it\n+            commit.file_changes.append(fr.FileChange(b'D', dirname))\n+\n+      # Rewrite .gitsvnextmodules to contain the unhandled externals only,\n+      # delete it if empty (all externals converted).\n+      if gitsvnextmodules.sections():\n+        # Create a blob object from the content and replace the original one.\n+        blob = create_blob(gitsvnextmodules)\n+        filter.insert(blob)\n+        change.blob_id = blob.id\n+      else:\n+        if change.type == b'M':\n+          # File became empty, delete it\n+          commit.file_changes.append(fr.FileChange(b'D', b'.gitsvnextmodules'))\n+          break # avoid endless for loop\n+        #else:\n+          # File was empty already, delete command already present in stream\n+\n+my_args = parse_args()\n+\n+# Use passed URL without trailing slash(es)\n+if my_args.svn_root_url:\n+  svn_root_url = my_args.svn_root_url.rstrip(\"/\")\n+\n+# Arguments forwarded to filter-repo\n+extra_args = []\n+if my_args.force:\n+  extra_args = ['--force']\n+if my_args.refs:\n+  extra_args += ['--refs'] + my_args.refs\n+\n+cat_file_process = subprocess.Popen(['git', 'cat-file', '--batch'],\n+                                    stdin = subprocess.PIPE,\n+                                    stdout = subprocess.PIPE)\n+if my_args.analyze:\n+  # Analysis\n+  reportdir = create_report_dir(my_args)\n+\n+  fr_args = fr.FilteringOptions.parse_args(['--dry-run']\n+                                           + extra_args)\n+  filter = fr.RepoFilter(fr_args, commit_callback=analyze_externals)\n+  filter.run()\n+\n+  write_analysis(reportdir)\n+else:\n+  # Conversion\n+  svn_git_mappings = read_mappings(my_args.svn_git_mapfiles)\n+\n+  # There are no references to commit hashes in commit messages because this\n+  # script runs on a Git repository converted from a Subversion repository.\n+  fr_args = fr.FilteringOptions.parse_args(['--preserve-commit-hashes',\n+                                            '--preserve-commit-encoding',\n+                                            '--replace-refs', 'update-no-add']\n+                                           + extra_args)\n+  filter = fr.RepoFilter(fr_args, commit_callback=insert_submodules)\n+  filter.run()\n+\n+  # Store commit-map after first run\n+  first_commit_map = parse_commit_map(get_commit_map_path())\n+\n+  filter = fr.RepoFilter(fr_args, commit_callback=delete_submodules)\n+  filter.run()\n+\n+  # Update commit-map after second run, based on original IDs\n+  second_commit_map = parse_commit_map(get_commit_map_path())\n+  merged_commit_map = merge_commit_maps(first_commit_map, second_commit_map)\n+  write_commit_map(merged_commit_map, get_commit_map_path())\n+\n+cat_file_process.stdin.close()\n+cat_file_process.wait()\n--\n2.37.3\n\n"},{"id":"464271","messageId":"CABPp-BEyemFxeCG-bmyiugfzCqZp=XbOJKVWrepjeTf9G5yHgQ@mail.gmail.com","threadId":"58569","inReplyTo":"20221005213633.7070-1-markus.heidelberg@web.de","subject":"Re: [filter-repo PATCH] contrib: new tool to convert svn:externals to Git submodules","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2022-10-06T04:04:05Z","receivedAt":"2022-10-06T04:04:26Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Oct 5, 2022 at 2:37 PM Markus Heidelberg\n<markus.heidelberg@web.de> wrote:\n>\n> This is meant to be a post-processing step after SVN-to-Git conversion\n> by SubGit (https://subgit.com/), which creates a \".gitsvnextmodules\"\n> file that we will use for svn:externals conversion.\n>\n> Signed-off-by: Markus Heidelberg <markus.heidelberg@web.de>\n> ---\n>\n> The SVN-to-Git mapping file format might have to be changed in the\n> future when implementing other features, don't know yet. At least I\n> wanted to publish this script now that the basics work. Maybe there are\n> even some other late-adopters for whose this script might still be\n> useful.\n>\n> I'm eager for hints where descriptions/comments are confusing or\n> incomplete.\n>\n> This script was already used for migration of an SVN repository in our\n> company.\n>\n> Relates to https://github.com/newren/git-filter-repo/issues/14\n\nCool, thanks!  It's been a _long_ time since I've used subversion, and\nI never used subversion externals, so there's not a lot of sanity\nchecking I can do.  Re-reading my comments on that issue from nearly\nthree years ago it looks like I dug in a little bit, just enough to\ngive some getting started pointers, but I don't even remember writing\nthose comments nor what I found.  I can verify it sounds like my\nwriting, and I hope my comments helped.\n\nAnyway, I did give a quick glance over the script, and it seems\nreasonable from a high-level.  As I commented in the issue, it seems\nlike a nice addition to the contrib examples.  So, I've included it.\nThanks!\n\n>\n>  contrib/filter-repo-demos/README.md           |   1 +\n>  .../filter-repo-demos/convert-svnexternals    | 587 ++++++++++++++++++\n>  2 files changed, 588 insertions(+)\n>  create mode 100644 contrib/filter-repo-demos/convert-svnexternals\n>\n> diff --git a/contrib/filter-repo-demos/README.md b/contrib/filter-repo-demos/README.md\n> index ef021b1..9612428 100644\n> --- a/contrib/filter-repo-demos/README.md\n> +++ b/contrib/filter-repo-demos/README.md\n> @@ -16,6 +16,7 @@ lint-history         |Run some lint command on all non-binary files in history.\n>  clean-ignore         |Delete files from history which match current gitignore rules.\n>  filter-lamely (or filter&#8209;branch&#8209;ish) |A nearly bug compatible re-implementation of filter-branch (the git testsuite passes using it instead of filter-branch), with some performance tricks to make it several times faster (though it's still glacially slow compared to filter-repo).\n>  bfg-ish              |A re-implementation of most of BFG Repo Cleaner, with new features and bug fixes.\n> +convert-svnexternals |Insert Git submodules according to SVN externals.\n>\n>  ## Purpose\n>\n> diff --git a/contrib/filter-repo-demos/convert-svnexternals b/contrib/filter-repo-demos/convert-svnexternals\n> new file mode 100644\n> index 0000000..0c81507\n> --- /dev/null\n> +++ b/contrib/filter-repo-demos/convert-svnexternals\n> @@ -0,0 +1,587 @@\n> +#!/usr/bin/env python3\n> +\n> +\"\"\"\n> +This is a program that will insert Git submodules according to SVN externals\n> +definitions (svn:externals properties) from the original Subversion repository\n> +throughout the history.\n> +\n> +Information about the externals is obtained from the \".gitsvnextmodules\" file\n> +created during SVN-to-Git conversion by SubGit (https://subgit.com/). Its\n> +config option \"translate.externals=true\" had to be used therefore.\n> +\n> +Actual modifications:\n> +- Insert gitlinks (mode 160000) into the tree.\n> +- Add .gitmodules file with relevant sections.\n> +- Remove sections converted to submodules from .gitsvnextmodules file\n> +  and delete it if empty.\n> +\n> +.gitsvnextmodules example:\n> +[submodule \"somedir/extdir\"]\n> +       path = somedir/extdir\n> +       owner = somedir\n> +       url = https://svn.example.com/somesvnrepo/trunk\n> +       revision = 1234\n> +       branch = /\n> +       fetch = :refs/remotes/git-svn\n> +       remote = svn\n> +       type = dir\n> +\n> +Resulting addition in \"somedir\" tree (cat-file pretty-print format):\n> +160000 commit 1234123412341234123412341234123412341234 extdir\n> +\n> +Resulting .gitmodules entry:\n> +[submodule \"somedir/extdir\"]\n> +       path = somedir/extdir\n> +       url = https://git.example.com/somegitrepo.git\n> +\n> +SVN-to-Git mapping file:\n> +Can be created from SubGit's \"refs/svn/map\".\n> +One line per mapping in following format:\n> +<svn url> TAB <svn rev> TAB <git url> TAB <git commit> TAB <state>\n> +- Leading '#' can be used for comments.\n> +- <svn url> must not contain a trailing slash.\n> +- <state> has to be \"commit\" to be usable, but can be \"missing\" if <git commit>\n> +  does not exist in the repository anymore. Adopted from git-cat-file output.\n> +Example:\n> +https://svn.example.com/somesvnrepo/trunk      1234    https://git.example.com/somegitrepo.git 1234123412341234123412341234123412341234        commit\n> +\n> +Features:\n> +- Repeatedly added/removed externals will be handled properly.\n> +- Externals replaced by directly added files and vice versa will be handled\n> +  properly.\n> +\n> +Caveats:\n> +- This script must NOT be run repeatedly. A second invocation would lead to a\n> +  different result in case the externals could only be converted partially.\n> +- Inconsistent SVN repositories (with failing checkout) not handled, i.e.\n> +  - normal directory and external with the same path\n> +  - external path not existing for the given revision\n> +- No attention was paid to non-ASCII and special characters in gitlink paths,\n> +  might cause problems.\n> +- There is no error handling for mandatory options missing in .gitsvnextmodules\n> +  file. The script would crash in case of such buggy files, but that shouldn't\n> +  happen in practice.\n> +\n> +TODO:\n> +- Add external files directly.\n> +- Alternatively add external directories directly instead of using a submodule.\n> +\"\"\"\n> +\n> +\"\"\"\n> +Please see the\n> +  ***** API BACKWARD COMPATIBILITY CAVEAT *****\n> +near the top of git-filter-repo.\n> +\"\"\"\n> +\n> +import argparse\n> +import os\n> +import sys\n> +import shutil\n> +import subprocess\n> +import configparser\n> +from urllib.parse import urlsplit\n> +\n> +try:\n> +  import git_filter_repo as fr\n> +except ImportError:\n> +  raise SystemExit(\"Error: Couldn't find git_filter_repo.py.  Did you forget to make a symlink to git-filter-repo named git_filter_repo.py or did you forget to put the latter in your PYTHONPATH?\")\n> +\n> +svn_root_url = \"\"\n> +svn_git_mappings = []\n> +\n> +def parse_args():\n> +  \"\"\"\n> +  Parse and return arguments for this script.\n> +\n> +  Also do some argument sanity checks and adaptions.\n> +  \"\"\"\n> +  parser = argparse.ArgumentParser(\n> +      description=\"Add Git submodules according to svn:externals from .gitsvnextmodules. \"\n> +                  \"As preparation for this conversion process, an analysis can be performed.\")\n> +\n> +  parser.add_argument('--force', '-f', action='store_true',\n> +      help=\"Rewrite repository history even if the current repo does not \"\n> +           \"look like a fresh clone.\")\n> +  parser.add_argument('--refs', nargs='+',\n> +      help=\"Limit history rewriting to the specified refs. Option is directly \"\n> +           \"forwarded to git-filter-repo, see there for details and caveats. \"\n> +           \"Use for debugging purposes only!\")\n> +  parser.add_argument('--svn-root-url',\n> +      help=\"Root URL of the corresponding SVN repository, \"\n> +           \"needed for conversion of relative to absolute external URLs.\")\n> +\n> +  analysis = parser.add_argument_group(title=\"Analysis\")\n> +  analysis.add_argument('--analyze', action='store_true',\n> +      help=\"Analyze repository history and create auxiliary files for conversion process.\")\n> +  analysis.add_argument('--report-dir', type=os.fsencode,\n> +      help=\"Directory to write report, defaults to GIT_DIR/filter-repo/svnexternals, \"\n> +           \"refuses to run if exists, --force delete existing dir first.\")\n> +\n> +  conversion = parser.add_argument_group(title=\"Conversion\")\n> +  conversion.add_argument('--svn-git-mapfiles', type=os.fsencode, nargs='+', metavar='MAPFILE',\n> +      help=\"Files with SVN-to-Git revision mappings for SVN externals conversion.\")\n> +\n> +  args = parser.parse_args()\n> +\n> +  if args.analyze and args.svn_git_mapfiles:\n> +    raise SystemExit(\"Error: --svn-git-mapfiles makes no sense with --analyze.\")\n> +\n> +  if not args.analyze and not args.svn_git_mapfiles:\n> +    raise SystemExit(\"Error: --svn-git-mapfiles is required for the conversion process.\")\n> +\n> +  return args\n> +\n> +def read_mappings(mapfiles):\n> +  \"\"\"\n> +  Read files with SVN-to-Git mappings and return a list of mappings from it.\n> +  \"\"\"\n> +  mappings = []\n> +  for mapfile in mapfiles:\n> +    with open(mapfile, \"rb\") as f:\n> +      for line in f:\n> +        line = line.rstrip(b'\\r\\n')\n> +\n> +        # Skip blank and comment lines\n> +        if not line or line.startswith(b'#'):\n> +          continue\n> +\n> +        # Convert to string for use with configparser later\n> +        line = line.decode()\n> +\n> +        # Parse the line\n> +        fields = line.split('\\t', 4)\n> +        mapping = {'svn_url': fields[0],\n> +                   'svn_rev': int(fields[1]),\n> +                   'git_url': fields[2],\n> +                   'git_commit': fields[3],\n> +                   'state': fields[4]}\n> +\n> +        mappings.append(mapping)\n> +  return mappings\n> +\n> +cat_file_process = None\n> +def parse_config(blob_id):\n> +  \"\"\"\n> +  Create a configparser object for a .gitsvnextmodules/.gitmodules file from\n> +  its blob ID.\n> +  \"\"\"\n> +  parsed_config = configparser.ConfigParser()\n> +\n> +  if blob_id is not None:\n> +    # Get the blob contents\n> +    cat_file_process.stdin.write(blob_id + b'\\n')\n> +    cat_file_process.stdin.flush()\n> +    objhash, objtype, objsize = cat_file_process.stdout.readline().split()\n> +    contents_plus_newline = cat_file_process.stdout.read(int(objsize)+1)\n> +\n> +    # Parse it\n> +    parsed_config.read_string(contents_plus_newline.decode())\n> +\n> +  return parsed_config\n> +\n> +def create_blob(parsed_config):\n> +  \"\"\"\n> +  Create a filter-repo blob object from a .gitsvnextmodules/.gitmodules\n> +  configparser object according to Git config style.\n> +  \"\"\"\n> +  lines = []\n> +  for sec in parsed_config.sections():\n> +    lines.append(\"[\" + sec + \"]\\n\")\n> +    for opt in parsed_config.options(sec):\n> +      lines.append(\"\\t\" + opt + \" = \" + parsed_config[sec][opt] + \"\\n\")\n> +\n> +  return fr.Blob(''.join(lines).encode())\n> +\n> +def get_git_url(svn_url):\n> +  \"\"\"\n> +  Get the Git URL for a corresponding SVN URL.\n> +  \"\"\"\n> +  for entry in svn_git_mappings:\n> +    if entry['svn_url'] == svn_url:\n> +      return entry['git_url']\n> +  else:\n> +    return None\n> +\n> +def get_git_commit_hash(svn_url, svn_rev):\n> +  \"\"\"\n> +  Get the Git commit hash for its corresponding SVN URL+revision.\n> +\n> +  The mapping is not restricted to the exact revision, but also uses the next\n> +  lower revision found. Needed when the revision was set to that of the root\n> +  URL instead of to that of the specific subdirectory (e.g. trunk). TortoiseSVN\n> +  behaves so when setting the external to HEAD.\n> +  \"\"\"\n> +  ent = None\n> +  rev = 0\n> +\n> +  for entry in svn_git_mappings:\n> +    if (entry['svn_url'] == svn_url\n> +          and entry['svn_rev'] <= svn_rev\n> +          and entry['svn_rev'] > rev):\n> +      ent = entry\n> +      rev = entry['svn_rev']\n> +\n> +  if ent is not None and ent['state'] == \"commit\":\n> +    return ent['git_commit']\n> +  else:\n> +    return None\n> +\n> +def get_absolute_svn_url(svnext_url, svn_root_url):\n> +  \"\"\"\n> +  Convert a relative svn:externals URL to an absolute one.\n> +\n> +  If the format is unsupported, return the URL unchanged with success=False.\n> +  If no root URL is given or the URL is absolute already, return it unchanged.\n> +\n> +  In all cases, even if returned \"unchanged\", trailing slashes are removed.\n> +  \"\"\"\n> +  # Remove trailing slash(es)\n> +  svnext_url = svnext_url.rstrip(\"/\")\n> +  svn_root_url = svn_root_url.rstrip(\"/\")\n> +\n> +  # Normalize URLs in relative format\n> +  svn_root_parsed = urlsplit(svn_root_url)\n> +  if svnext_url.startswith((\"../\", \"^/../\")): # unsupported\n> +    return (False, svnext_url)\n> +  elif not svn_root_url:\n> +    pass # unchanged\n> +  elif svnext_url.startswith(\"^/\"):\n> +    svnext_url = svn_root_url + svnext_url[1:]\n> +  elif svnext_url.startswith(\"//\"):\n> +    svnext_url = svn_root_parsed.scheme + \":\" + svnext_url\n> +  elif svnext_url.startswith(\"/\"):\n> +    svnext_url = svn_root_parsed.scheme + \"://\" + svn_root_parsed.netloc + svnext_url\n> +\n> +  return True, svnext_url\n> +\n> +def add_submodule_tree_entry(commit, parsed_config, section):\n> +  \"\"\"\n> +  Add a submodule entry to the tree of a Git commit.\n> +\n> +  SVN externals information obtained from parsed .gitsvnextmodules file.\n> +  \"\"\"\n> +  # Skip type=file (SVN file external), not possible as submodule\n> +  if parsed_config[section]['type'] != 'dir':\n> +    return False\n> +\n> +  success, svn_url = get_absolute_svn_url(parsed_config[section]['url'], svn_root_url)\n> +  # Skip unsupported URL format\n> +  if not success:\n> +    return False\n> +\n> +  # Get SVN revision\n> +  if parsed_config.has_option(section, 'revision'):\n> +    svn_rev = int(parsed_config[section]['revision'])\n> +  else:\n> +    # TODO: revision has to be guessed according to commit timestamp, skip for now\n> +    return False\n> +\n> +  # SVN url+revision mapping to Git commit\n> +  git_hash = get_git_commit_hash(svn_url, svn_rev)\n> +  # Skip missing or unusable mapping\n> +  if git_hash is None:\n> +    return False\n> +  git_hash = git_hash.encode()\n> +\n> +  dirname = parsed_config[section]['path'].encode()\n> +\n> +  # Add gitlink to tree\n> +  commit.file_changes.append(fr.FileChange(b'M', dirname, git_hash, b'160000'))\n> +\n> +  return True\n> +\n> +def get_commit_map_path():\n> +  \"\"\"\n> +  Return path to commit-map file.\n> +  \"\"\"\n> +  git_dir = fr.GitUtils.determine_git_dir(b'.')\n> +  return os.path.join(git_dir, b'filter-repo', b'commit-map')\n> +\n> +def parse_commit_map(commit_map_file):\n> +  \"\"\"\n> +  Parse commit-map file and return a dictionary.\n> +  \"\"\"\n> +  parsed_map = {}\n> +  with open(commit_map_file, \"rb\") as f:\n> +    for line in f:\n> +      line = line.rstrip(b'\\r\\n')\n> +\n> +      # Skip blank lines\n> +      if not line:\n> +        continue\n> +\n> +      # Store old/new commits, also the \"old\"/\"new\" header in the first line\n> +      old, new = line.split()\n> +      parsed_map[old] = new\n> +  return parsed_map\n> +\n> +def merge_commit_maps(old_commit_map, new_commit_map):\n> +  \"\"\"\n> +  Merge old and new commit-map by omitting intermediate commits.\n> +\n> +  Return the merged dictionary.\n> +  \"\"\"\n> +  merged_map = {}\n> +  for (key, old_val) in old_commit_map.items():\n> +    new_val = new_commit_map[old_val] if old_val in new_commit_map else old_val\n> +    merged_map[key] = new_val\n> +  return merged_map\n> +\n> +def write_commit_map(commit_map, commit_map_file):\n> +  \"\"\"\n> +  Write commit-map dictionary to file.\n> +  \"\"\"\n> +  with open(commit_map_file, 'wb') as f:\n> +    for (old, new) in commit_map.items():\n> +      f.write(b'%-40s %s\\n' % (old, new))\n> +\n> +def create_report_dir(args):\n> +  \"\"\"\n> +  Create the directory for analysis report.\n> +  \"\"\"\n> +  if args.report_dir:\n> +    reportdir = args.report_dir\n> +  else:\n> +    git_dir = fr.GitUtils.determine_git_dir(b'.')\n> +\n> +  # Create the report directory as necessary\n> +    results_tmp_dir = os.path.join(git_dir, b'filter-repo')\n> +    if not os.path.isdir(results_tmp_dir):\n> +      os.mkdir(results_tmp_dir)\n> +    reportdir = os.path.join(results_tmp_dir, b'svnexternals')\n> +\n> +  if os.path.isdir(reportdir):\n> +    if args.force:\n> +      sys.stdout.write(\"Warning: Removing recursively: \\\"%s\\\"\" % fr.decode(reportdir))\n> +      shutil.rmtree(reportdir)\n> +    else:\n> +      sys.stdout.write(\"Error: dir already exists (use --force to delete): \\\"%s\\\"\\n\" % fr.decode(reportdir))\n> +      sys.exit(1)\n> +\n> +  os.mkdir(reportdir)\n> +\n> +  return reportdir\n> +\n> +analysis = {'dir_ext_orig': [],\n> +            'dir_ext_abs': [],\n> +            'file_ext_orig': [],\n> +            'file_ext_abs': []}\n> +def write_analysis(reportdir):\n> +  \"\"\"\n> +  Prepare analysis and write it to files in report directory.\n> +  \"\"\"\n> +  analysis['dir_ext_orig'].sort()\n> +  analysis['dir_ext_abs'].sort()\n> +  analysis['file_ext_orig'].sort()\n> +  analysis['file_ext_abs'].sort()\n> +\n> +  sys.stdout.write(\"Writing reports to %s...\" % fr.decode(reportdir))\n> +  sys.stdout.flush()\n> +\n> +  with open(os.path.join(reportdir, b\"dir-externals-original.txt\"), 'wb') as f:\n> +    for url in analysis['dir_ext_orig']:\n> +      f.write((\"%s\\n\" % url).encode())\n> +\n> +  with open(os.path.join(reportdir, b\"dir-externals-absolute.txt\"), 'wb') as f:\n> +    for url in analysis['dir_ext_abs']:\n> +      f.write((\"%s\\n\" % url).encode())\n> +\n> +  with open(os.path.join(reportdir, b\"file-externals-original.txt\"), 'wb') as f:\n> +    for url in analysis['file_ext_orig']:\n> +      f.write((\"%s\\n\" % url).encode())\n> +\n> +  with open(os.path.join(reportdir, b\"file-externals-absolute.txt\"), 'wb') as f:\n> +    for url in analysis['file_ext_abs']:\n> +      f.write((\"%s\\n\" % url).encode())\n> +\n> +  sys.stdout.write(\"done.\\n\")\n> +\n> +def analyze_externals(commit, metadata):\n> +  \"\"\"\n> +  Generate/extend analysis of SVN externals for a Git commit.\n> +\n> +  Used as filter-repo commit callback.\n> +  \"\"\"\n> +  for change in commit.file_changes:\n> +    if change.filename == b'.gitsvnextmodules' and change.type == b'M':\n> +      gitsvnextmodules = parse_config(change.blob_id)\n> +\n> +      for sec in gitsvnextmodules.sections():\n> +        url = gitsvnextmodules[sec]['url']\n> +        success, abs_url = get_absolute_svn_url(url, svn_root_url)\n> +\n> +        # List of svn:externals URLs, also add the URL to the absolute list if\n> +        # conversion was not successful\n> +        if gitsvnextmodules[sec]['type'] == 'dir':\n> +          if url not in analysis['dir_ext_orig']:\n> +            analysis['dir_ext_orig'].append(url)\n> +          if abs_url not in analysis['dir_ext_abs']:\n> +            analysis['dir_ext_abs'].append(abs_url)\n> +        else:\n> +          if url not in analysis['file_ext_orig']:\n> +            analysis['file_ext_orig'].append(url)\n> +          if abs_url not in analysis['file_ext_abs']:\n> +            analysis['file_ext_abs'].append(abs_url)\n> +\n> +def insert_submodules(commit, metadata):\n> +  \"\"\"\n> +  Insert submodules for a Git commit.\n> +\n> +  Used as filter-repo commit callback.\n> +\n> +  Since .gitsvnextmodules just contains the svn:externals state for the given\n> +  commit, we cannot derive specific changes from that file.\n> +  So we can only add/modify the gitlinks according to .gitsvnextmodules\n> +  (without knowing whether adding a new or modifying an existing or even\n> +  \"modifying\" an unchanged submodule, but none of that really matters).\n> +  We do not have information about deleted externals, those will be handled in\n> +  a separate filter run afterwards.\n> +\n> +  The .gitmodules file however will already be correct in this function because\n> +  we don't need to know about specific changes to add, modify or delete it.\n> +  \"\"\"\n> +  for change in commit.file_changes:\n> +    if change.filename == b'.gitsvnextmodules' and change.type in (b'M', b'D'):\n> +      gitsvnextmodules = parse_config(change.blob_id)\n> +      gitmodules = configparser.ConfigParser()\n> +\n> +      # Add gitlinks to the tree and prepare .gitmodules file content\n> +      for sec in gitsvnextmodules.sections():\n> +        if add_submodule_tree_entry(commit, gitsvnextmodules, sec):\n> +          # Gitlink added\n> +          # -> Add this entry to .gitmodules as well\n> +\n> +          # Create the section name string manually, do not rely on\n> +          # .gitsvnextmodules to always use the proper section name.\n> +          sec_name = 'submodule \"' + gitsvnextmodules[sec]['path'] + '\"'\n> +          gitmodules[sec_name] = {}\n> +\n> +          # submodule.<name>.path\n> +          gitmodules[sec_name]['path'] = gitsvnextmodules[sec]['path']\n> +\n> +          # submodule.<name>.url\n> +          success, svn_url = get_absolute_svn_url(gitsvnextmodules[sec]['url'], svn_root_url)\n> +          git_url = get_git_url(svn_url)\n> +          if git_url is not None:\n> +            gitmodules[sec_name]['url'] = git_url\n> +          else:\n> +            # Abort, but this will not happen in practice, catched in\n> +            # add_submodule_tree_entry() via get_git_commit_hash() already.\n> +            raise SystemExit(\"Error: No Git URL found in mapping although a commit hash could be found.\")\n> +\n> +      # Write blob and adapt tree for .gitmodules\n> +      if gitmodules.sections():\n> +        # Create a blob object from the content and add it to the tree.\n> +        blob = create_blob(gitmodules)\n> +        filter.insert(blob)\n> +        commit.file_changes.append(fr.FileChange(b'M', b'.gitmodules', blob.id, b'100644'))\n> +      else:\n> +        # Delete the file, even if a \"git rm\" of all submodules keeps it empty.\n> +        commit.file_changes.append(fr.FileChange(b'D', b'.gitmodules'))\n> +\n> +def delete_submodules(commit, metadata):\n> +  \"\"\"\n> +  Delete submodules from a Git commit.\n> +\n> +  Used as filter-repo commit callback.\n> +\n> +  Delete all submodules (inserted in the previous filter run) without an entry\n> +  in .gitsvnextmodules, these were real deletions of externals, which couldn't\n> +  be detected before.\n> +  Only the tree entries have to be removed because the .gitmodules file is\n> +  already in correct state from previous filter run.\n> +  \"\"\"\n> +  for change in commit.file_changes:\n> +    if change.filename == b'.gitsvnextmodules' and change.type in (b'M', b'D'):\n> +      gitsvnextmodules = parse_config(change.blob_id)\n> +\n> +      # Search for all submodules in the tree\n> +      output = subprocess.check_output('git ls-tree -d -r -z'.split() + [commit.original_id])\n> +      for line in output.split(b'\\x00'):\n> +        if not line:\n> +          continue\n> +        mode_objtype_objid, dirname = line.split(b'\\t', 1)\n> +        mode, objtype, objid = mode_objtype_objid.split(b' ')\n> +        if mode == b'160000' and objtype == b'commit':\n> +          # Submodule found\n> +          # -> Delete it if there is no corresponding entry in\n> +          #    .gitsvnextmodules, keep/reinsert it otherwise\n> +          for sec in gitsvnextmodules.sections():\n> +            if gitsvnextmodules[sec]['path'].encode() == dirname:\n> +              # Reinsert it, might have been deleted in previous commits\n> +              if add_submodule_tree_entry(commit, gitsvnextmodules, sec):\n> +                # And remove the config section because this external has been\n> +                # converted\n> +                gitsvnextmodules.remove_section(sec)\n> +                break\n> +          else:\n> +            # Delete it\n> +            commit.file_changes.append(fr.FileChange(b'D', dirname))\n> +\n> +      # Rewrite .gitsvnextmodules to contain the unhandled externals only,\n> +      # delete it if empty (all externals converted).\n> +      if gitsvnextmodules.sections():\n> +        # Create a blob object from the content and replace the original one.\n> +        blob = create_blob(gitsvnextmodules)\n> +        filter.insert(blob)\n> +        change.blob_id = blob.id\n> +      else:\n> +        if change.type == b'M':\n> +          # File became empty, delete it\n> +          commit.file_changes.append(fr.FileChange(b'D', b'.gitsvnextmodules'))\n> +          break # avoid endless for loop\n> +        #else:\n> +          # File was empty already, delete command already present in stream\n> +\n> +my_args = parse_args()\n> +\n> +# Use passed URL without trailing slash(es)\n> +if my_args.svn_root_url:\n> +  svn_root_url = my_args.svn_root_url.rstrip(\"/\")\n> +\n> +# Arguments forwarded to filter-repo\n> +extra_args = []\n> +if my_args.force:\n> +  extra_args = ['--force']\n> +if my_args.refs:\n> +  extra_args += ['--refs'] + my_args.refs\n> +\n> +cat_file_process = subprocess.Popen(['git', 'cat-file', '--batch'],\n> +                                    stdin = subprocess.PIPE,\n> +                                    stdout = subprocess.PIPE)\n> +if my_args.analyze:\n> +  # Analysis\n> +  reportdir = create_report_dir(my_args)\n> +\n> +  fr_args = fr.FilteringOptions.parse_args(['--dry-run']\n> +                                           + extra_args)\n> +  filter = fr.RepoFilter(fr_args, commit_callback=analyze_externals)\n> +  filter.run()\n> +\n> +  write_analysis(reportdir)\n> +else:\n> +  # Conversion\n> +  svn_git_mappings = read_mappings(my_args.svn_git_mapfiles)\n> +\n> +  # There are no references to commit hashes in commit messages because this\n> +  # script runs on a Git repository converted from a Subversion repository.\n> +  fr_args = fr.FilteringOptions.parse_args(['--preserve-commit-hashes',\n> +                                            '--preserve-commit-encoding',\n> +                                            '--replace-refs', 'update-no-add']\n> +                                           + extra_args)\n> +  filter = fr.RepoFilter(fr_args, commit_callback=insert_submodules)\n> +  filter.run()\n> +\n> +  # Store commit-map after first run\n> +  first_commit_map = parse_commit_map(get_commit_map_path())\n> +\n> +  filter = fr.RepoFilter(fr_args, commit_callback=delete_submodules)\n> +  filter.run()\n> +\n> +  # Update commit-map after second run, based on original IDs\n> +  second_commit_map = parse_commit_map(get_commit_map_path())\n> +  merged_commit_map = merge_commit_maps(first_commit_map, second_commit_map)\n> +  write_commit_map(merged_commit_map, get_commit_map_path())\n> +\n> +cat_file_process.stdin.close()\n> +cat_file_process.wait()\n> --\n> 2.37.3\n"},{"id":"464279","messageId":"FCD8CA29-25B8-43B6-8473-5654D9F0D99A@web.de","threadId":"58569","inReplyTo":"CABPp-BEyemFxeCG-bmyiugfzCqZp=XbOJKVWrepjeTf9G5yHgQ@mail.gmail.com","subject":"Re: [filter-repo PATCH] contrib: new tool to convert svn:externals to Git submodules","fromName":"Markus Heidelberg","fromEmail":"markus.heidelberg@web.de","sentAt":"2022-10-06T08:35:32Z","receivedAt":"2022-10-06T08:40:46Z","isPatch":true,"sender":{"key":"markus.heidelberg@web.de","avatar":"https://avatars.githubusercontent.com/u/6334512?v=4"},"body":"\n\nAm 6. Oktober 2022 06:04:05 MESZ schrieb Elijah Newren <newren@gmail.com>:\n>On Wed, Oct 5, 2022 at 2:37 PM Markus Heidelberg\n><markus.heidelberg@web.de> wrote:\n>>\n>> This is meant to be a post-processing step after SVN-to-Git conversion\n>> by SubGit (https://subgit.com/), which creates a \".gitsvnextmodules\"\n>> file that we will use for svn:externals conversion.\n>>\n>> Signed-off-by: Markus Heidelberg <markus.heidelberg@web.de>\n>> ---\n>>\n>> The SVN-to-Git mapping file format might have to be changed in the\n>> future when implementing other features, don't know yet. At least I\n>> wanted to publish this script now that the basics work. Maybe there are\n>> even some other late-adopters for whose this script might still be\n>> useful.\n>>\n>> I'm eager for hints where descriptions/comments are confusing or\n>> incomplete.\n>>\n>> This script was already used for migration of an SVN repository in our\n>> company.\n>>\n>> Relates to https://github.com/newren/git-filter-repo/issues/14\n>\n>Cool, thanks!  It's been a _long_ time since I've used subversion, and\n>I never used subversion externals, so there's not a lot of sanity\n>checking I can do.  Re-reading my comments on that issue from nearly\n>three years ago it looks like I dug in a little bit, just enough to\n>give some getting started pointers, but I don't even remember writing\n>those comments nor what I found.  I can verify it sounds like my\n>writing, and I hope my comments helped.\n>\n>Anyway, I did give a quick glance over the script, and it seems\n>reasonable from a high-level.  As I commented in the issue, it seems\n>like a nice addition to the contrib examples.  So, I've included it.\n>Thanks!\n\nOh, that went fast! And your comments definitely helped for implementation, I read them several times :) \n\nThanks for inclusion!\n"}]}