{"thread":{"id":"11063","subject":"importing bk into git","startedAt":"2007-11-29T21:32:03Z","lastAt":"2007-12-08T19:19:09Z","messageCount":6,"participants":["Christoph","Andreas Ericsson","Johannes Schindelin","David Kettler"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"61427","messageId":"200711292232.03352.christoph.duelli@gmx.de","threadId":"11063","inReplyTo":null,"subject":"importing bk into git","fromName":"Christoph","fromEmail":"christoph.duelli@gmx.de","sentAt":"2007-11-29T21:32:03Z","receivedAt":"2007-11-29T21:32:03Z","isPatch":false,"sender":{"key":"christoph.duelli@gmx.de","avatar":null},"body":"I am trying to import a BitKeeper repo into a (new) git repo.\n\nI am trying with the script bk2git.py that I found on the web.\nThis does not quite work - I fear script is no longer working with the current \ngit release. (I am using the current git release.)\n\nIf I have understood the script correctly, it does repeated bk checkouts and \nimports the updates the git repo diff of the (next) checkout etc.\n\nIt seems this script tries to do so by settings environment vars\nGIT_OBJECT_DIRECTORY and GIT_INDEX_FILE\nto point at the git repo.\n\nThe bk checkout are done at a temp. dir (tmp_dir).\n\n\nThe following lines fail\n  os.system(\"cd %s; git-ls-files --deleted | xargs \ngit-update-cache --remove\" % tmp_dir)\n\nwith: fatal: Not a git repository\nxargs: git-update-cache: No such file or directory\n\nThe problem seems to be that the script cd's into the temp dir (which is not a \ngit repo) and the git-ls-files fails to find a git repo there.\nI think the issue might be that an earlier version of git was perhaps able to \nfind the repo by means of the env. vars mentioned above.\n\nAny idea if/how I can fix this?\nThanks for any ideas and best regards\n\nChristoph\n(Sorry, my python and git skills are so far very limited.)\n\nPS: I have attached the script I downloaded from the net.\n-- \nFORTUNE'S PARTY TIPS\t\t#14\n\nTired of finding that other people are helping themselves to your good\nliquor at BYOB parties?  Take along a candle, which you insert and\nlight after you've opened the bottle.  No one ever expects anything\ndrinkable to be in a bottle which has a candle stuck in its neck.\n"},{"id":"61485","messageId":"474FC2C5.8060400@op5.se","threadId":"11063","inReplyTo":"200711292232.03352.christoph.duelli@gmx.de","subject":"Re: importing bk into git","fromName":"Andreas Ericsson","fromEmail":"ae@op5.se","sentAt":"2007-11-30T07:59:01Z","receivedAt":"2007-11-30T07:59:01Z","isPatch":false,"sender":{"key":"ae@op5.se","avatar":"https://gravatar.com/avatar/426e89595c75a8f5252dd0c989e5fabe5bcac616e68557427ad9aef6b0ca342a?d=mp&s=160"},"body":"Christoph wrote:\n> I am trying to import a BitKeeper repo into a (new) git repo.\n> \n> I am trying with the script bk2git.py that I found on the web.\n> This does not quite work - I fear script is no longer working with the current \n> git release. (I am using the current git release.)\n> \n> If I have understood the script correctly, it does repeated bk checkouts and \n> imports the updates the git repo diff of the (next) checkout etc.\n> \n> It seems this script tries to do so by settings environment vars\n> GIT_OBJECT_DIRECTORY and GIT_INDEX_FILE\n> to point at the git repo.\n> \n> The bk checkout are done at a temp. dir (tmp_dir).\n> \n> \n> The following lines fail\n>   os.system(\"cd %s; git-ls-files --deleted | xargs \n> git-update-cache --remove\" % tmp_dir)\n> \n> with: fatal: Not a git repository\n> xargs: git-update-cache: No such file or directory\n> \n\nYou may have better luck using \"git update-index\" instead of\ngit-update-cache. If that doesn't work, try finding out which\nversion of git the importer script was written against and try\nusing that version of git for the import.\n\nIf you run into problems while using a newer git on the\nimported repository that'll be a different discussion.\n\n> The problem seems to be that the script cd's into the temp dir (which is not a \n> git repo) and the git-ls-files fails to find a git repo there.\n> I think the issue might be that an earlier version of git was perhaps able to \n> find the repo by means of the env. vars mentioned above.\n> \n\nIt should still do this, afaik, although it's probably better\nto just use GIT_DIR nowadays.\n\n-- \nAndreas Ericsson                   andreas.ericsson@op5.se\nOP5 AB                             www.op5.se\nTel: +46 8-230225                  Fax: +46 8-230231\n"},{"id":"61492","messageId":"Pine.LNX.4.64.0711301135050.27959@racer.site","threadId":"11063","inReplyTo":"474FC2C5.8060400@op5.se","subject":"[PATCH] Replace the word 'update-cache' by 'update-index' everywhere","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2007-11-30T11:35:23Z","receivedAt":"2007-11-30T11:35:23Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"\nSigned-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>\n---\n Documentation/user-manual.txt |    2 +-\n Makefile                      |    2 +-\n configure.ac                  |    2 +-\n t/t4000-diff-format.sh        |    2 +-\n t/t4001-diff-rename.sh        |    2 +-\n t/t4100/t-apply-1.patch       |    2 +-\n t/t4100/t-apply-2.patch       |    2 +-\n t/t4100/t-apply-5.patch       |    2 +-\n t/t4100/t-apply-6.patch       |    2 +-\n 9 files changed, 9 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/user-manual.txt b/Documentation/user-manual.txt\nindex 0aaed10..93a47b4 100644\n--- a/Documentation/user-manual.txt\n+++ b/Documentation/user-manual.txt\n@@ -3707,7 +3707,7 @@ should use the `--remove` and `--add` flags respectively.\n NOTE! A `--remove` flag does 'not' mean that subsequent filenames will\n necessarily be removed: if the files still exist in your directory\n structure, the index will be updated with their new status, not\n-removed. The only thing `--remove` means is that update-cache will be\n+removed. The only thing `--remove` means is that update-index will be\n considering a removed file to be a valid thing, and if the file really\n does not exist any more, it will update the index accordingly.\n \ndiff --git a/Makefile b/Makefile\nindex 4454116..f000a5e 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -111,7 +111,7 @@ all::\n # times (my ext3 doesn't).\n #\n # Define USE_STDEV below if you want git to care about the underlying device\n-# change being considered an inode change from the update-cache perspective.\n+# change being considered an inode change from the update-index perspective.\n #\n # Define ASCIIDOC8 if you want to format documentation with AsciiDoc 8\n #\ndiff --git a/configure.ac b/configure.ac\nindex 7bcf1a4..5f8a15b 100644\n--- a/configure.ac\n+++ b/configure.ac\n@@ -415,7 +415,7 @@ GIT_PARSE_WITH(iconv))\n # times (my ext3 doesn't).\n #\n # Define USE_STDEV below if you want git to care about the underlying device\n-# change being considered an inode change from the update-cache perspective.\n+# change being considered an inode change from the update-index perspective.\n \n \n ## Output files\ndiff --git a/t/t4000-diff-format.sh b/t/t4000-diff-format.sh\nindex 7d92ae3..c44b27a 100755\n--- a/t/t4000-diff-format.sh\n+++ b/t/t4000-diff-format.sh\n@@ -16,7 +16,7 @@ cat path0 >path1\n chmod +x path1\n \n test_expect_success \\\n-    'update-cache --add two files with and without +x.' \\\n+    'update-index --add two files with and without +x.' \\\n     'git update-index --add path0 path1'\n \n mv path0 path0-\ndiff --git a/t/t4001-diff-rename.sh b/t/t4001-diff-rename.sh\nindex 877c1ea..a326924 100755\n--- a/t/t4001-diff-rename.sh\n+++ b/t/t4001-diff-rename.sh\n@@ -27,7 +27,7 @@ Line 15\n '\n \n test_expect_success \\\n-    'update-cache --add a file.' \\\n+    'update-index --add a file.' \\\n     'git update-index --add path0'\n \n test_expect_success \\\ndiff --git a/t/t4100/t-apply-1.patch b/t/t4100/t-apply-1.patch\nindex de58751..f98baa8 100644\n--- a/t/t4100/t-apply-1.patch\n+++ b/t/t4100/t-apply-1.patch\n@@ -90,7 +90,7 @@ diff --git a/Documentation/git.txt b/Documentation/git.txt\n diff --git a/Makefile b/Makefile\n --- a/Makefile\n +++ b/Makefile\n-@@ -30,7 +30,7 @@ PROG=   git-update-cache git-diff-files \n+@@ -30,7 +30,7 @@ PROG=   git-update-index git-diff-files \n  \tgit-checkout-cache git-diff-tree git-rev-tree git-ls-files \\\n  \tgit-check-files git-ls-tree git-merge-base git-merge-cache \\\n  \tgit-unpack-file git-export git-diff-cache git-convert-cache \\\ndiff --git a/t/t4100/t-apply-2.patch b/t/t4100/t-apply-2.patch\nindex cfdc808..f5c7d60 100644\n--- a/t/t4100/t-apply-2.patch\n+++ b/t/t4100/t-apply-2.patch\n@@ -9,7 +9,7 @@ diff --git a/Makefile b/Makefile\n -\tgit-deltafy-script\n +\tgit-deltafy-script git-fetch-script\n  \n- PROG=   git-update-cache git-diff-files git-init-db git-write-tree \\\n+ PROG=   git-update-index git-diff-files git-init-db git-write-tree \\\n  \tgit-read-tree git-commit-tree git-cat-file git-fsck-cache \\\n diff --git a/git-pull-script b/git-fetch-script\n similarity index 87%\ndiff --git a/t/t4100/t-apply-5.patch b/t/t4100/t-apply-5.patch\nindex de11623..ad45c51 100644\n--- a/t/t4100/t-apply-5.patch\n+++ b/t/t4100/t-apply-5.patch\n@@ -200,7 +200,7 @@ diff a/Documentation/git.txt b/Documentation/git.txt\n diff a/Makefile b/Makefile\n --- a/Makefile\n +++ b/Makefile\n-@@ -30,7 +30,7 @@ PROG=   git-update-cache git-diff-files \n+@@ -30,7 +30,7 @@ PROG=   git-update-index git-diff-files \n  \tgit-checkout-cache git-diff-tree git-rev-tree git-ls-files \\\n  \tgit-check-files git-ls-tree git-merge-base git-merge-cache \\\n  \tgit-unpack-file git-export git-diff-cache git-convert-cache \\\ndiff --git a/t/t4100/t-apply-6.patch b/t/t4100/t-apply-6.patch\nindex d975363..a72729a 100644\n--- a/t/t4100/t-apply-6.patch\n+++ b/t/t4100/t-apply-6.patch\n@@ -8,7 +8,7 @@ diff a/Makefile b/Makefile\n -\tgit-deltafy-script\n +\tgit-deltafy-script git-fetch-script\n  \n- PROG=   git-update-cache git-diff-files git-init-db git-write-tree \\\n+ PROG=   git-update-index git-diff-files git-init-db git-write-tree \\\n  \tgit-read-tree git-commit-tree git-cat-file git-fsck-cache \\\n diff a/git-fetch-script b/git-fetch-script\n --- /dev/null\n-- \n1.5.3.6.2088.g8c260\n"},{"id":"61721","messageId":"t1o1wa432gc.fsf@ednux512.dsto.defence.gov.au","threadId":"11063","inReplyTo":"200711292232.03352.christoph.duelli@gmx.de","subject":"Re: importing bk into git","fromName":"David Kettler","fromEmail":"david.kettler@dsto.defence.gov.au","sentAt":"2007-12-03T03:02:43Z","receivedAt":"2007-12-03T03:02:43Z","isPatch":false,"sender":{"key":"david.kettler@dsto.defence.gov.au","avatar":null},"body":"G'day,\n\nI modified that script to convert a number of our repositories in\nFebruary.  The version below worked for me at the time, but I'm not\nable to test it now as our BK license has expired.  In particular I'm\nnot sure if the bk_info.split line is correct; I had a reduced form of\nthis line in the file which now looks obviously wrong.\n\nThe script is slow; most of the time is in the bk export for every\nrevision.  There are probably dumb things in there; I don't know\npython and I was just starting with git.\n\nChanges from the version I downloaded from the web include:\n  - sundry changes to make it work for me\n  - separate committers file to translate user names to full names\n  - specify a git dir template\n  - copy tags from BK\n  - minimal conversion of ignore files\n  - increased recursion limit to handle large number of commits\n\nI hope this is useful to someone.\n\nregards, David.\n\n--snip,snip--\n# Convert a BK repository to GIT\n# usage: bk2git BK_REPO [GIT_REPO]\n# Single branch only.\n\nimport os\nimport time\nimport sys\n\nsys.setrecursionlimit(10000);\n\ntemplates_dir = \"/tmp/bk2git/templates\"\ncommitters_file = \"/tmp/bk2git/committers\"\ntmp_dir = \"/tmp/bk-export%d\" % os.getpid()\n\n# Get repository locations.\nif len(sys.argv) < 2 or len(sys.argv) > 3:\n    print \"usage: bk2git BK_REPO [GIT_REPO]\"\n    sys.exit(1);\n\nbk_dir = sys.argv[1]\nif len(sys.argv) == 3:\n    git_dir = sys.argv[2]\nelse:\n    git_dir = bk_dir + \".git\"\n\nprint \"BK  \" + bk_dir\nprint \"GIT \" + git_dir\n\n\n# Get committer names.\nf = file(committers_file, \"r\")\ncommitters = {}\nfor line in f:\n    [m,n] = line.split(\" \",1)\n    committers[m] = n.strip();\nf.close()\n\n# Get tree of commits.\nf = os.popen(\"cd %s; bk prs -d':REV:\\\\t:PARENT:\\\\t:MPARENT:\\\\t\\\\n' ChangeSet\" % bk_dir)\nf.readline()\nparents={}\nfor rev in f:\n    [n,p] = rev.rstrip().split(\"\\t\",1)\n    parents[n] = p.split(\"\\t\")\nf.close()\n\n# Get tags.\nf = os.popen(\"cd %s;  bk changes -t -n -d':I:\\\\t:TAG:'\" % bk_dir)\ntags={}\nfor rev in f:\n    [n,t] = rev.rstrip().split(\"\\t\",1)\n    tags[n] = t\nf.close()\n\n# Initialize git repository.\nos.system(\"mkdir %s\" % git_dir)\nos.chdir(git_dir)\nos.system(\"git --bare init --template=%s\" % templates_dir)\nos.system(\"git-config core.bare false\")\n\nunknown = {}\ndef get_name(email):\n    if committers.has_key(email):\n        return committers[email]\n    unknown[email] = True\n    return \"*Unknown*\"\n\ndef git_commit(rev, p):\n    os.chdir(tmp_dir)\n    os.symlink(git_dir, \"%s/.git\" % tmp_dir)\n#    os.system(\"pwd; ls -AlR\")\n    os.system(\"git-ls-files -z --deleted | xargs -0 git-update-index --remove\")\n    os.system(\"git-ls-files -z --others | xargs -0 git-update-index --add\")\n    os.system(\"git-ls-files -z | xargs -0 git-update-index\")\n    treeid = os.popen(\"git-write-tree\").read().rstrip()\n    print \"wrote tree as %s\" % treeid\n    os.chdir(git_dir)\n    os.system(\"rm -Rf %s\" % tmp_dir)\n\n    bk_info = os.popen(\"cd %s; bk prs -r%s -d':KEY:\\\\n:UTC:\\\\n:USER:@:HOST:\\\\n$each(:C:){:C\\\\n}\\\\n' ChangeSet | sed 1d\" % (bk_dir, rev)).read()\n\n    [key, date, user, comments] = bk_info.split(\"\\n\", 3)\n#    [key, date, user] = bk_info.split(\"\\n\", 2)\n    [name, machine] = user.split(\"@\", 1);\n    f = file(\"/tmp/git-comments\",\"w\")\n    f.write(comments)\n    f.write(\"BK KEY: %s\\n\" % key)\n    f.close()\n    sdate = str(int(time.mktime(time.strptime(date+\" UTC\", \"%Y%m%d%H%M%S %Z\"))))\n    os.putenv(\"GIT_AUTHOR_DATE\", sdate)\n    os.putenv(\"GIT_AUTHOR_EMAIL\", user)\n    os.putenv(\"GIT_AUTHOR_NAME\", get_name(name))\n    os.putenv(\"GIT_COMMITTER_DATE\", sdate)\n    os.putenv(\"GIT_COMMITTER_EMAIL\", user)\n    os.putenv(\"GIT_COMMITTER_NAME\", get_name(name))\n    \n    commitid = os.popen(\"git-commit-tree %s %s < /tmp/git-comments\" % (treeid, \" \".join([\"-p \"+a for a in p]))).read().rstrip()\n    print \"committed %s as %s\" % (rev, commitid)\n\n    if tags.has_key(rev):\n        os.system(\"git-tag %s %s\" % (tags[rev], commitid))\n        print \"tagged %s\" % tags[rev]\n\n    return commitid\n\nos.system(\"mkdir %s; touch %s/initial\" % (tmp_dir, tmp_dir))\nresolved = {'1.1': git_commit(\"1.1\",[])}\n\ndef res(ver):\n    if resolved.has_key(ver):\n        return\n    \n    for v in parents[ver]:\n        res(v)\n\n    os.system(\"cd %s; bk export -r%s %s\" % (bk_dir, ver, tmp_dir))\n    ignore = \"%s/.gitignore\" % tmp_dir\n    os.system(\"cd %s; bk co -kpq -r@%s BitKeeper/etc/ignore | sed '/^BitKeeper/d;/^PENDING/d' > %s\" % (bk_dir, ver, ignore))\n    os.system(\"test -s %s || rm %s\" % (ignore, ignore))\n\n    resolved[ver] = git_commit(ver, [resolved[v] for v in parents[ver]])\n    return resolved[ver]\n\ntot = os.popen(\"cd %s; bk prs -r+ -d':REV:' ChangeSet | tail -n 1\" % bk_dir).read()\nprint \"Exporting bitkeeper up to version %s\" % tot\n\nHEAD = res(tot)\nprint \"HEAD: %s\" % HEAD\nfile(\"%s/refs/heads/master\" % git_dir,\"w\").write(HEAD + \"\\n\")\nos.system(\"git-config core.bare true\")\nos.system(\"git gc\")\nprint unknown.keys()\n--snip,snip--\n-- \nIMPORTANT: This email remains the property of the Australian Defence\nOrganisation and is subject to the jurisdiction of section 70 of the\nCRIMES ACT 1914. If you have received this email in error, you are\nrequested to contact the sender and delete the email.\n"},{"id":"61818","messageId":"200712032159.57017.christoph.duelli@gmx.de","threadId":"11063","inReplyTo":"t1o1wa432gc.fsf@ednux512.dsto.defence.gov.au","subject":"Re: importing bk into git","fromName":"Christoph","fromEmail":"christoph.duelli@gmx.de","sentAt":"2007-12-03T20:59:56Z","receivedAt":"2007-12-03T20:59:56Z","isPatch":false,"sender":{"key":"christoph.duelli@gmx.de","avatar":null},"body":"On Monday 03 December 2007 04:02:43 you wrote:\n> G'day,\n>\n> I modified that script to convert a number of our repositories in\n> February.  The version below worked for me at the time, but I'm not\n> able to test it now as our BK license has expired.  In particular I'm\n> not sure if the bk_info.split line is correct; I had a reduced form of\n> this line in the file which now looks obviously wrong.\n>\n> The script is slow; most of the time is in the bk export for every\n> revision.  There are probably dumb things in there; I don't know\n> python and I was just starting with git.\n>\n> Changes from the version I downloaded from the web include:\n>   - sundry changes to make it work for me\n>   - separate committers file to translate user names to full names\n>   - specify a git dir template\n>   - copy tags from BK\n>   - minimal conversion of ignore files\n>   - increased recursion limit to handle large number of commits\n>\n> I hope this is useful to someone.\nThanks,\n\nI have made some modifications to get the script working as well. I was able \nto convert some simple (really small test repos).I wanted to try to convert \nmy real big repo (>10k files, >5000 revisions) before mailing it. However, as \nrunning this conversion will probably take more than 24hours I will do so \nnext weekend.\nI will check your script and integrate my changes (if there are any relative \nto yours) after that.\n\nbest regards\nChristoph\n"},{"id":"62426","messageId":"200712082019.10273.christoph.duelli@gmx.de","threadId":"11063","inReplyTo":"474FC2C5.8060400@op5.se","subject":"Re: importing bk into git (succeeded)","fromName":"Christoph","fromEmail":"christoph.duelli@gmx.de","sentAt":"2007-12-08T19:19:09Z","receivedAt":"2007-12-08T19:19:09Z","isPatch":false,"sender":{"key":"christoph.duelli@gmx.de","avatar":null},"body":"On Friday 30 November 2007 08:59:01 you wrote:\n> Christoph wrote:\n> > I am trying to import a BitKeeper repo into a (new) git repo.\n> >\n> > I am trying with the script bk2git.py that I found on the web.\n> > This does not quite work - I fear script is no longer working with the\n> > current git release. (I am using the current git release.)\n[snip]\n> > The following lines fail\n> >   os.system(\"cd %s; git-ls-files --deleted | xargs\n> > git-update-cache --remove\" % tmp_dir)\n[snip]\n> It should still do this, afaik, although it's probably better\n> to just use GIT_DIR nowadays.\nUsing GIT_DIR works, one has to set it to point to the .git directory (I had \nassumed the git_dir to be the one *containing* .git).\n\nAnother point with the original script was that you had to have all commiters \nin the mapping (email -> name), otherwise it would not work. \n(Supplying '*Unknown*' fixed this.)\n\nI have added\n- better arguments parsing (see --help)\n- ability to do incremental conversions (--incr-db, -r)\n- different levels of verbosity\n\nI have attached a working version of the script. I have added comments that \n(try to) explain the script if someone else has trouble with it.\n\nMoreover, it is very helpful to put the directories inside a ramdisk. \nOtherwise, you have to be extremely patient.\n(You have to be patient anyway. For a bk repo of some 14k files (>110MB  \nwhen 'clean', 8000 changesets) the script took some 11hrs).\nAnother issue (when using ramdisks) is memory. On my machine memory is scarce \n(only 1 GB). So the ever growing bare repo (can't be gc'ed before getting its \nhead) exhausted the ramdisk space. I worked around this by doing an \nincremental conversion. After each increment a gc is possible and the git \nrepo shrinks to a managable size (and still fits inside the ramdisk).\n\nSo, to sum up: converting a big repo is no fun, but it works, given enough \ntime (and ram).\n\nThanks, best regards\nChristoph\n-- \nA billion here, a couple of billion there -- first thing you know it\nadds up to be real money.\n\t\t-- Senator Everett McKinley Dirksen\n"}]}