{"thread":{"id":"1028","subject":"[PATCH] Add git-relink-script to fix up missing hardlinks","startedAt":"2005-06-26T18:15:16Z","lastAt":"2005-06-27T01:11:13Z","messageCount":6,"participants":["Ryan Anderson","Jeff Garzik","Junio C Hamano","Jan Harkes"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"5278","messageId":"20050626181516.GC20369@mythryan2.michonline.com","threadId":"1028","inReplyTo":null,"subject":"[PATCH] Add git-relink-script to fix up missing hardlinks","fromName":"Ryan Anderson","fromEmail":"ryan@michonline.com","sentAt":"2005-06-26T18:15:16Z","receivedAt":"2005-06-26T18:15:16Z","isPatch":true,"sender":{"key":"ryan@michonline.com","avatar":null},"body":"\nAdd git-relink-script\n\nThis will scan 2 or more object repositories and look for common objects, check\nif they are hardlinked, and replace one with a hardlink to the other if not.\n\nThis version warns when skipping files because of size differences, and\nhandle more than 2 repositories automatically.\n\nSigned-off-by: Ryan Anderson <ryan@michonline.com>\n\ndiff --git a/Makefile b/Makefile\n--- a/Makefile\n+++ b/Makefile\n@@ -25,7 +25,7 @@ SCRIPTS=git git-apply-patch-script git-m\n \tgit-deltafy-script git-fetch-script git-status-script git-commit-script \\\n \tgit-log-script git-shortlog git-cvsimport-script git-diff-script \\\n \tgit-reset-script git-add-script git-checkout-script git-clone-script \\\n-\tgitk git-cherry git-rebase-script\n+\tgitk git-cherry git-rebase-script git-relink-script\n \n PROG=   git-update-cache git-diff-files git-init-db git-write-tree \\\n \tgit-read-tree git-commit-tree git-cat-file git-fsck-cache \\\ndiff --git a/git-relink-script b/git-relink-script\nnew file mode 100644\n--- /dev/null\n+++ b/git-relink-script\n@@ -0,0 +1,173 @@\n+#!/usr/bin/env perl\n+# Copyright 2005, Ryan Anderson <ryan@michonline.com>\n+# Distribution permitted under the GPL v2, as distributed\n+# by the Free Software Foundation.\n+# Later versions of the GPL at the discretion of Linus Torvalds\n+#\n+# Scan two git object-trees, and hardlink any common objects between them.\n+\n+use 5.006;\n+use strict;\n+use warnings;\n+use Getopt::Long;\n+\n+sub get_canonical_form($);\n+sub do_scan_directory($$$);\n+sub compare_two_files($$);\n+sub usage();\n+sub link_two_files($$);\n+\n+# stats\n+my $total_linked = 0;\n+my $total_already = 0;\n+my ($linked,$already);\n+\n+my $fail_on_different_sizes = 0;\n+my $help = 0;\n+GetOptions(\"safe\" => \\$fail_on_different_sizes,\n+\t   \"help\" => \\$help);\n+\n+usage() if $help;\n+\n+my (@dirs) = @ARGV;\n+\n+usage() if (!defined $dirs[0] || !defined $dirs[1]);\n+\n+$_ = get_canonical_form($_) foreach (@dirs);\n+\n+my $master_dir = pop @dirs;\n+\n+opendir(D,$master_dir . \"objects/\")\n+\tor die \"Failed to open $master_dir/objects/ : $!\";\n+\n+my @hashdirs = grep !/^\\.{1,2}$/, readdir(D);\n+\n+foreach my $repo (@dirs) {\n+\t$linked = 0;\n+\t$already = 0;\n+\tprintf(\"Searching '%s' and '%s' for common objects and hardlinking them...\\n\",\n+\t\t$master_dir,$repo);\n+\n+\tforeach my $hashdir (@hashdirs) {\n+\t\tdo_scan_directory($master_dir, $hashdir, $repo);\n+\t}\n+\n+\tprintf(\"Linked %d files, %d were already linked.\\n\",$linked, $already);\n+\n+\t$total_linked += $linked;\n+\t$total_already += $already;\n+}\n+\n+printf(\"Totals: Linked %d files, %d were already linked.\\n\",\n+\t$total_linked, $total_already);\n+\n+\n+sub do_scan_directory($$$) {\n+\tmy ($srcdir, $subdir, $dstdir) = @_;\n+\n+\tmy $sfulldir = sprintf(\"%sobjects/%s/\",$srcdir,$subdir);\n+\tmy $dfulldir = sprintf(\"%sobjects/%s/\",$dstdir,$subdir);\n+\n+\topendir(S,$sfulldir)\n+\t\tor die \"Failed to opendir $sfulldir: $!\";\n+\n+\tforeach my $file (grep(!/\\.{1,2}$/, readdir(S))) {\n+\t\tmy $sfilename = $sfulldir . $file;\n+\t\tmy $dfilename = $dfulldir . $file;\n+\n+\t\tcompare_two_files($sfilename,$dfilename);\n+\n+\t}\n+\tclosedir(S);\n+}\n+\n+sub compare_two_files($$) {\n+\tmy ($sfilename, $dfilename) = @_;\n+\n+\t# Perl's stat returns relevant information as follows:\n+\t# 0 = dev number\n+\t# 1 = inode number\n+\t# 7 = size\n+\tmy @sstatinfo = stat($sfilename);\n+\tmy @dstatinfo = stat($dfilename);\n+\n+\tif (@sstatinfo == 0 && @dstatinfo == 0) {\n+\t\tdie sprintf(\"Stat of both %s and %s failed: %s\\n\",$sfilename, $dfilename, $!);\n+\n+\t} elsif (@dstatinfo == 0) {\n+\t\treturn;\n+\t}\n+\n+\tif ( ($sstatinfo[0] == $dstatinfo[0]) &&\n+\t     ($sstatinfo[1] != $dstatinfo[1])) {\n+\t\tif ($sstatinfo[7] == $dstatinfo[7]) {\n+\t\t\tlink_two_files($sfilename, $dfilename);\n+\n+\t\t} else {\n+\t\t\tmy $err = sprintf(\"ERROR: File sizes are not the same, cannot relink %s to %s.\\n\",\n+\t\t\t\t$sfilename, $dfilename);\n+\t\t\tif ($fail_on_different_sizes) {\n+\t\t\t\tdie $err;\n+\t\t\t} else {\n+\t\t\t\twarn $err;\n+\t\t\t}\n+\t\t}\n+\n+\t} elsif ( ($sstatinfo[0] == $dstatinfo[0]) &&\n+\t     ($sstatinfo[1] == $dstatinfo[1])) {\n+\t\t$already++;\n+\t}\n+}\n+\n+sub get_canonical_form($) {\n+\tmy $dir = shift;\n+\tmy $original = $dir;\n+\n+\tdie \"$dir is not a directory.\" unless -d $dir;\n+\n+\t$dir .= \"/\" unless $dir =~ m#/$#;\n+\t$dir .= \".git/\" unless $dir =~ m#\\.git/$#;\n+\n+\tdie \"$original does not have a .git/ subdirectory.\\n\" unless -d $dir;\n+\n+\treturn $dir;\n+}\n+\n+sub link_two_files($$) {\n+\tmy ($sfilename, $dfilename) = @_;\n+\tmy $tmpdname = sprintf(\"%s.old\",$dfilename);\n+\trename($dfilename,$tmpdname)\n+\t\tor die sprintf(\"Failure renaming %s to %s: %s\",\n+\t\t\t$dfilename, $tmpdname, $!);\n+\n+\tif (! link($sfilename,$dfilename)) {\n+\t\tmy $failtxt = \"\";\n+\t\tunless (rename($tmpdname,$dfilename)) {\n+\t\t\t$failtxt = sprintf(\n+\t\t\t\t\"Git Repository containing %s is probably corrupted, \" .\n+\t\t\t\t\"please copy '%s' to '%s' to fix.\\n\",\n+\t\t\t\t$tmpdname, $dfilename);\n+\t\t}\n+\n+\t\tdie sprintf(\"Failed to link %s to %s: %s\\n%s\" .\n+\t\t\t$sfilename, $dfilename,\n+\t\t\t$!, $dfilename, $failtxt);\n+\t}\n+\n+\tunlink($tmpdname)\n+\t\tor die sprintf(\"Unlink of %s failed: %s\\n\",\n+\t\t\t$dfilename, $!);\n+\n+\t$linked++;\n+}\n+\n+\n+sub usage() {\n+\tprint(\"Usage: $0 [--safe] <dir> [<dir> ...] <master_dir> \\n\");\n+\tprint(\"All directories should contain a .git/objects/ subdirectory.\\n\");\n+\tprint(\"Options\\n\");\n+\tprint(\"\\t--safe\\t\" .\n+\t\t\"Stops if two objects with the same hash exist but \" .\n+\t\t\"have different sizes.  Default is to warn and continue.\\n\");\n+\texit(1);\n+}\n-- \n\nRyan Anderson\n  sometimes Pug Majere\n"},{"id":"5279","messageId":"42BEF5CA.1060809@pobox.com","threadId":"1028","inReplyTo":"20050626181516.GC20369@mythryan2.michonline.com","subject":"Re: [PATCH] Add git-relink-script to fix up missing hardlinks","fromName":"Jeff Garzik","fromEmail":"jgarzik@pobox.com","sentAt":"2005-06-26T18:36:58Z","receivedAt":"2005-06-26T18:36:58Z","isPatch":true,"sender":{"key":"jgarzik@pobox.com","avatar":null},"body":"Ryan Anderson wrote:\n> Add git-relink-script\n> \n> This will scan 2 or more object repositories and look for common objects, check\n> if they are hardlinked, and replace one with a hardlink to the other if not.\n> \n> This version warns when skipping files because of size differences, and\n> handle more than 2 repositories automatically.\n> \n> Signed-off-by: Ryan Anderson <ryan@michonline.com>\n\nThanks for posting this.\n"},{"id":"5281","messageId":"7v7jghq6lt.fsf@assigned-by-dhcp.cox.net","threadId":"1028","inReplyTo":"20050626181516.GC20369@mythryan2.michonline.com","subject":"Re: [PATCH] Add git-relink-script to fix up missing hardlinks","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-06-26T19:07:58Z","receivedAt":"2005-06-26T19:07:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Not that I think it matters that much anymore since I am\nproposing removal of \"delta\" support and Linus seems to be\ninclined in the same direction, but I said \"most of the time\" in\nthe earlier message on this same topic for a reason:\n\n    Message-ID: <7vy89h36da.fsf@assigned-by-dhcp.cox.net>\n    Subject: Re: RFE: git relink\n    Date: Fri, 10 Jun 2005 20:44:01 -0700\n    References: <42A88C07.5050907@pobox.com>\n\n    Whoever is doing this script needs to be a bit careful.\n\n    ...\n\n    Ryan Anderson code will notice delta vs full object case most of\n    the time because it checks and makes sure the sizes of\n    corresponding files from two repositories match.  The problem\n    with the code is that it dies, instead of just ignoring, when\n    size differs....\n\nYour latest version has an option not to die which is very good\n[*1*], but in a very narrow corner case, without comparing the\nfile contents, I think the code would still do a wrong thing.\nTwo trees can store the same object both in delitified form but\nbased on different base objects, and the deltified\nrepresentation still having the same length, no?  And I suspect\nyou would end up linking them together, corrupting one of the\ntrees.\n\nOf course, even when you do not have \"delta\", if an object in\none tree is corrupted (but has the correct size), you would end\nup relinking the corrupt one into another tree, nuking a good\ncopy, if you do not compare the file contents.\n\nIf/when/after the proposed removal of \"delta\" support happens, I\nthink the correct way to do git-relink-script would be to keep\nmost of your latest version intact, except:\n\n (1) make it always die when you see differences in size.\n     Without \"delta\" in the repository, SHA1 files that\n     represent the same object must have the same size.\n\n (2) make --safe also check on file contents.  You do not need\n     the flag for the \"delta\" reason anymore, so I am suggesting\n     reusing the flag to detect file corruption, to be extra\n     safe, when the user permits you to spend cycles to be more\n     careful.\n\n[Footnote]\n\n*1* and other parts of the script all look nicely done.\n"},{"id":"5283","messageId":"7vy88wq5hs.fsf@assigned-by-dhcp.cox.net","threadId":"1028","inReplyTo":"7v7jghq6lt.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Add git-relink-script to fix up missing hardlinks","fromName":"Junio C Hamano","fromEmail":"junkio@cox.net","sentAt":"2005-06-26T19:31:59Z","receivedAt":"2005-06-26T19:31:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":">>>>> \"JCH\" == Junio C Hamano <junkio@cox.net> writes:\n\nJCH> Your latest version has an option not to die which is very\nJCH> good, but in a very narrow corner case, without comparing\nJCH> the file contents, I think the code would still do a wrong\nJCH> thing.\n\nHaving said that, the corner case is narrow enough (and\nhopefully to be gone soon) that I think the current version is\nperfectly acceptable for inclusion.\n\nLinus, please apply.  What it does is useful to encourage the\n\"one topic, one tree\" use pattern, the officially recommended\nway IIUC.\n\nI am a bit puzzled, though, why Jeff was the original requestor\nfor this feature --- I thought he handles 50 heads in one\nrepository which means there is no multiple repositories to\nrelink across.\n"},{"id":"5286","messageId":"42BF058F.8010301@pobox.com","threadId":"1028","inReplyTo":"7vy88wq5hs.fsf@assigned-by-dhcp.cox.net","subject":"Re: [PATCH] Add git-relink-script to fix up missing hardlinks","fromName":"Jeff Garzik","fromEmail":"jgarzik@pobox.com","sentAt":"2005-06-26T19:44:15Z","receivedAt":"2005-06-26T19:44:15Z","isPatch":true,"sender":{"key":"jgarzik@pobox.com","avatar":null},"body":"Junio C Hamano wrote:\n> I am a bit puzzled, though, why Jeff was the original requestor\n> for this feature --- I thought he handles 50 heads in one\n> repository which means there is no multiple repositories to\n> relink across.\n\nSure there are.  Just watching my submissions on the mailing list, you \ncan see ones mentioned such as \"misc-2.6\", \"libata-dev\", \"netdev-2.6\", etc.:\n\n[jgarzik@pretzel repo]$ ls -FC\n  config-2.4      ethtool/    libata-dev/  netdev-2.6/     sparse/\n  config-2.6      git/        linux-2.6/   old-SCM/\n  config-2.6-uml  git-tools/  misc-2.6/    scsi-misc-2.6/\n\nAnd I always keep an unmodified 'vanilla' tree from which everything is \nsourced (and hardlinked to).\n\nIt's a categorization system, a namespace.  Top-level repositories are \nbroad categories, and branches sub-divide those categories.\n\n\tJeff\n"},{"id":"5295","messageId":"20050627011113.GA26049@delft.aura.cs.cmu.edu","threadId":"1028","inReplyTo":"42BF058F.8010301@pobox.com","subject":"Re: [PATCH] Add git-relink-script to fix up missing hardlinks","fromName":"Jan Harkes","fromEmail":"jaharkes@cs.cmu.edu","sentAt":"2005-06-27T01:11:13Z","receivedAt":"2005-06-27T01:11:13Z","isPatch":true,"sender":{"key":"jaharkes@cs.cmu.edu","avatar":"https://gravatar.com/avatar/cf95aecd150ca8ef33d6edc337ac4bb9e13aa4246fc3679257d578c7fddc1633?d=mp&s=160"},"body":"On Sun, Jun 26, 2005 at 03:44:15PM -0400, Jeff Garzik wrote:\n> Junio C Hamano wrote:\n> >I am a bit puzzled, though, why Jeff was the original requestor\n> >for this feature --- I thought he handles 50 heads in one\n> >repository which means there is no multiple repositories to\n> >relink across.\n> \n> Sure there are.  Just watching my submissions on the mailing list, you \n> can see ones mentioned such as \"misc-2.6\", \"libata-dev\", \"netdev-2.6\", etc.:\n> \n> [jgarzik@pretzel repo]$ ls -FC\n>  config-2.4      ethtool/    libata-dev/  netdev-2.6/     sparse/\n>  config-2.6      git/        linux-2.6/   old-SCM/\n>  config-2.6-uml  git-tools/  misc-2.6/    scsi-misc-2.6/\n\nI actually have been using subdirectories in refs/heads with quite a bit\nof success. All of the core tools have no problem with them and only\ngitweb and gitk need some small changes to show them correctly.\n\nThe subdirectories in refs/heads are user specific in my case, this end\nup being pretty useful when combined with Coda's directory ACLs, each\nuser can maintain their own branches, but cannot modify anyone elses.\nAll developers have insert, read and lookup rights on the main objects\nrepository, they can add new objects, but not remove or overwrite any\nexisting ones.\n\nMy branch names end up looking something like 'jaharkes/wdonly',\n'awolbach/expand', etc. and it works like a charm.\n\nJan\n"}]}