git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 3/3] git-svn: Reduce temp file usage when dealing with non-links

From
EWEric Wong <normalperson@yhbt.net>
Date
Aug 13, 2008, 03:29 UTC
Message-ID
<20080813032956.GC5904@untitled>
In-Reply-To
<1218556876-26554-1-git-send-email-marcus@griep.us>
Marcus Griep <marcus@griep.us> wrote:
Show 21 quoted lines
> Currently, in sub 'close_file', git-svn creates a temporary file and
> copies the contents of the blob to be written into it. This is useful
> for symlinks because svn stores symlinks in the form:
> 
> link $FILE_PATH
> 
> Git creates a blob only out of '$FILE_PATH' and uses file mode to
> indicate that the blob should be interpreted as a symlink.
> 
> As git-hash-object is invoked with --stdin-paths, a duplicate of the
> link from svn must be created that leaves off the first five bytes,
> i.e. 'link '. However, this is wholly unnecessary for normal blobs,
> though, as we already have a temp file with their contents. Copying
> the entire file gains nothing, and effectively requires a file to be
> written twice before making it into the object db.
> 
> This patch corrects that issue, holding onto the substr-like
> duplication for symlinks, but skipping it altogether for normal blobs
> by reusing the existing temp file.
> 
> Signed-off-by: Marcus Griep <marcus@griep.us>
Thank you Marcus!
Acked-by: Eric Wong <normalperson@yhbt.net>
Show 71 quoted lines
> ---
>  git-svn.perl |   46 ++++++++++++++++++++++------------------------
>  1 files changed, 22 insertions(+), 24 deletions(-)
> 
> diff --git a/git-svn.perl b/git-svn.perl
> index 9eae5e8..95d1510 100755
> --- a/git-svn.perl
> +++ b/git-svn.perl
> @@ -3268,38 +3268,36 @@ sub close_file {
>  				    "expected: $exp\n    got: $got\n";
>  			}
>  		}
> -		sysseek($fh, 0, 0) or croak $!;
>  		if ($fb->{mode_b} == 120000) {
> -			eval {
> -				sysread($fh, my $buf, 5) == 5 or croak $!;
> -				$buf eq 'link ' or die "$path has mode 120000",
> -						       " but is not a link";
> -			};
> -			if ($@) {
> -				warn "$@\n";
> -				sysseek($fh, 0, 0) or croak $!;
> -			}
> -		}
> -
> -		my $tmp_fh = Git::temp_acquire('svn_hash');
> -		my $result;
> -		while ($result = sysread($fh, my $string, 1024)) {
> -			my $wrote = syswrite($tmp_fh, $string, $result);
> -			defined($wrote) && $wrote == $result
> -				or croak("write ",
> -					$tmp_fh->filename, ": $!\n");
> -		}
> -		defined $result or croak $!;
> +			sysseek($fh, 0, 0) or croak $!;
> +			sysread($fh, my $buf, 5) == 5 or croak $!;
>  
> +			unless ($buf eq 'link ') {
> +				warn "$path has mode 120000",
> +						" but is not a link\n";
> +			} else {
> +				my $tmp_fh = Git::temp_acquire('svn_hash');
> +				my $res;
> +				while ($res = sysread($fh, my $str, 1024)) {
> +					my $out = syswrite($tmp_fh, $str, $res);
> +					defined($out) && $out == $res
> +						or croak("write ",
> +							$tmp_fh->filename,
> +							": $!\n");
> +				}
> +				defined $result or croak $!;
>  
> -		Git::temp_release($fh, 1);
> +				($fh, $tmp_fh) = ($tmp_fh, $fh);
> +				Git::temp_release($tmp_fh, 1);
> +			}
> +		}
>  
>  		$hash = $::_repository->hash_and_insert_object(
> -				$tmp_fh->filename);
> +				$fh->filename);
>  		$hash =~ /^[a-f\d]{40}$/ or die "not a sha1: $hash\n";
>  
>  		Git::temp_release($fb->{base}, 1);
> -		Git::temp_release($tmp_fh, 1);
> +		Git::temp_release($fh, 1);
>  	} else {
>  		$hash = $fb->{blob} or die "no blob information\n";
>  	}
> -- 
> 1.6.0.rc2.6.g8eda3
Previous: Marcus GriepNext: Marcus Griep
Message 19 of 43 in “git-svn: Make it scream by minimizing temp files”
  1. git-svn: Make it scream by minimizing temp filesMarcus Griep, Aug 8, 2008
  2. Junio C HamanoAug 8, 2008
  3. Marcus GriepAug 9, 2008
  4. Eric WongAug 9, 2008
  5. Marcus GriepAug 9, 2008
  6. Eric WongAug 10, 2008
  7. Junio C HamanoAug 10, 2008
  8. Eric WongAug 10, 2008
  9. Junio C HamanoAug 10, 2008
  10. Eric WongAug 10, 2008
  11. 0/3 git-svn and temporary file improvementsMarcus Griep, Aug 11, 2008
  12. 1/3 Git.pm: Add faculties to allow temp files to be cachedMarcus Griep, Aug 11, 2008
  13. 2/3 git-svn: Make it scream by minimizing temp filesMarcus Griep, Aug 11, 2008
  14. 3/3 git-svn: Reduce temp file usage when dealing with non-linksMarcus Griep, Aug 11, 2008
  15. Eric WongAug 12, 2008
  16. Marcus GriepAug 12, 2008
  17. 3/3 git-svn: Reduce temp file usage when dealing with non-linksMarcus Griep, Aug 12, 2008
  18. 3/3 git-svn: Reduce temp file usage when dealing with non-linksMarcus Griep, Aug 12, 2008
  19. Eric WongAug 13, 2008
  20. Marcus GriepAug 13, 2008
  21. Eric WongAug 13, 2008
  22. Eric WongAug 12, 2008
  23. Marcus GriepAug 12, 2008
  24. 2/3 git-svn: Make it incrementally faster by minimizing temp filesMarcus Griep, Aug 12, 2008
  25. Eric WongAug 13, 2008
  26. Eric WongAug 12, 2008
  27. Marcus GriepAug 12, 2008
  28. 1/3 Git.pm: Add faculties to allow temp files to be cachedMarcus Griep, Aug 12, 2008
  29. Eric WongAug 13, 2008
  30. Lea WiemannAug 13, 2008
  31. Marcus GriepAug 13, 2008
  32. Marcus GriepAug 13, 2008
  33. Junio C HamanoAug 13, 2008
  34. Lea WiemannAug 13, 2008
  35. Git.pm: require Perl 5.6.1Lea Wiemann, Aug 13, 2008
  36. Eric WongAug 14, 2008
  37. Git.pm: Make File::Spec and File::Temp requirement lazyMarcus Griep, Aug 15, 2008
  38. Bryan DonlanAug 15, 2008
  39. Marcus GriepAug 15, 2008
  40. Git.pm: Make File::Spec and File::Temp requirement lazyMarcus Griep, Aug 15, 2008
  41. Miklos VajnaAug 13, 2008
  42. Junio C HamanoAug 14, 2008
  43. Marcus GriepAug 14, 2008

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.