threads / patch / 30240

patchExplain how svn-fe parses filenames in SVN dumps

Subject: [PATCH] Explain how svn-fe parses filenames in SVN dumps

## tl;dr

7 messages between Apr 14, 2012 and Apr 15, 2012. Diffs are folded; open one to read it.

replies: 6people: 2as markdown or json

Andrew Sayers· Apr 14, 2012, 17:03 UTC · lore

The documentation for the SVN dumpfile format says that filenames "may be interpreted as binary data in any encoding by client tools", but users might be surprised that svn-fe's handling differs from svn's.

Before version 1.2.0, `svn add` supported files containing characters in the range 0x01-0x1F, and Subversion still supports existing files that contain those characters. The newline character is explicitly discussed so that users with ancient repositories understand why they can't be supported by tools that read the SVN dump format.

The documentation for the SVN dumpfile format describes records as containing "a group of RFC822-style header lines", and its full text can be read as implying newline characters are reserved for use by the format. This reading is slightly charitable, but it avoids the need to discuss the format's design issues in a context where few readers will be interested.

Signed-off-by: Andrew Sayers <andrew-git@pileofstuff.org>
---
 contrib/svn-fe/svn-fe.txt |    8 ++++++++
 1 files changed, 8 insertions(+), 0 deletions(-)
Show changes to contrib/svn-fe/svn-fe.txt +8 −0
diff --git a/contrib/svn-fe/svn-fe.txt b/contrib/svn-fe/svn-fe.txt
index 1128ab2..c079abe 100644
--- a/contrib/svn-fe/svn-fe.txt
+++ b/contrib/svn-fe/svn-fe.txt
@@ -59,6 +59,14 @@ to put each project in its own repository and to separate the history
 of each branch.  The 'git filter-branch --subdirectory-filter' command
 may be useful for this purpose.
 
+Filenames are interpreted by svn-fe as binary data, and may contain
+any character except NUL (0x00) and newline (0x0A).  The NUL
+character is not valid in git paths, and the newline character is
+reserved for use by the (line-based) Subversion dumpfile format.
+This differs from Subversion, which requires filenames to contain
+only legal XML characters and disallows tabs characters, carriage
+returns and newlines.
+
 BUGS
 ----
 Empty directories and unknown properties are silently discarded.

-- 
1.7.1
Jonathan Nieder· Apr 14, 2012, 17:14 UTC · re: Andrew Sayers · lore

Re: [PATCH] Explain how svn-fe parses filenames in SVN dumps

Hi,
Andrew Sayers wrote:
> Before version 1.2.0, `svn add` supported files containing characters in the
> range 0x01-0x1F, and Subversion still supports existing files that contain
> those characters.
Because of the above,
[...]
Show 14 quoted lines
> +++ b/contrib/svn-fe/svn-fe.txt
> @@ -59,6 +59,14 @@ to put each project in its own repository and to separate the history
>  of each branch.  The 'git filter-branch --subdirectory-filter' command
>  may be useful for this purpose.
>  
> +Filenames are interpreted by svn-fe as binary data, and may contain
> +any character except NUL (0x00) and newline (0x0A).  The NUL
> +character is not valid in git paths, and the newline character is
> +reserved for use by the (line-based) Subversion dumpfile format.
> +This differs from Subversion, which requires filenames to contain
> +only legal XML characters and disallows tabs characters, carriage
> +returns and newlines.
> +
>  BUGS

this description and the location of this description seem quite misleading. Isn't what the reader needs to know something like the following?

	BUGS
	----
	Due to limitations in the Subversion dumpfile format, svn-fe
	does not support filenames with newlines.  Since version 1.2.0,
	"svn add" forbids adding such filenames but some historical
	repositories contain them.  An import can appear to succeed and
	produce incorrect results when such pathological filenames are
	present.

Thanks, Jonathan

Andrew Sayers· Apr 14, 2012, 17:37 UTC · re: Jonathan Nieder · lore

Re: [PATCH] Explain how svn-fe parses filenames in SVN dumps

On 14/04/12 18:14, Jonathan Nieder wrote:
Show 16 quoted lines
> this description and the location of this description seem quite
> misleading.  Isn't what the reader needs to know something like the
> following?
> 
> 	BUGS
> 	----
> 	Due to limitations in the Subversion dumpfile format, svn-fe
> 	does not support filenames with newlines.  Since version 1.2.0,
> 	"svn add" forbids adding such filenames but some historical
> 	repositories contain them.  An import can appear to succeed and
> 	produce incorrect results when such pathological filenames are
> 	present.
> 
> Thanks,
> Jonathan
>

I went back and forth a bit while writing the text. Newlines are only one special case, albeit an important one that I hadn't expressed clearly enough. For example, the handling of NUL characters is worse than newlines (a quick test suggests svn-fe terminates parsing altogether if it sees one), but SVN has never allowed the creation of files with NULs so arguably it's even less important.

I'm warming again to the idea of explicitly mentioning that newlines
cause breakage, but it feels like the wider story is worth telling too.
 If this were a man page I'd consider adding a section or something, but
I'm not sure what level of verbosity you're looking for in this file.
	- Andrew
Jonathan Nieder· Apr 14, 2012, 18:18 UTC · re: Andrew Sayers · lore

Re: [PATCH] Explain how svn-fe parses filenames in SVN dumps

Andrew Sayers wrote:
>                  For example, the handling of NUL characters is worse
> than newlines

Filenames are C strings. Filenames with NUL bytes simply do not exist, at least as long as one is using the ANSI C or POSIX interfaces for file access.

So as far as I can tell the story really is only about newlines. (If svn-fe tried to push history back to Subversion, life would presumably be more complicated.)

Hoping that clarifies a little, Jonathan

Andrew Sayers· Apr 14, 2012, 21:56 UTC · re: Jonathan Nieder · lore

Re: [PATCH] Explain how svn-fe parses filenames in SVN dumps

On 14/04/12 19:18, Jonathan Nieder wrote:
> So as far as I can tell the story really is only about newlines.  (If
> svn-fe tried to push history back to Subversion, life would presumably
> be more complicated.)

I think I've been trying to balance two incompatible use cases - ordinary users that just need a heads-up about a bug that might bite them, and authors of related tools (i.e. me) that need technical information about svn-fe.

I think your text serves ordinary users better, so I'll investigate pushing history back to Subversion tomorrow and think how to write something more technical. I'm afraid my patch etiquette fails me here though - is it better for me to roll your text into a new patch, or let you do that and submit my text separately?

	- Andrew
Jonathan Nieder· Apr 15, 2012, 05:08 UTC · re: Andrew Sayers · lore

Re: [PATCH] Explain how svn-fe parses filenames in SVN dumps

Andrew Sayers wrote:
>                            I'm afraid my patch etiquette fails me here
> though - is it better for me to roll your text into a new patch, or let
> you do that and submit my text separately?

If the example I sent looks good, please feel free to morph it into a new patch. Thanks again for your work, and glad I could help in some small way.

Ciao, Jonathan

← back to recent threads