git-svn and a nested branches folder

> I'm having some trouble with using git-svn to fetch a repository, and I
> think it's because the repository doesn't store branches as a flat list
> directly under the 'branches' directory.
>
> Basically, we have a structure like this:
>
> |
> +-trunk
> +-tags
> +-branches
>   + category-a
>     + branch-a
>     + branch-b
>   + category-b
>     + branch-c
>     + branch-d
>
> etc. category-a and category-b are simple directories created using svn
> mkdir. The branches are created using svn cp.
>
> It helps us to organise the branches better, but the rationale is
> besides the point. The problem is that git-svn seems to want to
> treat category-a and category-b as branches, which isn't right at
> all. As a result, git-svn seems to skip most (if not all) revisions
> that occur in these directories and creates a lot of entries in
> unhandled.log.

> Russ Brown <pickscrape@gmail.com> writes:
> 
>> I'm having some trouble with using git-svn to fetch a repository, and I
>> think it's because the repository doesn't store branches as a flat list
>> directly under the 'branches' directory.
>>
>> Basically, we have a structure like this:
>>
>> |
>> +-trunk
>> +-tags
>> +-branches
>>   + category-a
>>     + branch-a
>>     + branch-b
>>   + category-b
>>     + branch-c
>>     + branch-d
>>
>> etc. category-a and category-b are simple directories created using svn
>> mkdir. The branches are created using svn cp.
>>
>> It helps us to organise the branches better, but the rationale is
>> besides the point. The problem is that git-svn seems to want to
>> treat category-a and category-b as branches, which isn't right at
>> all. As a result, git-svn seems to skip most (if not all) revisions
>> that occur in these directories and creates a lot of entries in
>> unhandled.log.
> 
> So what did you specify in your .git/config file regarding the svn
> structure?
> 

> David Kastrup wrote:
>> Russ Brown <pickscrape@gmail.com> writes:
>> 
>>> I'm having some trouble with using git-svn to fetch a repository, and I
>>> think it's because the repository doesn't store branches as a flat list
>>> directly under the 'branches' directory.
>>>
>>> Basically, we have a structure like this:
>>>
>>> |
>>> +-trunk
>>> +-tags
>>> +-branches
>>>   + category-a
>>>     + branch-a
>>>     + branch-b
>>>   + category-b
>>>     + branch-c
>>>     + branch-d
>>>
>>> etc. category-a and category-b are simple directories created using svn
>>> mkdir. The branches are created using svn cp.
>>>
>>> It helps us to organise the branches better, but the rationale is
>>> besides the point. The problem is that git-svn seems to want to
>>> treat category-a and category-b as branches, which isn't right at
>>> all. As a result, git-svn seems to skip most (if not all) revisions
>>> that occur in these directories and creates a lot of entries in
>>> unhandled.log.
>> 
>> So what did you specify in your .git/config file regarding the svn
>> structure?
>
> I specified the 'branches' directory, but that's because earlier in
> the life of the repo we did just do the flat branch layout, but
> decided to make it more structured once that got unwieldy.

> Russ Brown <pickscrape@gmail.com> writes:
> 
>> David Kastrup wrote:
>>> Russ Brown <pickscrape@gmail.com> writes:
>>>
>>>> I'm having some trouble with using git-svn to fetch a repository, and I
>>>> think it's because the repository doesn't store branches as a flat list
>>>> directly under the 'branches' directory.
>>>>
>>>> Basically, we have a structure like this:
>>>>
>>>> |
>>>> +-trunk
>>>> +-tags
>>>> +-branches
>>>>   + category-a
>>>>     + branch-a
>>>>     + branch-b
>>>>   + category-b
>>>>     + branch-c
>>>>     + branch-d
>>>>
>>>> etc. category-a and category-b are simple directories created using svn
>>>> mkdir. The branches are created using svn cp.
>>>>
>>>> It helps us to organise the branches better, but the rationale is
>>>> besides the point. The problem is that git-svn seems to want to
>>>> treat category-a and category-b as branches, which isn't right at
>>>> all. As a result, git-svn seems to skip most (if not all) revisions
>>>> that occur in these directories and creates a lot of entries in
>>>> unhandled.log.
>>> So what did you specify in your .git/config file regarding the svn
>>> structure?
>> I specified the 'branches' directory, but that's because earlier in
>> the life of the repo we did just do the flat branch layout, but
>> decided to make it more structured once that got unwieldy.
> 
> Cough, cough.  _What_ did you specify in your .git/config file
> regarding the svn structure?  Please quote the section.
> 

>> Is it possible to specify more than one folder for the branches
>> option?
> 
> It is possible to adapt the config section to the actual layout.  If
> not otherwise, by starting with
> git svn init
> with a clean slate, editing the config file, and only then actually
> fetching stuff.
> 
> However, git-svn will not magically start guessing that you changed
> your structure around.  You have to edit the configuration
> appropriately.
> 

> David Kastrup wrote:
> > Russ Brown <pickscrape@gmail.com> writes:
> > 
> >> David Kastrup wrote:
> >>> Russ Brown <pickscrape@gmail.com> writes:
> >>>
> >>>> I'm having some trouble with using git-svn to fetch a repository, and I
> >>>> think it's because the repository doesn't store branches as a flat list
> >>>> directly under the 'branches' directory.
> >>>>
> >>>> Basically, we have a structure like this:
> >>>>
> >>>> |
> >>>> +-trunk
> >>>> +-tags
> >>>> +-branches
> >>>>   + category-a
> >>>>     + branch-a
> >>>>     + branch-b
> >>>>   + category-b
> >>>>     + branch-c
> >>>>     + branch-d
> >>>>
> >>>> etc. category-a and category-b are simple directories created using svn
> >>>> mkdir. The branches are created using svn cp.
> >>>>
> >>>> It helps us to organise the branches better, but the rationale is
> >>>> besides the point. The problem is that git-svn seems to want to
> >>>> treat category-a and category-b as branches, which isn't right at
> >>>> all. As a result, git-svn seems to skip most (if not all) revisions
> >>>> that occur in these directories and creates a lot of entries in
> >>>> unhandled.log.
> >>> So what did you specify in your .git/config file regarding the svn
> >>> structure?
> >> I specified the 'branches' directory, but that's because earlier in
> >> the life of the repo we did just do the flat branch layout, but
> >> decided to make it more structured once that got unwieldy.
> > 
> > Cough, cough.  _What_ did you specify in your .git/config file
> > regarding the svn structure?  Please quote the section.
> > 
> 
> Erm, sorry.
> 
> [svn-remote "svn"]
>         url = svn://svn.<name>.com
>         fetch = trunk:refs/remotes/trunk
>         branches = branches/*:refs/remotes/*
>         tags = tags/*:refs/remotes/tags/*
> 
> (URL changed in case it annoys my employers)
> 
> I didn't write this by hand: it was generated by git-svn init.
> 

> On Tue, Sep 04, 2007 at 10:21:22AM -0500, Russ Brown wrote:
>> David Kastrup wrote:
>>> Russ Brown <pickscrape@gmail.com> writes:
>>>
>>>> David Kastrup wrote:
>>>>> Russ Brown <pickscrape@gmail.com> writes:
>>>>>
>>>>>> I'm having some trouble with using git-svn to fetch a repository, and I
>>>>>> think it's because the repository doesn't store branches as a flat list
>>>>>> directly under the 'branches' directory.
>>>>>>
>>>>>> Basically, we have a structure like this:
>>>>>>
>>>>>> |
>>>>>> +-trunk
>>>>>> +-tags
>>>>>> +-branches
>>>>>>   + category-a
>>>>>>     + branch-a
>>>>>>     + branch-b
>>>>>>   + category-b
>>>>>>     + branch-c
>>>>>>     + branch-d
>>>>>>
>>>>>> etc. category-a and category-b are simple directories created using svn
>>>>>> mkdir. The branches are created using svn cp.
>>>>>>
>>>>>> It helps us to organise the branches better, but the rationale is
>>>>>> besides the point. The problem is that git-svn seems to want to
>>>>>> treat category-a and category-b as branches, which isn't right at
>>>>>> all. As a result, git-svn seems to skip most (if not all) revisions
>>>>>> that occur in these directories and creates a lot of entries in
>>>>>> unhandled.log.
>>>>> So what did you specify in your .git/config file regarding the svn
>>>>> structure?
>>>> I specified the 'branches' directory, but that's because earlier in
>>>> the life of the repo we did just do the flat branch layout, but
>>>> decided to make it more structured once that got unwieldy.
>>> Cough, cough.  _What_ did you specify in your .git/config file
>>> regarding the svn structure?  Please quote the section.
>>>
>> Erm, sorry.
>>
>> [svn-remote "svn"]
>>         url = svn://svn.<name>.com
>>         fetch = trunk:refs/remotes/trunk
>>         branches = branches/*:refs/remotes/*
>>         tags = tags/*:refs/remotes/tags/*
>>
>> (URL changed in case it annoys my employers)
>>
>> I didn't write this by hand: it was generated by git-svn init.
>>
> 
> Try something like this:
> 
> [svn-remote "svn"]
> 	# trunk
> 	fetch = trunk:refs/remotes/trunk
> 
> 	# branches
> 	fetch = branches/category-a/branch_a:refs/remotes/svn/branch_a
> 	fetch = branches/category-a/branch_b:refs/remotes/svn/branch_b
> 	fetch = branches/category-b/branch_c:refs/remotes/svn/branch_c
> 
> 	# tags
> 	tags = tags/*:refs/remotes/tags/*
> 
> 
> (Not sure if wildcards will work here, but I'm sure you could experiment and try
>  it out :-)
> 

> I'm having some trouble with using git-svn to fetch a repository, and I
> think it's because the repository doesn't store branches as a flat list
> directly under the 'branches' directory.
> 
> Basically, we have a structure like this:
> 
> |
> +-trunk
> +-tags
> +-branches
>   + category-a
>     + branch-a
>     + branch-b
>   + category-b
>     + branch-c
>     + branch-d
> 
> etc. category-a and category-b are simple directories created using svn
> mkdir. The branches are created using svn cp.
> 
> It helps us to organise the branches better, but the rationale is
> besides the point. The problem is that git-svn seems to want to treat
> category-a and category-b as branches, which isn't right at all. As a
> result, git-svn seems to skip most (if not all) revisions that occur in
> these directories and creates a lot of entries in unhandled.log.

> I've had a quick look at the source myself, but perl isn't my strong
> point. What I think it should do is something like recurse down the tree
> from the directory given looking for folders that copy from trunk (or
> some other branch that already exists). That would work perfectly well
> for the default flat branch storage method as well as the one we use.

> The only other problem is in branch naming, which could clash if you
> only use the outer-most directory name, so I'd suggest something that
> involves concatenating the folders in the path relative to 'branches' to
> keep them unique (if git can handle slashes in branch names then all the
> better).

> Russ Brown <pickscrape@gmail.com> wrote:
>> Hi,
> 
> Hi Russ,
> 
>> I'm having some trouble with using git-svn to fetch a repository, and I
>> think it's because the repository doesn't store branches as a flat list
>> directly under the 'branches' directory.
>>
>> Basically, we have a structure like this:
>>
>> |
>> +-trunk
>> +-tags
>> +-branches
>>   + category-a
>>     + branch-a
>>     + branch-b
>>   + category-b
>>     + branch-c
>>     + branch-d
>>
>> etc. category-a and category-b are simple directories created using svn
>> mkdir. The branches are created using svn cp.
>>
>> It helps us to organise the branches better, but the rationale is
>> besides the point. The problem is that git-svn seems to want to treat
>> category-a and category-b as branches, which isn't right at all. As a
>> result, git-svn seems to skip most (if not all) revisions that occur in
>> these directories and creates a lot of entries in unhandled.log.
> 
> This is a known problem with the way git-svn handles branches and tags.
> Nested branch (and tags) structures aren't supported with globbing and
> so you can't have more than one branches/tags specification in your
> config.
> 
>> I've also encountered an index corruption in one of the
>> .git/svn/<branch>/index files which I think it probably related.
> 
> Not sure about that.  git-svn should detect and attempt to fix index
> corruptions (which happened for me since I interrupt git-svn quite
> often when working on it).
> 

>> I've had a quick look at the source myself, but perl isn't my strong
>> point. What I think it should do is something like recurse down the tree
>> from the directory given looking for folders that copy from trunk (or
>> some other branch that already exists). That would work perfectly well
>> for the default flat branch storage method as well as the one we use.
> 
> The globbing functionality of branches in git-svn is some of the ugliest
> code I've ever written in my life.  Somehow I got it working for the
> simple general cases but I've been afraid to touch it for a while...
> 

>> The only other problem is in branch naming, which could clash if you
>> only use the outer-most directory name, so I'd suggest something that
>> involves concatenating the folders in the path relative to 'branches' to
>> keep them unique (if git can handle slashes in branch names then all the
>> better).
> 
> As Peter suggested, disable globbing for branches and use explicit
> fetch refspecs for now...
> 

> Eric Wong wrote:
> > Russ Brown <pickscrape@gmail.com> wrote:
> >> Basically, we have a structure like this:
> >>
> >> |
> >> +-trunk
> >> +-tags
> >> +-branches
> >>   + category-a
> >>     + branch-a
> >>     + branch-b
> >>   + category-b
> >>     + branch-c
> >>     + branch-d

> >> The only other problem is in branch naming, which could clash if you
> >> only use the outer-most directory name, so I'd suggest something that
> >> involves concatenating the folders in the path relative to 'branches' to
> >> keep them unique (if git can handle slashes in branch names then all the
> >> better).
> > 
> > As Peter suggested, disable globbing for branches and use explicit
> > fetch refspecs for now...
> > 
> 
> I've actually knocked up a rough script which generates a list of
> refspec lines for you given a repo URL, trunk reference and branches
> directory. It uses svn log -v --xml and pipes it through a couple of
> XSLT templates, and basically looks for all copies that copy from trunk
> (recursively: so it includes branches of branches too). I can post it to
> the list if you'd find it useful or interesting.
> 
> It's generating output that looks sensible to me, but the results aren't
> quite what I'd expected. I'll paste a sample in here in case there's
> anything obvious someone might spot that I've missed
> 
> # This line was generated by git-svn init, and I kept it
> fetch = all/trunk:refs/remotes/trunk
> 
> # These lines generated by my tool, dirnames replaced for security reasons:
> 
> fetch = branches/folder/projecta:refs/remotes/svn/folder/projecta
> fetch = branches/folder/projectb:refs/remotes/svn/folder/projectb
> fetch = branches/folder/projectc:refs/remotes/svn/folder/projectc
> fetch = branches/folder/projectd:refs/remotes/svn/folder/projectd
> fetch = branches/folder/projecte:refs/remotes/svn/folder/projecte
> fetch = branches/folder/projectf:refs/remotes/svn/folder/projectf
> fetch = branches/folder/projectg:refs/remotes/svn/folder/projectg
> 
> git branch -a doesn't list any of those branches after fetch completes.
> Looking back at the output from fetch, all revisions applied were to trunk.
> 
> Anything wrong with those fetch lines?

> Russ Brown <pickscrape@gmail.com> wrote:
>> Eric Wong wrote:
>>> Russ Brown <pickscrape@gmail.com> wrote:
>>>> Basically, we have a structure like this:
>>>>
>>>> |
>>>> +-trunk
>>>> +-tags
>>>> +-branches
>>>>   + category-a
>>>>     + branch-a
>>>>     + branch-b
>>>>   + category-b
>>>>     + branch-c
>>>>     + branch-d
> 
>>>> The only other problem is in branch naming, which could clash if you
>>>> only use the outer-most directory name, so I'd suggest something that
>>>> involves concatenating the folders in the path relative to 'branches' to
>>>> keep them unique (if git can handle slashes in branch names then all the
>>>> better).
>>> As Peter suggested, disable globbing for branches and use explicit
>>> fetch refspecs for now...
>>>
>> I've actually knocked up a rough script which generates a list of
>> refspec lines for you given a repo URL, trunk reference and branches
>> directory. It uses svn log -v --xml and pipes it through a couple of
>> XSLT templates, and basically looks for all copies that copy from trunk
>> (recursively: so it includes branches of branches too). I can post it to
>> the list if you'd find it useful or interesting.
>>
>> It's generating output that looks sensible to me, but the results aren't
>> quite what I'd expected. I'll paste a sample in here in case there's
>> anything obvious someone might spot that I've missed
>>
>> # This line was generated by git-svn init, and I kept it
>> fetch = all/trunk:refs/remotes/trunk
>>
>> # These lines generated by my tool, dirnames replaced for security reasons:
>>
>> fetch = branches/folder/projecta:refs/remotes/svn/folder/projecta
>> fetch = branches/folder/projectb:refs/remotes/svn/folder/projectb
>> fetch = branches/folder/projectc:refs/remotes/svn/folder/projectc
>> fetch = branches/folder/projectd:refs/remotes/svn/folder/projectd
>> fetch = branches/folder/projecte:refs/remotes/svn/folder/projecte
>> fetch = branches/folder/projectf:refs/remotes/svn/folder/projectf
>> fetch = branches/folder/projectg:refs/remotes/svn/folder/projectg
>>
>> git branch -a doesn't list any of those branches after fetch completes.
>> Looking back at the output from fetch, all revisions applied were to trunk.
>>
>> Anything wrong with those fetch lines?
> 
> From your tree diagram, it seemed that trunk/ and branches/ were at
> the same depth in your SVN repository.  However, in your generated
> fetch lines they all started with "branches/" in front, yet your
> trunk fetch line had "all/" in front of trunk, so maybe prefixing
> the generated ones with "all/" helps?
>