parsecvs and unnamed branches

I'm getting thousands of messages about unnamed branches and even 'unnamed branch from master-UNNAMED-BRANCH'.

How do you get unnamed branches into CVS, are these check-in errors or are people actually working on unnamed branches? Or is parsecvs not finding all of the branch info?

On Fri, 2006-06-16 at 17:44 -0400, Jon Smirl wrote:

branch names rely on a special 'branch tag' in the "symbols" section of the CVS file, but actual branches are flagged directly in the revision list. I don't know how it happens, but ,v files often end up with branches in the revision tree which haven't an associated tag. Go figure.

For example, in the top level mozilla/Makefile.in,v file, you'll see a branch from version 1.36 with an initial commit 1.36.2.1. Using the wacky CVS branch revision numbering scheme, there should be an associated tag for version 1.36.0.2 (yes, the last two digits are flipped). But, none is present in the file.

The reverse situation also occurs, with tags for branches that have no revisions in the file. This case makes sense -- until you make a change in a file along a branch, there will be no other record in the file of where the branch came from.

I'd love to figure out a better mechanism for merging these nameless branches into the resulting repository, but I don't know how to correlate unnamed branches in one file with unnamed branches in other files.

The current scheme of making up a fixed name and hoping that there aren't multiple unmamed branches from the same root is probably fraught with peril.

On 6/16/06, Keith Packard <keithp@keithp.com> wrote:

I was reading the CVS manual and it talks about magic branch number as being the ones with zero in them. Doesn't go into a lot of detail. Apparently they are autogenerated internally.

http://ximbiot.com/cvs/wiki/index.php?title=CVS--Concurrent_Versions_System_v1.12.12.1:_Branching_and_merging#Magic_branch_numbers

On 6/16/06, Keith Packard <keithp@keithp.com> wrote:

There is a branch label for SeaMonkey_M8_BRANCH:1.36.0.4 that does seems to correspond to anything. Could that be the missing tag?

On Fri, 2006-06-16 at 18:39 -0400, Jon Smirl wrote:

There's no particular reason to believe it should be; remember that every file gets a 'magic branch' tag whenever you create a branch in the repository, but only files with commits along that branch ever see the revision tree information related to it, so you can expect to see many branch tags without associated branch revisions in any particular ,v file. Looking at the commits along 1.3.2, they don't seem particularily related to the SeaMonkey_M8 branch.

My parsecvs job died after 5 hours of CPU time. Does this tell you anything?

Pack pack-e28915a5ea09143a9139e84e24534ed888bf1c45 created

Error: branch cycle
*** glibc detected *** parsecvs: munmap_chunk(): invalid pointer: 0x0a820198 ***
*** glibc detected *** parsecvs: corrupted double-linked list: 0x45b1e158 ***
======= Backtrace: =========
/lib/libc.so.6[0x45a502c6]
/lib/libc.so.6[0x45a5235a]
/lib/libc.so.6(calloc+0x8d)[0x45a539a1]
/lib/ld-linux.so.2[0x459db1ba]
/lib/ld-linux.so.2[0x459d6f8a]
/lib/ld-linux.so.2[0x459d91e1]
/lib/ld-linux.so.2[0x459e2204]
/lib/ld-linux.so.2[0x459de7b9]
/lib/ld-linux.so.2[0x459e1d0a]
/lib/libc.so.6[0x45ae9c3e]
/lib/ld-linux.so.2[0x459de7b9]
/lib/libc.so.6(__libc_dlopen_mode+0x55)[0x45ae9dc9]
/lib/libc.so.6[0x45ac90f6]
/lib/libc.so.6(backtrace+0x109)[0x45ac9295]
/lib/libc.so.6[0x45a4aa61]
/lib/libc.so.6(__libc_free+0x179)[0x45a554f0]
parsecvs[0x804dec8]
parsecvs[0x804df1e]
parsecvs[0x804ccce]
/lib/libc.so.6(__libc_start_main+0xdc)[0x45a03724]
parsecvs[0x8049cc1]
======= Memory map: ========
08048000-08065000 r-xp 00000000 09:01 4052384
/home/jonsmirl/workspace/parsecvs/parsecvs
08065000-08067000 rw-p 0001c000 09:01 4052384
/home/jonsmirl/workspace/parsecvs/parsecvs
08067000-459c9000 rw-p 08067000 00:00 0          [heap]
459d1000-459ea000 r-xp 00000000 03:06 4243239    /lib/ld-2.4.so
459ea000-459eb000 r--p 00018000 03:06 4243239    /lib/ld-2.4.so
459eb000-459ec000 rw-p 00019000 03:06 4243239    /lib/ld-2.4.so
459ee000-45b1b000 r-xp 00000000 03:06 4243241    /lib/libc-2.4.so
45b1b000-45b1d000 r--p 0012d000 03:06 4243241    /lib/libc-2.4.so
45b1d000-45b1e000 rw-p 0012f000 03:06 4243241    /lib/libc-2.4.so
45b1e000-45b21000 rw-p 45b1e000 00:00 0
45b23000-45b25000 r-xp 00000000 03:06 4243258    /lib/libdl-2.4.so
45b25000-45b26000 r--p 00001000 03:06 4243258    /lib/libdl-2.4.so
45b26000-45b27000 rw-p 00002000 03:06 4243258    /lib/libdl-2.4.so
45c6e000-45c80000 r-xp 00000000 03:06 1782100    /usr/lib/libz.so.1.2.3
45c80000-45c81000 rw-p 00011000 03:06 1782100    /usr/lib/libz.so.1.2.3
46497000-46499000 r-xp 00000000 03:06 4244426    /lib/libcom_err.so.2.1
46499000-4649a000 rw-p 00001000 03:06 4244426    /lib/libcom_err.so.2.1
4649c000-464ab000 r-xp 00000000 03:06 4244425    /lib/libresolv-2.4.so
464ab000-464ac000 r--p 0000e000 03:06 4244425    /lib/libresolv-2.4.so
464ac000-464ad000 rw-p 0000f000 03:06 4244425    /lib/libresolv-2.4.so
464ad000-464af000 rw-p 464ad000 00:00 0
464bb000-465da000 r-xp 00000000 03:06 4244427    /lib/libcrypto.so.0.9.8a
465da000-465ed000 rw-p 0011e000 03:06 4244427    /lib/libcrypto.so.0.9.8a
465ed000-465f0000 rw-p 465ed000 00:00 0
465f2000-465f5000 r-xp 00000000 03:06 1785207    /usr/lib/libkrb5support.so.0.0
465f5000-465f6000 rw-p 00002000 03:06 1785207    /usr/lib/libkrb5support.so.0.0
465f8000-4666b000 r-xp 00000000 03:06 1788263    /usr/lib/libkrb5.so.3.2
4666b000-4666d000 rw-p 00073000 03:06 1788263    /usr/lib/libkrb5.so.3.2
4666f000-46687000 r-xp 00000000 03:06 1788264    /usr/lib/libgssapi_krb5.so.2.2
46687000-46688000 rw-p 00017000 03:06 1788264    /usr/lib/libgssapi_krb5.so.2.2
4668a000-466ae000 r-xp 00000000 03:06 1788262    /usr/lib/libk5crypto.so.3.0
466ae000-466af000 rw-p 00024000 03:06 1788262    /usr/lib/libk5crypto.so.3.0
467e0000-46821000 r-xp 00000000 03:06 4244428    /lib/libssl.so.0.9.8a
46821000-46825000 rw-p 00040000 03:06 4244428    /lib/libssl.so.0.9.8a
84400000-84421000 rw-p 84400000 00:00 0
84421000-84500000 ---p 84421000 00:00 0
84518000-84523000 r-xp 00000000 03:06 4244801
/lib/libgcc_s-4.1.1-20060525.so.1
84523000-84524000 rw-p 0000a000 03:06 4244801
/lib/libgcc_s-4.1.1-20060525.so.1
84538000-85438000 rw-p 84538000 00:00 0
854bc000-856bc000 rw-p 854bc000 00:00 0
8577f000-86b7f000 rw-p 8577f000 00:00 0
86bc1000-86dc1000 rw-p 86bc1000 00:00 0
86e65000-88465000 rw-p 86e65000 00:00 0
884e8000-893e8000 rw-p 884e8000 00:00 0
893e9000-896e9000 rw-p 893e9000 00:00 0
896ea000-89bea000 rw-p 896ea000 00:00 0
89c2c000-89f2c000 rw-p 89c2c000 00:00 0
89f4d000-8a14d000 rw-p 89f4d000 00:00 0
8a16e000-8a76e000 rw-p 8a16e000 00:00 0
8a76f000-8af6f000 rw-p 8a76f000 00:00 0
8b056000-8b156000 rw-p 8b056000 00:00 0
8b177000-8b477000 rw-p 8b177000 00:00 0
8b478000-8b978000 rw-p 8b478000 00:00 0
8b979000-8ba79000 rw-p 8b979000 00:00 0
8ba9a000-8bc9a000 rw-p 8ba9a000 00:00 0
8bd1e000-8c11e000 rw-p 8bd1e000 00:00 0
8c11f000-8c41f000 rw-p 8c11f000 00:00 0
8c420000-8c520000 rw-p 8c420000 00:00 0
8c582000-8c682000 rw-p 8c582000 00:00 0
8c683000-8d383000 rw-p 8c683000 00:00 0
8d426000-8d5260Aborted
[jonsmirl@jonsmirl parsecvs]$

Hi, Jon!

On Fri, 2006-06-16 at 23:02 -0400, Jon Smirl wrote:

Obviously, memory corruption. Valgrind is likely to help, but it may take 50 hours rather than 5. It may still be worth it. Make sure to use the latest version of Valgrind and compile parsecvs without optimization with full debug information. If you can get debug info for libc, install it (on Fedora: "yum install glibc-debuginfo").

You see, even some libc symbols can be found, but parsecvs is opaque. That's why debug information is useful. Make sure to keep the sources around for debugging.

On 6/16/06, Pavel Roskin <proski@gnu.org> wrote:

Parsecvs was compiled '-O2 -g' why didn't it decode the addresses to symbols?

The 'Error: branch cycle' message was critical, the app was in the process of doing exit clean up with the link list error was found. If the list is linked in a circle it is likely that the routine freeing it corrupted memory. So the real error is why did I get 'Error: branch cycle'.

On Fri, 2006-06-16 at 23:31 -0400, Jon Smirl wrote:

Sorry, I was too quick to put blame on you. Maybe glibc can only list its own symbols.

I could reproduce the problem trivially with a single file, and here's what Valgrind says:

==11154== Invalid free() / delete / delete[] ==11154== at 0x4905423: free (vg_replace_malloc.c:233) ==11154== by 0x40C136: git_pack_directory (git.c:620) ==11154== by 0x40C1B4: git_rev_list_pack (git.c:639) ==11154== by 0x4067DA: main (parsecvs.c:785)

git_pack_directory() tries to free() the result of git_system_to_string(), which is in turn a result of atom(). My understanding is that atoms should not be freed. They are not freed in other cases.

Patch:

On 6/17/06, Pavel Roskin <proski@gnu.org> wrote:

I had already caught that one, the fix was a few mails back. git_dir is an atom and shouldn't be freed with free.

After five hours I hit this: fprintf (stderr, "Error: branch cycle\n");

static rev_ref *
rev_ref_tsort (rev_ref *refs, rev_list *head)
{
    rev_ref *done = NULL;
    rev_ref **done_tail = &done;
    rev_ref *r, **prev;

//    fprintf (stderr, "Tsort refs:\n");
    while (refs) {
        for (prev = &refs; (r = *prev); prev = &(*prev)->next) {
            if (rev_ref_is_ready (r->name, head, done)) {
                break;
            }
        }
        if (!r) {
            fprintf (stderr, "Error: branch cycle\n");

            return NULL;
        }
        *prev = r->next;
        *done_tail = r;
//      fprintf (stderr, "\t%s\n", r->name);
        r->next = NULL;
        done_tail = &r->next;
    }
    return done;
}

which returned null up to here

    if (rev_mode == ExecuteGit && pack_objcount && autopack)
        git_rev_list_pack (pack_start, strip);
    load_status_next ();
    rl = rev_list_merge (head);

    if (rl) {
        switch (rev_mode) {
        case ExecuteGraph:
            dump_rev_graph (rl, NULL);
            break;
        case ExecuteSplits:
            dump_splits (rl);
            break;
        case ExecuteGit:
            git_rev_list_commit (rl, strip);
            break;
        }
    }
    if (rl)
        rev_list_free (rl, 0);
    while (head) {
        rl = head;
        head = head->next;
        rev_list_free (rl, 1);

    }
    discard_atoms ();
    rev_free_dirs ();
    rev_commit_cleanup ();
    git_free_author_map ();
    return err;

On Sat, 2006-06-17 at 00:15 -0400, Jon Smirl wrote:

I see it now. My patch was wrong - there is another free(git_dir) in the same function. Keith, please remove both.

This is more like a logical error. Maybe you actually have circling branches due to causality violations or something :-)

Sure, Valgrind would be still useful to make sure it's not something mundane.

On Sat, 2006-06-17 at 00:15 -0400, Jon Smirl wrote:

I haven't figured it out yet either; mine didn't detect the loop though, it just ended up spinning in the tsort code, unable to compute a valid order to execute branches in. Something funky must be up with the mozilla branches.

What this code does is find an order that will 'work' when computing branch contents. The requirement is that the 'parent' branch be computed before any 'child' branches.

It does this with a nice quadratic algorithm, building a list of 'ready' branches who have no 'unready' dependencies in any of the incoming file objects. If there are conflicts where one incoming file shows branch 'B' as the parent of branch 'A' while another shows branch 'A' as the parent of branch 'B', the sorting cannot succeed.

Ideally, I'd figure out a way to eliminate the parent/child relationship and just treat the branches as peers with a common ancestor. I haven't figure out how to manage that yet; attempting to find the precise divergence point where the child forks from the parent remains complicated, it seems like trying to do that without a strong parent/child relationship would be even more error prone.

Better error messsages here would clearly help discover which branches were in conflict, and show the files causing problems.

On 6/17/06, Keith Packard <keithp@keithp.com> wrote:

Have you checked parsecvs on the 38 test repositories in the cvs2svn source?

On Sat, 2006-06-17 at 01:51 -0400, Jon Smirl wrote:

Those all run to completion without significant error (there are a couple of tests with invalid symbol names that currently elicit errors).

I haven't validated that the imports are correct though; spot checks seem to indicate that the problems encountered during the cvs2svn development aren't the same as the problems we're finding.