threads / discuss / 35647

git-p4: exception when cloning a perforce repository

Subject: git-p4: exception when cloning a perforce repository

## tl;dr

11 messages between Jan 13, 2014 and Jan 20, 2014.

replies: 10people: 2as markdown or json

Damien Gérard· Jan 13, 2014, 13:37 UTC · lore
Hi !
I am trying to clone a perforce repository via git and I am having the following backtrace :
{14:20}~/projects/####:master ✗ ➭ git p4 clone //depot/@all .
Importing revision …
[...]
Importing revision 59702 (45%)Traceback (most recent call last):
  File "/opt/git/libexec/git-core/git-p4", line 3287, in <module>
    main()
  File "/opt/git/libexec/git-core/git-p4", line 3281, in main
    if not cmd.run(args):
  File "/opt/git/libexec/git-core/git-p4", line 3155, in run
    if not P4Sync.run(self, depotPaths):
  File "/opt/git/libexec/git-core/git-p4", line 3008, in run
    self.importChanges(changes)
  File "/opt/git/libexec/git-core/git-p4", line 2680, in importChanges
    self.initialParent)
  File "/opt/git/libexec/git-core/git-p4", line 2304, in commit
    self.streamP4Files(new_files)
  File "/opt/git/libexec/git-core/git-p4", line 2218, in streamP4Files
    cb=streamP4FilesCbSelf)
  File "/opt/git/libexec/git-core/git-p4", line 482, in p4CmdList
    cb(entry)
  File "/opt/git/libexec/git-core/git-p4", line 2212, in streamP4FilesCbSelf
    self.streamP4FilesCb(entry)
  File "/opt/git/libexec/git-core/git-p4", line 2167, in streamP4FilesCb
    self.streamOneP4File(self.stream_file, self.stream_contents)
  File "/opt/git/libexec/git-core/git-p4", line 2078, in streamOneP4File
    if data[-1] == '\n':
IndexError: string index out of range

git —version: git version 1.8.5.2.309.ga25014b [last commit from master from github.com/git/git] os : ubuntu 13.10

Any ideas ? :)

Best regards, Damien

Pete Wyckoff· Jan 14, 2014, 00:18 UTC · re: Damien Gérard · lore

Re: git-p4: exception when cloning a perforce repository

damien@iwi.me wrote on Mon, 13 Jan 2014 14:37 +0100:
Show 6 quoted lines
> I am trying to clone a perforce repository via git and I am having the following backtrace :
> 
> {14:20}~/projects/####:master ✗ ➭ git p4 clone //depot/@all .
> Importing revision …
> [...]
> Importing revision 59702 (45%)Traceback (most recent call last):
[..]
Show 6 quoted lines
>   File "/opt/git/libexec/git-core/git-p4", line 2078, in streamOneP4File
>     if data[-1] == '\n':
> IndexError: string index out of range
> 
> git —version: git version 1.8.5.2.309.ga25014b [last commit from master from github.com/git/git]
> os : ubuntu 13.10
This code:
        if type_base == "symlink":
            git_mode = "120000"
            # p4 print on a symlink sometimes contains "target\n";
            # if it does, remove the newline
            data = ''.join(contents)
  ==>       if data[-1] == '\n':
                contents = [data[:-1]]
            else:
                contents = [data]

means that data is an empty string. Implies you've got a symlink pointing to nothing. Is that even possible?

It could be this is a regression introduced at 1292df1 (git-p4: Fix occasional truncation of symlink contents., 2013-08-08). The old way of doing data[:-1] unconditionally would have worked but was broken for other reasons.

Could you investigate the symlink a bit? We're looking for one in change 59702 that points to nowhere. Maybe do:

    $ p4 describe -s 59702

and see if you can figure out which of those could be a symlink, then inspect it:

    $ p4 fstat //depot/symlink@59702
    (probably shows it is "headRev 1")
    $ p4 print -q //depot/symlink#1
    $ p4 print -q //depot/symlink#1 | od -c
Thanks for checking this depot info first.
		-- Pete
Pete Wyckoff· Jan 14, 2014, 23:24 UTC · re: Pete Wyckoff · lore

Re: git-p4: exception when cloning a perforce repository

pw@padd.com wrote on Mon, 13 Jan 2014 19:18 -0500:
Show 51 quoted lines
> damien@iwi.me wrote on Mon, 13 Jan 2014 14:37 +0100:
> > I am trying to clone a perforce repository via git and I am having the following backtrace :
> > 
> > {14:20}~/projects/####:master ✗ ➭ git p4 clone //depot/@all .
> > Importing revision …
> > [...]
> > Importing revision 59702 (45%)Traceback (most recent call last):
> [..]
> >   File "/opt/git/libexec/git-core/git-p4", line 2078, in streamOneP4File
> >     if data[-1] == '\n':
> > IndexError: string index out of range
> > 
> > git —version: git version 1.8.5.2.309.ga25014b [last commit from master from github.com/git/git]
> > os : ubuntu 13.10
> 
> This code:
> 
>         if type_base == "symlink":
>             git_mode = "120000"
>             # p4 print on a symlink sometimes contains "target\n";
>             # if it does, remove the newline
>             data = ''.join(contents)
>   ==>       if data[-1] == '\n':
>                 contents = [data[:-1]]
>             else:
>                 contents = [data]
> 
> means that data is an empty string.  Implies you've got a
> symlink pointing to nothing.  Is that even possible?
> 
> It could be this is a regression introduced at 1292df1 (git-p4:
> Fix occasional truncation of symlink contents., 2013-08-08).  The
> old way of doing data[:-1] unconditionally would have worked but
> was broken for other reasons.
> 
> Could you investigate the symlink a bit?  We're looking for
> one in change 59702 that points to nowhere.  Maybe do:
> 
>     $ p4 describe -s 59702
> 
> and see if you can figure out which of those could be a symlink, then
> inspect it:
> 
>     $ p4 fstat //depot/symlink@59702
>     (probably shows it is "headRev 1")
> 
>     $ p4 print -q //depot/symlink#1
> 
>     $ p4 print -q //depot/symlink#1 | od -c
> 
> Thanks for checking this depot info first.

I've tried to hack a test that produces a null symlink, and having done so, find an error later on trying to generate a symlink that points to "". So the "easy" fix of checking for an empty string is unlikely to work for your repo.

Curious as to how you managed to generate such a thing. If you find the file, and can get at the p4 depot, the full ,v file would be interesting too.

		-- Pete
Damien Gérard· Jan 15, 2014, 08:56 UTC · re: Pete Wyckoff · lore

Re: git-p4: exception when cloning a perforce repository

On 15 Jan 2014, at 00:24, Pete Wyckoff <pw@padd.com> wrote:
Show 30 quoted lines
> pw@padd.com wrote on Mon, 13 Jan 2014 19:18 -0500:
>> damien@iwi.me wrote on Mon, 13 Jan 2014 14:37 +0100:
>>> I am trying to clone a perforce repository via git and I am having the following backtrace :
>>> 
>>> {14:20}~/projects/####:master ✗ ➭ git p4 clone //depot/@all .
>>> Importing revision …
>>> [...]
>>> Importing revision 59702 (45%)Traceback (most recent call last):
>> [..]
>>>  File "/opt/git/libexec/git-core/git-p4", line 2078, in streamOneP4File
>>>    if data[-1] == '\n':
>>> IndexError: string index out of range
>>> 
>>> git —version: git version 1.8.5.2.309.ga25014b [last commit from master from github.com/git/git]
>>> os : ubuntu 13.10
>> 
>> This code:
>> 
>>        if type_base == "symlink":
>>            git_mode = "120000"
>>            # p4 print on a symlink sometimes contains "target\n";
>>            # if it does, remove the newline
>>            data = ''.join(contents)
>>  ==>       if data[-1] == '\n':
>>                contents = [data[:-1]]
>>            else:
>>                contents = [data]
>> 
>> means that data is an empty string.  Implies you've got a
>> symlink pointing to nothing.  Is that even possible?
It does not seem so but I am so sure.
Show 32 quoted lines
>> It could be this is a regression introduced at 1292df1 (git-p4:
>> Fix occasional truncation of symlink contents., 2013-08-08).  The
>> old way of doing data[:-1] unconditionally would have worked but
>> was broken for other reasons.
>> 
>> Could you investigate the symlink a bit?  We're looking for
>> one in change 59702 that points to nowhere.  Maybe do:
>> 
>>    $ p4 describe -s 59702
>> 
>> and see if you can figure out which of those could be a symlink, then
>> inspect it:
>> 
>>    $ p4 fstat //depot/symlink@59702
>>    (probably shows it is "headRev 1")
>> 
>>    $ p4 print -q //depot/symlink#1
>> 
>>    $ p4 print -q //depot/symlink#1 | od -c
>> 
>> Thanks for checking this depot info first.
> 
> I've tried to hack a test that produces a null symlink,
> and having done so, find an error later on trying to
> generate a symlink that points to "".  So the "easy"
> fix of checking for an empty string is unlikely to work
> for your repo.
> 
> Curious as to how you managed to generate such a thing.
> If you find the file, and can get at the p4 depot, the
> full ,v file would be interesting too.
> 
Indeed, those files are symlinks actually.  But it sems they are all valid.
Here is what I can get :
Change 59702 by ##############@VS9 on 2009/03/24 15:53:39
	OpenSSL 0.9.8j
Affected files ...

... //depot/openssl/0.9.8j/openssl/include/openssl/aes.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/asn1.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/asn1_mac.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/asn1t.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/bio.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/blowfish.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/bn.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/buffer.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/cast.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/comp.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/conf.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/conf_api.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/crypto.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/des.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/des_old.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/dh.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/dsa.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/dso.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/dtls1.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/e_os2.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/ebcdic.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/ec.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/ecdh.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/ecdsa.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/engine.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/err.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/evp.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/fips.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/fips_rand.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/hmac.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/idea.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/krb5_asn.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/kssl.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/lhash.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/md2.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/md4.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/md5.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/obj_mac.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/objects.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/ocsp.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/opensslconf.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/opensslv.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/ossl_typ.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/pem.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/pem2.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/pkcs12.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/pkcs7.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/pq_compat.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/pqueue.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/rand.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/rc2.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/rc4.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/ripemd.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/rsa.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/safestack.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/sha.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/ssl.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/ssl2.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/ssl23.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/ssl3.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/stack.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/store.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/symhacks.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/tls1.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/tmdiff.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/txt_db.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/ui.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/ui_compat.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/x509.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/x509_vfy.h#2 edit ... //depot/openssl/0.9.8j/openssl/include/openssl/x509v3.h#2 edit

Just in case :

$ p4 describe -s 59700 59700 - no such changelist.

p4 fstat //depot/openssl/0.9.8j/openssl/include/openssl/bn.h@59702 ... depotFile //depot/openssl/0.9.8j/openssl/include/openssl/bn.h ... headAction edit ... headType symlink ... headTime 1237906419 ... headRev 2 ... headChange 59702 ... headModTime 1231329423

p4 print -q //depot/openssl/0.9.8j/openssl/include/openssl/bn.h#2 | od -c 0000000

p4 print //depot/openssl/0.9.8j/openssl/include/openssl/bn.h#1 //depot/openssl/0.9.8j/openssl/include/openssl/bn.h#1 - add change 59574 (text)

 p4 print  //depot/openssl/0.9.8j/openssl/include/openssl/bn.h#2
//depot/openssl/0.9.8j/openssl/include/openssl/bn.h#2 - edit change 59702 (symlink)
Pete Wyckoff· Jan 16, 2014, 13:08 UTC · re: Damien Gérard · lore

Re: git-p4: exception when cloning a perforce repository

damien@iwi.me wrote on Wed, 15 Jan 2014 09:56 +0100:
Show 16 quoted lines
> p4 fstat  //depot/openssl/0.9.8j/openssl/include/openssl/bn.h@59702 
> ... depotFile //depot/openssl/0.9.8j/openssl/include/openssl/bn.h
> ... headAction edit
> ... headType symlink
> ... headTime 1237906419
> ... headRev 2
> ... headChange 59702
> ... headModTime 1231329423
> 
> p4 print -q //depot/openssl/0.9.8j/openssl/include/openssl/bn.h#2 | od -c
> 0000000
> 
> p4 print  //depot/openssl/0.9.8j/openssl/include/openssl/bn.h#1        
> //depot/openssl/0.9.8j/openssl/include/openssl/bn.h#1 - add change 59574 (text)
>  p4 print  //depot/openssl/0.9.8j/openssl/include/openssl/bn.h#2
> //depot/openssl/0.9.8j/openssl/include/openssl/bn.h#2 - edit change 59702 (symlink)

That's interesting. When I do the equivalent "p4 print" commands it shows something like this.

arf-git-test$ p4 fstat //depot/bn.h ... depotFile //depot/bn.h ... clientFile /dev/shm/trash directory.t9802-git-p4-filetype/cli/bn.h ... isMapped ... headAction edit ... headType symlink ... headTime 1389876870 ... headRev 2 ... headChange 8 ... headModTime 1389876870 ... haveRev 2

arf-git-test$ p4 print //depot/bn.h#1 //depot/bn.h#1 - add change 7 (text) file-text

arf-git-test$ p4 print //depot/bn.h#2 //depot/bn.h#2 - edit change 8 (symlink) /elsewhere/bn.h

I don't know how you manage to get a symlink with an empty destination like that.

I'll work on a way to hack around this failure. In the mean time, if you're game, it might be fun to see what p4 does with such a repository. You could make a client for just that little subdir, check out at 59702 and see what is there:

mkdir testmess
cd testmess
cat <<EOF | p4 client -i
Client: testmess
Description: testmess
Root: $(pwd)
View: //depot/openssl/0.9.8j/openssl/include/openssl/... //testmess/...
EOF

then take a look at how p4 represents the "empty" symlink in the filesystem:

p4 sync @59702 ls -la bn.h

		-- Pete
Damien Gérard· Jan 16, 2014, 13:46 UTC · re: Pete Wyckoff · lore

Re: git-p4: exception when cloning a perforce repository

On 16 Jan 2014, at 14:08, Pete Wyckoff <pw@padd.com> wrote:
Show 63 quoted lines
> damien@iwi.me wrote on Wed, 15 Jan 2014 09:56 +0100:
>> p4 fstat  //depot/openssl/0.9.8j/openssl/include/openssl/bn.h@59702 
>> ... depotFile //depot/openssl/0.9.8j/openssl/include/openssl/bn.h
>> ... headAction edit
>> ... headType symlink
>> ... headTime 1237906419
>> ... headRev 2
>> ... headChange 59702
>> ... headModTime 1231329423
>> 
>> p4 print -q //depot/openssl/0.9.8j/openssl/include/openssl/bn.h#2 | od -c
>> 0000000
>> 
>> p4 print  //depot/openssl/0.9.8j/openssl/include/openssl/bn.h#1        
>> //depot/openssl/0.9.8j/openssl/include/openssl/bn.h#1 - add change 59574 (text)
>> p4 print  //depot/openssl/0.9.8j/openssl/include/openssl/bn.h#2
>> //depot/openssl/0.9.8j/openssl/include/openssl/bn.h#2 - edit change 59702 (symlink)
> 
> That's interesting.  When I do the equivalent "p4 print" commands
> it shows something like this.
> 
> arf-git-test$ p4 fstat //depot/bn.h
> ... depotFile //depot/bn.h
> ... clientFile /dev/shm/trash directory.t9802-git-p4-filetype/cli/bn.h
> ... isMapped 
> ... headAction edit
> ... headType symlink
> ... headTime 1389876870
> ... headRev 2
> ... headChange 8
> ... headModTime 1389876870
> ... haveRev 2
> 
> arf-git-test$ p4 print //depot/bn.h#1
> //depot/bn.h#1 - add change 7 (text)
> file-text
> 
> arf-git-test$ p4 print //depot/bn.h#2
> //depot/bn.h#2 - edit change 8 (symlink)
> /elsewhere/bn.h
> 
> I don't know how you manage to get a symlink with an empty
> destination like that.
> 
> I'll work on a way to hack around this failure.  In the mean time,
> if you're game, it might be fun to see what p4 does with such a
> repository.  You could make a client for just that little subdir,
> check out at 59702 and see what is there:
> 
> mkdir testmess
> cd testmess
> cat <<EOF | p4 client -i
> Client: testmess
> Description: testmess
> Root: $(pwd)
> View: //depot/openssl/0.9.8j/openssl/include/openssl/... //testmess/...
> EOF
> 
> then take a look at how p4 represents the "empty" symlink
> in the filesystem:
> 
> p4 sync @59702
> ls -la bn.h
I’ve tried exactly your commands, and I’ve got an empty folder..

{14:38}~/p4/testmess ➭ p4 sync @59702 //depot/openssl/0.9.8j/openssl/include/openssl/aes.h#2 - refreshing /home/dgerard/p4/testmess/aes.h //depot/openssl/0.9.8j/openssl/include/openssl/asn1.h#2 - refreshing /home/dgerard/p4/testmess/asn1.h //depot/openssl/0.9.8j/openssl/include/openssl/asn1_mac.h#2 - refreshing /home/dgerard/p4/testmess/asn1_mac.h //depot/openssl/0.9.8j/openssl/include/openssl/asn1t.h#2 - refreshing /home/dgerard/p4/testmess/asn1t.h //depot/openssl/0.9.8j/openssl/include/openssl/bio.h#2 - refreshing /home/dgerard/p4/testmess/bio.h //depot/openssl/0.9.8j/openssl/include/openssl/blowfish.h#2 - refreshing /home/dgerard/p4/testmess/blowfish.h //depot/openssl/0.9.8j/openssl/include/openssl/bn.h#2 - refreshing /home/dgerard/p4/testmess/bn.h //depot/openssl/0.9.8j/openssl/include/openssl/buffer.h#2 - refreshing /home/dgerard/p4/testmess/buffer.h […]

{14:39}~/p4/testmess ➭ ls -la total 12 drwxr-xr-x 2 dgerard dgerard 4096 janv. 16 14:37 . drwxr-xr-x 4 dgerard dgerard 4096 janv. 16 14:34 .. -rw-r--r-- 1 dgerard dgerard 93 janv. 16 14:37 .perforce 

Then I tried to sync the previous changeset, which is ok :

{14:44}~/p4/testmess ➭ p4 sync -f @59701 //depot/openssl/0.9.8j/openssl/include/openssl/aes.h#1 - updating /home/dgerard/p4/testmess/aes.h […]

{14:44}~/p4/testmess ➭ l total 0 -r--r--r-- 1 dgerard dgerard 0 janv. 16 14:44 aes.h -r--r--r-- 1 dgerard dgerard 0 janv. 16 14:44 asn1.h -r--r--r-- 1 dgerard dgerard 0 janv. 16 14:44 asn1_mac.h -r--r--r-- 1 dgerard dgerard 0 janv. 16 14:44 asn1t.h -r--r--r-- 1 dgerard dgerard 0 janv. 16 14:44 bio.h -r--r--r-- 1 dgerard dgerard 0 janv. 16 14:44 blowfish.h -r--r--r-- 1 dgerard dgerard 0 janv. 16 14:44 bn.h -r--r--r-- 1 dgerard dgerard 0 janv. 16 14:44 buffer.h […]

However, when trying to sync to the appropriate changeset :

{14:44}~/p4/testmess ➭ p4 sync -f @59702 //depot/openssl/0.9.8j/openssl/include/openssl/aes.h#2 - updating /home/dgerard/p4/testmess/aes.h rename: /home/dgerard/p4/testmess/aes.h: No such file or directory //depot/openssl/0.9.8j/openssl/include/openssl/asn1.h#2 - updating /home/dgerard/p4/testmess/asn1.h rename: /home/dgerard/p4/testmess/asn1.h: No such file or directory //depot/openssl/0.9.8j/openssl/include/openssl/asn1_mac.h#2 - updating /home/dgerard/p4/testmess/asn1_mac.h rename: /home/dgerard/p4/testmess/asn1_mac.h: No such file or directory //depot/openssl/0.9.8j/openssl/include/openssl/asn1t.h#2 - updating /home/dgerard/p4/testmess/asn1t.h rename: /home/dgerard/p4/testmess/asn1t.h: No such file or directory //depot/openssl/0.9.8j/openssl/include/openssl/bio.h#2 - updating /home/dgerard/p4/testmess/bio.h rename: /home/dgerard/p4/testmess/bio.h: No such file or directory //depot/openssl/0.9.8j/openssl/include/openssl/blowfish.h#2 - updating /home/dgerard/p4/testmess/blowfish.h rename: /home/dgerard/p4/testmess/blowfish.h: No such file or directory //depot/openssl/0.9.8j/openssl/include/openssl/bn.h#2 - updating /home/dgerard/p4/testmess/bn.h rename: /home/dgerard/p4/testmess/bn.h: No such file or directory

And the folder remains untouched. Quite strange for me...

Pete Wyckoff· Jan 16, 2014, 14:45 UTC · re: Damien Gérard · lore

Re: git-p4: exception when cloning a perforce repository

damien@iwi.me wrote on Thu, 16 Jan 2014 14:46 +0100:
Show 128 quoted lines
> 
> On 16 Jan 2014, at 14:08, Pete Wyckoff <pw@padd.com> wrote:
> 
> > damien@iwi.me wrote on Wed, 15 Jan 2014 09:56 +0100:
> >> p4 fstat  //depot/openssl/0.9.8j/openssl/include/openssl/bn.h@59702 
> >> ... depotFile //depot/openssl/0.9.8j/openssl/include/openssl/bn.h
> >> ... headAction edit
> >> ... headType symlink
> >> ... headTime 1237906419
> >> ... headRev 2
> >> ... headChange 59702
> >> ... headModTime 1231329423
> >> 
> >> p4 print -q //depot/openssl/0.9.8j/openssl/include/openssl/bn.h#2 | od -c
> >> 0000000
> >> 
> >> p4 print  //depot/openssl/0.9.8j/openssl/include/openssl/bn.h#1        
> >> //depot/openssl/0.9.8j/openssl/include/openssl/bn.h#1 - add change 59574 (text)
> >> p4 print  //depot/openssl/0.9.8j/openssl/include/openssl/bn.h#2
> >> //depot/openssl/0.9.8j/openssl/include/openssl/bn.h#2 - edit change 59702 (symlink)
> > 
> > That's interesting.  When I do the equivalent "p4 print" commands
> > it shows something like this.
> > 
> > arf-git-test$ p4 fstat //depot/bn.h
> > ... depotFile //depot/bn.h
> > ... clientFile /dev/shm/trash directory.t9802-git-p4-filetype/cli/bn.h
> > ... isMapped 
> > ... headAction edit
> > ... headType symlink
> > ... headTime 1389876870
> > ... headRev 2
> > ... headChange 8
> > ... headModTime 1389876870
> > ... haveRev 2
> > 
> > arf-git-test$ p4 print //depot/bn.h#1
> > //depot/bn.h#1 - add change 7 (text)
> > file-text
> > 
> > arf-git-test$ p4 print //depot/bn.h#2
> > //depot/bn.h#2 - edit change 8 (symlink)
> > /elsewhere/bn.h
> > 
> > I don't know how you manage to get a symlink with an empty
> > destination like that.
> > 
> > I'll work on a way to hack around this failure.  In the mean time,
> > if you're game, it might be fun to see what p4 does with such a
> > repository.  You could make a client for just that little subdir,
> > check out at 59702 and see what is there:
> > 
> > mkdir testmess
> > cd testmess
> > cat <<EOF | p4 client -i
> > Client: testmess
> > Description: testmess
> > Root: $(pwd)
> > View: //depot/openssl/0.9.8j/openssl/include/openssl/... //testmess/...
> > EOF
> > 
> > then take a look at how p4 represents the "empty" symlink
> > in the filesystem:
> > 
> > p4 sync @59702
> > ls -la bn.h
> 
> I’ve tried exactly your commands, and I’ve got an empty folder..
> 
> {14:38}~/p4/testmess ➭ p4 sync @59702
> //depot/openssl/0.9.8j/openssl/include/openssl/aes.h#2 - refreshing /home/dgerard/p4/testmess/aes.h
> //depot/openssl/0.9.8j/openssl/include/openssl/asn1.h#2 - refreshing /home/dgerard/p4/testmess/asn1.h
> //depot/openssl/0.9.8j/openssl/include/openssl/asn1_mac.h#2 - refreshing /home/dgerard/p4/testmess/asn1_mac.h
> //depot/openssl/0.9.8j/openssl/include/openssl/asn1t.h#2 - refreshing /home/dgerard/p4/testmess/asn1t.h
> //depot/openssl/0.9.8j/openssl/include/openssl/bio.h#2 - refreshing /home/dgerard/p4/testmess/bio.h
> //depot/openssl/0.9.8j/openssl/include/openssl/blowfish.h#2 - refreshing /home/dgerard/p4/testmess/blowfish.h
> //depot/openssl/0.9.8j/openssl/include/openssl/bn.h#2 - refreshing /home/dgerard/p4/testmess/bn.h
> //depot/openssl/0.9.8j/openssl/include/openssl/buffer.h#2 - refreshing /home/dgerard/p4/testmess/buffer.h
> […]
> 
> 
> {14:39}~/p4/testmess ➭ ls -la
> total 12
> drwxr-xr-x 2 dgerard dgerard 4096 janv. 16 14:37 .
> drwxr-xr-x 4 dgerard dgerard 4096 janv. 16 14:34 ..
> -rw-r--r-- 1 dgerard dgerard   93 janv. 16 14:37 .perforce	
> 
> 
> Then I tried to sync the previous changeset, which is ok :
> 
> {14:44}~/p4/testmess ➭ p4 sync -f @59701
> //depot/openssl/0.9.8j/openssl/include/openssl/aes.h#1 - updating /home/dgerard/p4/testmess/aes.h
> […]
> 
> {14:44}~/p4/testmess ➭ l
> total 0
> -r--r--r-- 1 dgerard dgerard 0 janv. 16 14:44 aes.h
> -r--r--r-- 1 dgerard dgerard 0 janv. 16 14:44 asn1.h
> -r--r--r-- 1 dgerard dgerard 0 janv. 16 14:44 asn1_mac.h
> -r--r--r-- 1 dgerard dgerard 0 janv. 16 14:44 asn1t.h
> -r--r--r-- 1 dgerard dgerard 0 janv. 16 14:44 bio.h
> -r--r--r-- 1 dgerard dgerard 0 janv. 16 14:44 blowfish.h
> -r--r--r-- 1 dgerard dgerard 0 janv. 16 14:44 bn.h
> -r--r--r-- 1 dgerard dgerard 0 janv. 16 14:44 buffer.h
> […]
> 
> 
> 
> However, when trying to sync to the appropriate changeset :
> 
> {14:44}~/p4/testmess ➭ p4 sync -f @59702
> //depot/openssl/0.9.8j/openssl/include/openssl/aes.h#2 - updating /home/dgerard/p4/testmess/aes.h
> rename: /home/dgerard/p4/testmess/aes.h: No such file or directory
> //depot/openssl/0.9.8j/openssl/include/openssl/asn1.h#2 - updating /home/dgerard/p4/testmess/asn1.h
> rename: /home/dgerard/p4/testmess/asn1.h: No such file or directory
> //depot/openssl/0.9.8j/openssl/include/openssl/asn1_mac.h#2 - updating /home/dgerard/p4/testmess/asn1_mac.h
> rename: /home/dgerard/p4/testmess/asn1_mac.h: No such file or directory
> //depot/openssl/0.9.8j/openssl/include/openssl/asn1t.h#2 - updating /home/dgerard/p4/testmess/asn1t.h
> rename: /home/dgerard/p4/testmess/asn1t.h: No such file or directory
> //depot/openssl/0.9.8j/openssl/include/openssl/bio.h#2 - updating /home/dgerard/p4/testmess/bio.h
> rename: /home/dgerard/p4/testmess/bio.h: No such file or directory
> //depot/openssl/0.9.8j/openssl/include/openssl/blowfish.h#2 - updating /home/dgerard/p4/testmess/blowfish.h
> rename: /home/dgerard/p4/testmess/blowfish.h: No such file or directory
> //depot/openssl/0.9.8j/openssl/include/openssl/bn.h#2 - updating /home/dgerard/p4/testmess/bn.h
> rename: /home/dgerard/p4/testmess/bn.h: No such file or directory
> 
> And the folder remains untouched.
> Quite strange for me...

Oh cool, that helps a lot. P4 is just broken here, so we can get away with being a bit sloppy in git. I'll try just pretending "empty symlinks" are not in the repo. Hopefully you'll have a future commit in your p4 repo that brings back bn.h properly.

Still not sure about how I'll test this.
Thanks,
		-- Pete
Damien Gérard· Jan 16, 2014, 16:02 UTC · re: Pete Wyckoff · lore

Re: git-p4: exception when cloning a perforce repository

On 16 Jan 2014, at 15:45, Pete Wyckoff <pw@padd.com> wrote:
> Oh cool, that helps a lot.  P4 is just broken here, so we can get
> away with being a bit sloppy in git.  I'll try just pretending
> "empty symlinks" are not in the repo.  Hopefully you'll have a
> future commit in your p4 repo that brings back bn.h properly.

Thanks ! I would love to use git instead of perforce if possible :)

> Still not sure about how I'll test this.
I can test for you, no probleme with that.
Show 5 quoted lines
> 
> Thanks,
> 
> 		-- Pete
>
Pete Wyckoff· Jan 18, 2014, 18:22 UTC · re: Damien Gérard · lore

Re: git-p4: exception when cloning a perforce repository

damien@iwi.me wrote on Thu, 16 Jan 2014 17:02 +0100:
Show 14 quoted lines
> 
> On 16 Jan 2014, at 15:45, Pete Wyckoff <pw@padd.com> wrote:
> 
> > Oh cool, that helps a lot.  P4 is just broken here, so we can get
> > away with being a bit sloppy in git.  I'll try just pretending
> > "empty symlinks" are not in the repo.  Hopefully you'll have a
> > future commit in your p4 repo that brings back bn.h properly.
> 
> Thanks !
> I would love to use git instead of perforce if possible :)
> 
> > Still not sure about how I'll test this.
> 
> I can test for you, no probleme with that.

Any chance you can give this a go? I've a bigger patch in a longer series, but this should be the minimal fix. If it works, I'll ship it to Junio.

Thanks,
		-- Pete
----8<--------
From 8556ab04dd126184e26a380b7ed08998fd33debe Mon Sep 17 00:00:00 2001
From: Pete Wyckoff <pw@padd.com>
Date: Thu, 16 Jan 2014 18:34:09 -0500
Subject: [PATCH] git p4: work around p4 bug that causes empty symlinks
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

Damien Gérard highlights an interesting problem. Some p4 repositories end up with symlinks that have an empty target. It is not possible to create this with current p4, but they do indeed exist.

The effect in git p4 is that "p4 print" on the symlink returns an empty string, confusing the curret symlink-handling code.

In p4, syncing to a change that includes such a bogus symlink creates errors:

    //depot/empty-symlink - updating /home/me/p4/empty-symlink
    rename: /home/me/p4/empty-symlink: No such file or directory
and leaves no symlink.

Replicate the p4 behavior by ignoring these bogus symlinks. If they are fixed in later revisions, the symlink will be replaced properly.

Reported-by: Damien Gérard <damien@iwi.me>
Signed-off-by: Pete Wyckoff <pw@padd.com>
---
 git-p4.py | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
diff --git a/git-p4.py b/git-p4.py
index 5ea8bb8..e798ecf 100755
--- a/git-p4.py
+++ b/git-p4.py
@@ -2075,7 +2075,14 @@ class P4Sync(Command, P4UserMap):
             # p4 print on a symlink sometimes contains "target\n";
             # if it does, remove the newline
             data = ''.join(contents)
-            if data[-1] == '\n':
+            if not data:
+                # Some version of p4 allowed creating a symlink that pointed
+                # to nothing.  This causes p4 errors when checking out such
+                # a change, and errors here too.  Work around it by ignoring
+                # the bad symlink; hopefully a future change fixes it.
+                print "\nIgnoring empty symlink in %s" % file['depotFile']
+                return
+            elif data[-1] == '\n':
                 contents = [data[:-1]]
             else:
                 contents = [data]

-- 
1.8.5.2.320.g99957e5
Damien Gérard· Jan 20, 2014, 12:14 UTC · re: Pete Wyckoff · lore

Re: git-p4: exception when cloning a perforce repository

On 18 Jan 2014, at 19:22, Pete Wyckoff <pw@padd.com> wrote:
Show 19 quoted lines
> damien@iwi.me wrote on Thu, 16 Jan 2014 17:02 +0100:
>> 
>> On 16 Jan 2014, at 15:45, Pete Wyckoff <pw@padd.com> wrote:
>> 
>>> Oh cool, that helps a lot.  P4 is just broken here, so we can get
>>> away with being a bit sloppy in git.  I'll try just pretending
>>> "empty symlinks" are not in the repo.  Hopefully you'll have a
>>> future commit in your p4 repo that brings back bn.h properly.
>> 
>> Thanks !
>> I would love to use git instead of perforce if possible :)
>> 
>>> Still not sure about how I'll test this.
>> 
>> I can test for you, no probleme with that.
> 
> Any chance you can give this a go?  I've a bigger patch in
> a longer series, but this should be the minimal fix.  If it
> works, I'll ship it to Junio.

Yeah it seems to work fine ! I’ve finally imported all 152620 changesets :)

Thanks !
(git pull from this morning + patch)
Show 65 quoted lines
> 
> Thanks,
> 
> 		-- Pete
> 
> ----8<--------
> 
> From 8556ab04dd126184e26a380b7ed08998fd33debe Mon Sep 17 00:00:00 2001
> From: Pete Wyckoff <pw@padd.com>
> Date: Thu, 16 Jan 2014 18:34:09 -0500
> Subject: [PATCH] git p4: work around p4 bug that causes empty symlinks
> MIME-Version: 1.0
> Content-Type: text/plain; charset=UTF-8
> Content-Transfer-Encoding: 8bit
> 
> Damien Gérard highlights an interesting problem.  Some p4
> repositories end up with symlinks that have an empty target.  It
> is not possible to create this with current p4, but they do
> indeed exist.
> 
> The effect in git p4 is that "p4 print" on the symlink returns an
> empty string, confusing the curret symlink-handling code.
> 
> In p4, syncing to a change that includes such a bogus symlink
> creates errors:
> 
>    //depot/empty-symlink - updating /home/me/p4/empty-symlink
>    rename: /home/me/p4/empty-symlink: No such file or directory
> 
> and leaves no symlink.
> 
> Replicate the p4 behavior by ignoring these bogus symlinks.  If
> they are fixed in later revisions, the symlink will be replaced
> properly.
> 
> Reported-by: Damien Gérard <damien@iwi.me>
> Signed-off-by: Pete Wyckoff <pw@padd.com>
> ---
> git-p4.py | 9 ++++++++-
> 1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/git-p4.py b/git-p4.py
> index 5ea8bb8..e798ecf 100755
> --- a/git-p4.py
> +++ b/git-p4.py
> @@ -2075,7 +2075,14 @@ class P4Sync(Command, P4UserMap):
>             # p4 print on a symlink sometimes contains "target\n";
>             # if it does, remove the newline
>             data = ''.join(contents)
> -            if data[-1] == '\n':
> +            if not data:
> +                # Some version of p4 allowed creating a symlink that pointed
> +                # to nothing.  This causes p4 errors when checking out such
> +                # a change, and errors here too.  Work around it by ignoring
> +                # the bad symlink; hopefully a future change fixes it.
> +                print "\nIgnoring empty symlink in %s" % file['depotFile']
> +                return
> +            elif data[-1] == '\n':
>                 contents = [data[:-1]]
>             else:
>                 contents = [data]
> -- 
> 1.8.5.2.320.g99957e5
> 
>
Damien Gérard· Jan 20, 2014, 14:01 UTC · re: Pete Wyckoff · lore

Re: git-p4: exception when cloning a perforce repository

On 18 Jan 2014, at 19:22, Pete Wyckoff <pw@padd.com> wrote:
Show 20 quoted lines
> damien@iwi.me wrote on Thu, 16 Jan 2014 17:02 +0100:
>> 
>> On 16 Jan 2014, at 15:45, Pete Wyckoff <pw@padd.com> wrote:
>> 
>>> Oh cool, that helps a lot.  P4 is just broken here, so we can get
>>> away with being a bit sloppy in git.  I'll try just pretending
>>> "empty symlinks" are not in the repo.  Hopefully you'll have a
>>> future commit in your p4 repo that brings back bn.h properly.
>> 
>> Thanks !
>> I would love to use git instead of perforce if possible :)
>> 
>>> Still not sure about how I'll test this.
>> 
>> I can test for you, no probleme with that.
> 
> Any chance you can give this a go?  I've a bigger patch in
> a longer series, but this should be the minimal fix.  If it
> works, I'll ship it to Junio.
> 
Just for info, it works but it seems there are still some issues when a git repository is present within the perforce repo :

error: Invalid path 'Tools/Doc/bin/yuidoc/.git/FETCH_HEAD' error: Invalid path 'Tools/Doc/bin/yuidoc/.git/HEAD' error: Invalid path 'Tools/Doc/bin/yuidoc/.git/ORIG_HEAD’ [...]

Those files have been added then removed in another commit
I’ve have to make git reset —hard ‘HEAD^’ && git p4 sync to a clean staging area right after the "clone".
Show 64 quoted lines
> Thanks,
> 
> 		-- Pete
> 
> ----8<--------
> 
> From 8556ab04dd126184e26a380b7ed08998fd33debe Mon Sep 17 00:00:00 2001
> From: Pete Wyckoff <pw@padd.com>
> Date: Thu, 16 Jan 2014 18:34:09 -0500
> Subject: [PATCH] git p4: work around p4 bug that causes empty symlinks
> MIME-Version: 1.0
> Content-Type: text/plain; charset=UTF-8
> Content-Transfer-Encoding: 8bit
> 
> Damien Gérard highlights an interesting problem.  Some p4
> repositories end up with symlinks that have an empty target.  It
> is not possible to create this with current p4, but they do
> indeed exist.
> 
> The effect in git p4 is that "p4 print" on the symlink returns an
> empty string, confusing the curret symlink-handling code.
> 
> In p4, syncing to a change that includes such a bogus symlink
> creates errors:
> 
>    //depot/empty-symlink - updating /home/me/p4/empty-symlink
>    rename: /home/me/p4/empty-symlink: No such file or directory
> 
> and leaves no symlink.
> 
> Replicate the p4 behavior by ignoring these bogus symlinks.  If
> they are fixed in later revisions, the symlink will be replaced
> properly.
> 
> Reported-by: Damien Gérard <damien@iwi.me>
> Signed-off-by: Pete Wyckoff <pw@padd.com>
> ---
> git-p4.py | 9 ++++++++-
> 1 file changed, 8 insertions(+), 1 deletion(-)
> 
> diff --git a/git-p4.py b/git-p4.py
> index 5ea8bb8..e798ecf 100755
> --- a/git-p4.py
> +++ b/git-p4.py
> @@ -2075,7 +2075,14 @@ class P4Sync(Command, P4UserMap):
>             # p4 print on a symlink sometimes contains "target\n";
>             # if it does, remove the newline
>             data = ''.join(contents)
> -            if data[-1] == '\n':
> +            if not data:
> +                # Some version of p4 allowed creating a symlink that pointed
> +                # to nothing.  This causes p4 errors when checking out such
> +                # a change, and errors here too.  Work around it by ignoring
> +                # the bad symlink; hopefully a future change fixes it.
> +                print "\nIgnoring empty symlink in %s" % file['depotFile']
> +                return
> +            elif data[-1] == '\n':
>                 contents = [data[:-1]]
>             else:
>                 contents = [data]
> -- 
> 1.8.5.2.320.g99957e5
> 
>

← back to recent threads