{"thread":{"id":"37531","subject":"git-remote-fd problem","startedAt":"2014-09-10T13:29:00Z","lastAt":"2014-12-29T19:36:51Z","messageCount":4,"participants":["Jiri Sevcik","Ilari Liusvaara"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"249121","messageId":"CAOP4-939uAP2Tgofz01F40_Eb0F8GxBsU49HxNvAiyBCW6H08w@mail.gmail.com","threadId":"37531","inReplyTo":null,"subject":"git-remote-fd problem","fromName":"Jiri Sevcik","fromEmail":"jsevcik14@gmail.com","sentAt":"2014-09-10T13:29:00Z","receivedAt":"2014-09-10T13:29:00Z","isPatch":false,"sender":{"key":"jsevcik14@gmail.com","avatar":null},"body":"Hi!\nI have problem with using git-remote-fd function. I create two local\npipes for communication for locally running process git pull. Then I\nstart git-upload-pack on remote side (this process is started in\ndifferent part of my code  and it works correctly). Communication runs\nsuccessfully and when remote side finish, I want to close local pipes\nto finish local process. I call close function to all of these pipes,\nbut unfortunatelly local process doesnt finish (in this moment\ntransfered data aresaved in /.git/object/pack folder in .pack files)\nand hanging on. But if I kill child of this local process, downloaded\nfiles are unpacked and files are created sucessfully. I think this way\nis not so correct. Can someone help me and tell me what am I doing\nwrong? Thank You. Code is in an attachement (its part of big system).\n\n\n    #create local pipes for cimmunication\n    local_r, local_w = os.pipe()\n    remote_r, remote_w = os.pipe()\n\n    #start local git process\n    client_process = subprocess.Popen(\"/usr/bin/git pull fd::{0},{1}\".format(remote_r, local_w,), shell=True)\n    \n    #start git process on remote side \n    remoteGit = proc.runDaemon(\"sudo git-upload-pack /tmp/testGit\")\n\n    #set\n    epoll = select.epoll()\n\n    epoll.register(local_r, select.EPOLLIN)\n    #register from remote side\n    epoll.register(proc.fd, select.EPOLLIN)\n\n    while True:\n        events = epoll.poll(1)\n\n        for fd, event in events:\n            if fd == local_r:\n                #event on local process\n                if event & select.EPOLLIN:\n                    rd = os.read(local_r, 5000)\n\n                    if rd:\n                        #write data to remote side\n                        remoteGit.writeToChannel(rd)\n                    else:\n                        proc.writeError(\"Local socket write error\")\n                        client_process.kill()\n                        return 1\n                else:\n                    proc.writeError(\"Local socket error\")\n                    client_process.kill()\n                    return 1\n\n            else:\n                if event & select.EPOLLIN:\n                    #get all data from remote side\n                    data = remoteGit.getAll()\n                    remoteGit.stderrWrite()\n\n                    if not data:\n                        #remote side finished - close local pipes - local process wouldnt finish \n                        os.close(local_r) \n                        os.close(local_w)\n                        os.close(remote_r)\n                        os.close(remote_w)\n\n                        #get local process pid and kill his child to finish git-remote-fd\n                        gitPid = psutil.Process(client_process.pid).children()[0].pid\n                        os.kill(gitPid, signal.SIGTERM)\n\n                        return 0\n\n\n                    want = len(data)\n                    writed = 0\n                    offset = 0\n\n                    while(writed != want):\n                        #write data from remote side to local process\n                        wr = os.write(remote_w, data[offset:])\n\n                        if(wr < 0):\n                            proc.writeError(\"Local socket write error\")\n\n                            return 1\n\n                        writed += wr\n                        offset += wr\n\n                else:\n                    proc.writeError(\"Remote socket error\")\n                    client_process.kill()\n                    return -1"},{"id":"249133","messageId":"20140910144756.GA4267@LK-Perkele-VII","threadId":"37531","inReplyTo":"CAOP4-939uAP2Tgofz01F40_Eb0F8GxBsU49HxNvAiyBCW6H08w@mail.gmail.com","subject":"Re: git-remote-fd problem","fromName":"Ilari Liusvaara","fromEmail":"ilari.liusvaara@elisanet.fi","sentAt":"2014-09-10T14:47:57Z","receivedAt":"2014-09-10T14:47:57Z","isPatch":false,"sender":{"key":"ilari.liusvaara@elisanet.fi","avatar":null},"body":"On Wed, Sep 10, 2014 at 03:29:00PM +0200, Jiri Sevcik wrote:\n> Hi!\n> I have problem with using git-remote-fd function. I create two local\n> pipes for communication for locally running process git pull. Then I\n> start git-upload-pack on remote side (this process is started in\n> different part of my code  and it works correctly). Communication runs\n> successfully and when remote side finish, I want to close local pipes\n> to finish local process. I call close function to all of these pipes,\n> but unfortunatelly local process doesnt finish (in this moment\n> transfered data aresaved in /.git/object/pack folder in .pack files)\n> and hanging on. But if I kill child of this local process, downloaded\n> files are unpacked and files are created sucessfully. I think this way\n> is not so correct. Can someone help me and tell me what am I doing\n> wrong? Thank You. Code is in an attachement (its part of big system).\n\nThe remote-fd expects the transport to pass half-closes. So you can't\nclose all at once.\n\nLet there be pipes W and R and transport connection C.\n\n- W-read should be closed after being passed to remote-fd.\n- R-write should be closed after being passed to remote-fd.\n- Upon receiving \"no more data\" from C, close W-write.\n- Upon receiving EOF from R-read, close it and signal \"no more data\"\n  to C.\n\nIf you have server end, the same applies, but with remote-fd replaced\nby upload-pack/upload-archive/receive-pack.\n\n\n-Ilari\n"},{"id":"254134","messageId":"CAOP4-933YWoJ8QO0ZY2KG1cgtZ6jAR8oL4L9hw_0CcicDnepnQ@mail.gmail.com","threadId":"37531","inReplyTo":"CAOP4-92U+oFJJw38LWNmTqPtKMT=MDq0Ay9FiaGV5je77aUNpg@mail.gmail.com","subject":"Fwd: git-remote-fd problem","fromName":"Jiri Sevcik","fromEmail":"jsevcik14@gmail.com","sentAt":"2014-12-29T09:47:58Z","receivedAt":"2014-12-29T09:47:58Z","isPatch":false,"sender":{"key":"jsevcik14@gmail.com","avatar":null},"body":"> The remote-fd expects the transport to pass half-closes. So you can't\n> close all at once.\n>\n> Let there be pipes W and R and transport connection C.\n>\n> - W-read should be closed after being passed to remote-fd.\n> - R-write should be closed after being passed to remote-fd.\n> - Upon receiving \"no more data\" from C, close W-write.\n> - Upon receiving EOF from R-read, close it and signal \"no more data\"\n>   to C.\n\nHi, I followed your advices, correctly close pipes but git clone still\ndoesnt finish and hanging on.\nCode is in an attachement (its part of big system).\n\n\n\n    #create pipes\n    w_pipe = os.pipe()\n    r_pipe = os.pipe()\n\n    client_process = subprocess.Popen(\"/usr/bin/git clone fd::{0},{1} /tmp/gittest\".format(r_pipe[0], w_pipe[1]), shell=True)\n    #closing pipes\n    os.close(r_pipe[0]) \n    os.close(w_pipe[1])\n\n    epoll = select.epoll()\n    epoll.register(w_pipe[0], select.EPOLLIN)\n    epoll.register(proc.fd, select.EPOLLIN)\n\n    remoteGit = proc.runDaemon(\"git-upload-pack /tmp/testgit\")\n    \n    while True:\n        events = epoll.poll(1)\n\n        for fd, event in events:\n            if fd == w_pipe[0]:\n                if event & select.EPOLLIN:\n                    rd = os.read(w_pipe[0], 10000)\n                    if rd:\n                        #write data to remove git server\n                        remoteGit.writeToChannel(rd)\n                    else:\n                        proc.writeError(\"Local socket write error\")\n                        return 1\n                else:\n                    proc.writeError(\"Local socket error\")\n                    return 1\n\n            elif fd == proc.fd:\n                if event & select.EPOLLIN:\n                    #read data from remote git server\n                    data = remoteGit.getAll()\n                    remoteGit.stderrWrite()\n\n                    if not data:\n                        #remote server send EOF, close local pipe\n                        #but git clone is still running\n                        os.close(r_pipe[1])\n                        return 0\n                    \n                    want = len(data)\n\n                    writed = 0\n                    offset = 0\n                    \n                    while(writed != want):\n                        #write data from remote git server to local pipe\n                        wr = os.write(r_pipe[1], data[offset:])\n\n                        if(wr < 0):\n                            return 1\n\n                        writed += wr\n                        offset += wr\n\n                else:\n                    return -1              \n"},{"id":"254150","messageId":"20141229193651.GA16875@LK-Perkele-VII","threadId":"37531","inReplyTo":"CAOP4-933YWoJ8QO0ZY2KG1cgtZ6jAR8oL4L9hw_0CcicDnepnQ@mail.gmail.com","subject":"Re: Fwd: git-remote-fd problem","fromName":"Ilari Liusvaara","fromEmail":"ilari.liusvaara@elisanet.fi","sentAt":"2014-12-29T19:36:51Z","receivedAt":"2014-12-29T19:36:51Z","isPatch":false,"sender":{"key":"ilari.liusvaara@elisanet.fi","avatar":null},"body":"On Mon, Dec 29, 2014 at 10:47:58AM +0100, Jiri Sevcik wrote:\n> > The remote-fd expects the transport to pass half-closes. So you can't\n> > close all at once.\n> >\n> > Let there be pipes W and R and transport connection C.\n> >\n> > - W-read should be closed after being passed to remote-fd.\n> > - R-write should be closed after being passed to remote-fd.\n> > - Upon receiving \"no more data\" from C, close W-write.\n> > - Upon receiving EOF from R-read, close it and signal \"no more data\"\n> >   to C.\n> \n> Hi, I followed your advices, correctly close pipes but git clone still\n> doesnt finish and hanging on.\n> Code is in an attachement (its part of big system).\n\nFew ideas:\n- Check that git clone (and its subprocesses) don't inherit\nw_pipe[0] (/proc/<pid>/fd on Linux might be handy). If they do, that\nprevents this program from closing the pipe.\n\n- Setting environment variable GIT_TRANSLOOP_DEBUG to 1 might make\n  git spew lots of messages to stderr about reads, writes and closes.\n\n> \n>     #create pipes\n>     w_pipe = os.pipe()\n>     r_pipe = os.pipe()\n> \n>     client_process = subprocess.Popen(\"/usr/bin/git clone fd::{0},{1} /tmp/gittest\".format(r_pipe[0], w_pipe[1]), shell=True)\n>     #closing pipes\n>     os.close(r_pipe[0]) \n>     os.close(w_pipe[1])\n> \n>     epoll = select.epoll()\n>     epoll.register(w_pipe[0], select.EPOLLIN)\n>     epoll.register(proc.fd, select.EPOLLIN)\n> \n>     remoteGit = proc.runDaemon(\"git-upload-pack /tmp/testgit\")\n>     \n>     while True:\n>         events = epoll.poll(1)\n> \n>         for fd, event in events:\n>             if fd == w_pipe[0]:\n>                 if event & select.EPOLLIN:\n>                     rd = os.read(w_pipe[0], 10000)\n>                     if rd:\n>                         #write data to remove git server\n>                         remoteGit.writeToChannel(rd)\n>                     else:\n>                         proc.writeError(\"Local socket write error\")\n>                         return 1\n>                 else:\n>                     proc.writeError(\"Local socket error\")\n>                     return 1\n> \n>             elif fd == proc.fd:\n>                 if event & select.EPOLLIN:\n>                     #read data from remote git server\n>                     data = remoteGit.getAll()\n>                     remoteGit.stderrWrite()\n> \n>                     if not data:\n>                         #remote server send EOF, close local pipe\n>                         #but git clone is still running\n>                         os.close(r_pipe[1])\n>                         return 0\n>                     \n>                     want = len(data)\n> \n>                     writed = 0\n>                     offset = 0\n>                     \n>                     while(writed != want):\n>                         #write data from remote git server to local pipe\n>                         wr = os.write(r_pipe[1], data[offset:])\n> \n>                         if(wr < 0):\n>                             return 1\n> \n>                         writed += wr\n>                         offset += wr\n> \n>                 else:\n>                     return -1              \n\n-Ilari\n"}]}