{"thread":{"id":"51623","subject":"How to determine when to stop receiving pack content","startedAt":"2019-08-10T23:55:35Z","lastAt":"2019-08-12T00:22:50Z","messageCount":5,"participants":["Farhan Khan","Pratyush Yadav","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"380278","messageId":"c1754835efe3aa8a5ac93ee2db4a99c5@farhan.codes","threadId":"51623","inReplyTo":null,"subject":"How to determine when to stop receiving pack content","fromName":"Farhan Khan","fromEmail":"farhan@farhan.codes","sentAt":"2019-08-10T23:47:22Z","receivedAt":"2019-08-10T23:55:35Z","isPatch":false,"sender":{"key":"farhan@farhan.codes","avatar":"https://gravatar.com/avatar/64c1dc837f48efeff1b51db4131f6ebe84b1d0c1caa43f73d8d45f2dee8e67ce?d=mp&s=160"},"body":"Hi,\n\nI am trying to write an implementation of git clone over ssh and am a little confused how to determine a server response has ended. Specifically, after a client sends its requested 'want', the server sends the pack content over. However, how does the client know to stop reading data? If I run a simple read() of the file descriptor:\n\nA. If I use reading blocking, the client will wait until new data is available, potentially forever.\nB. If I use non-blocking, the client might terminate reading for new data, when in reality new data is in transit.\n\nI do not see a mechanism to specify the size or to indicate the end of the pack content. Am I missing something?\n\nThanks\n---\nFarhan Khan\nPGP Fingerprint: 1312 89CE 663E 1EB2 179C  1C83 C41D 2281 F8DA C0DE\n"},{"id":"380284","messageId":"20190811150450.5pvgvwcqrrvzggbo@localhost.localdomain","threadId":"51623","inReplyTo":"c1754835efe3aa8a5ac93ee2db4a99c5@farhan.codes","subject":"Re: How to determine when to stop receiving pack content","fromName":"Pratyush Yadav","fromEmail":"me@yadavpratyush.com","sentAt":"2019-08-11T15:04:50Z","receivedAt":"2019-08-11T15:04:56Z","isPatch":false,"sender":{"key":"me@yadavpratyush.com","avatar":"https://avatars.githubusercontent.com/u/8817931?v=4"},"body":"On 10/08/19 11:47PM, Farhan Khan wrote:\n> Hi,\n> \n> I am trying to write an implementation of git clone over ssh and am a little confused how to determine a server response has ended. Specifically, after a client sends its requested 'want', the server sends the pack content over. However, how does the client know to stop reading data? If I run a simple read() of the file descriptor:\n>\n> A. If I use reading blocking, the client will wait until new data is available, potentially forever.\n> B. If I use non-blocking, the client might terminate reading for new data, when in reality new data is in transit.\n> \n> I do not see a mechanism to specify the size or to indicate the end of the pack content. Am I missing something?\n\nWell, I am not very familiar with git-clone internals, but I did some \ndigging around, and I think I know what answer to your problem is.\n\nLooking at Documentation/technical/protocol-v2.txt:34, the flush packet \nindicates the end of a message. Looking in the output section of the \nfetch command (protocol-v2.txt:342), it sends you some optional \nsections, and then the packfile and then sends a flush packet.\n\nSo your read should stop reading data when it sees the flush packet.\n\nAnother way would be to look at the packfile contents. Looking at \nDocumentation/technical/pack-format.txt, the packfile contains the \nnumber of objects in the packfile, and each object entry has the object \nsize. So you can stop reading after you have received the last object in \nthe packfile (plus the 20-byte trailer).\n\nI don't know which is the better way, but the former seems like a better \nchoice to me.\n\n-- \nRegards,\nPratyush Yadav\n"},{"id":"380290","messageId":"xmqq7e7jcop1.fsf@gitster-ct.c.googlers.com","threadId":"51623","inReplyTo":"c1754835efe3aa8a5ac93ee2db4a99c5@farhan.codes","subject":"Re: How to determine when to stop receiving pack content","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-08-11T22:38:50Z","receivedAt":"2019-08-11T22:39:03Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Farhan Khan\" <farhan@farhan.codes> writes:\n\n> I am trying to write an implementation of git clone over ssh and\n> am a little confused how to determine a server response has\n> ended. Specifically, after a client sends its requested 'want',\n> the server sends the pack content over. However, how does the\n> client know to stop reading data? If I run a simple read() of the\n> file descriptor:\n>\n> A. If I use reading blocking, the client will wait until new data is available, potentially forever.\n> B. If I use non-blocking, the client might terminate reading for new data, when in reality new data is in transit.\n\nIt's TCP stream, so blocking read will tell you when the the other\nside finishes talking to you and disconnects.  Your read() will\nsignal EOF.  If you are paranoid and want to protect your reader\nagainst malicious writer, then you cannot trust anything the other\nside says (including possibly any \"I have N megabyte of data\" kind\nof length information), so you'd need to set up a timeout to get\nyourself out of a stuck read, but that is neither a news nor a\nrocket surgery ;-)\n\nThe \"upload-pack\" (the component that talks with your \"fetch\" and\n\"clone\"), after negotiating what objects to include in the data\ntransfer with the program on your side, produces a pack data stream,\nand is allowed to send additional \"garbage\" after that.\n\nThe receiving end, after finishing the negotiation, reads the pack\ndata stream (there is only one packfile contents in it) and parses\nit according to the packfile format so that it can find the end\n(cf. Documentation/technical/pack-format.txt).\n\nAfter seeing the end of the pack stream, anything that follows is\n\"garbage\" and is generally passed through to the standard output.\n\nThere are two codepaths on the receiving end (\"unpack-objects\" and\n\"index-pack --stdin\").  Most likely an initial \"clone\" would end up\nfollowing the latter, but for educational purposes, the unpack-objects\nmay be easier to follow.  These two codepaths are morally equivalent\nat the higher conceptual levels.\n"},{"id":"380291","messageId":"53515e0fb56928114a63bfe87e24c743@farhan.codes","threadId":"51623","inReplyTo":"20190811150450.5pvgvwcqrrvzggbo@localhost.localdomain","subject":"Re: How to determine when to stop receiving pack content","fromName":"Farhan Khan","fromEmail":"farhan@farhan.codes","sentAt":"2019-08-11T23:31:06Z","receivedAt":"2019-08-11T23:31:10Z","isPatch":false,"sender":{"key":"farhan@farhan.codes","avatar":"https://gravatar.com/avatar/64c1dc837f48efeff1b51db4131f6ebe84b1d0c1caa43f73d8d45f2dee8e67ce?d=mp&s=160"},"body":"August 11, 2019 11:04 AM, \"Pratyush Yadav\" <me@yadavpratyush.com> wrote:\n\n> On 10/08/19 11:47PM, Farhan Khan wrote:\n> \n>> Hi,\n>> \n>> I am trying to write an implementation of git clone over ssh and am a little confused how to\n>> determine a server response has ended. Specifically, after a client sends its requested 'want', the\n>> server sends the pack content over. However, how does the client know to stop reading data? If I\n>> run a simple read() of the file descriptor:\n>> \n>> A. If I use reading blocking, the client will wait until new data is available, potentially\n>> forever.\n>> B. If I use non-blocking, the client might terminate reading for new data, when in reality new data\n>> is in transit.\n>> \n>> I do not see a mechanism to specify the size or to indicate the end of the pack content. Am I\n>> missing something?\n> \n> Well, I am not very familiar with git-clone internals, but I did some\n> digging around, and I think I know what answer to your problem is.\n> \n> Looking at Documentation/technical/protocol-v2.txt:34, the flush packet\n> indicates the end of a message. Looking in the output section of the\n> fetch command (protocol-v2.txt:342), it sends you some optional\n> sections, and then the packfile and then sends a flush packet.\n> \n> So your read should stop reading data when it sees the flush packet.\n> \n> Another way would be to look at the packfile contents. Looking at\n> Documentation/technical/pack-format.txt, the packfile contains the\n> number of objects in the packfile, and each object entry has the object\n> size. So you can stop reading after you have received the last object in\n> the packfile (plus the 20-byte trailer).\n> \n> I don't know which is the better way, but the former seems like a better\n> choice to me.\n> \n> --\n> Regards,\n> Pratyush Yadav\n\nHi Pratyush,\n\nThanks for your reply!\n\nUnless I am mistaken, a pack file does not end in a flush_pkt. I ran some tests and did not see the stream end in \"0000\". Is there is a mistake somewhere on my end?\n\n---\nFarhan Khan\nPGP Fingerprint: 1312 89CE 663E 1EB2 179C  1C83 C41D 2281 F8DA C0DE\n"},{"id":"380292","messageId":"20190812002244.lvsokkxcpfshjiyq@localhost.localdomain","threadId":"51623","inReplyTo":"53515e0fb56928114a63bfe87e24c743@farhan.codes","subject":"Re: How to determine when to stop receiving pack content","fromName":"Pratyush Yadav","fromEmail":"me@yadavpratyush.com","sentAt":"2019-08-12T00:22:44Z","receivedAt":"2019-08-12T00:22:50Z","isPatch":false,"sender":{"key":"me@yadavpratyush.com","avatar":"https://avatars.githubusercontent.com/u/8817931?v=4"},"body":"On 11/08/19 11:31PM, Farhan Khan wrote:\n> August 11, 2019 11:04 AM, \"Pratyush Yadav\" <me@yadavpratyush.com> wrote:\n> \n> > On 10/08/19 11:47PM, Farhan Khan wrote:\n> > \n> >> Hi,\n> >> \n> >> I am trying to write an implementation of git clone over ssh and am a little confused how to\n> >> determine a server response has ended. Specifically, after a client sends its requested 'want', the\n> >> server sends the pack content over. However, how does the client know to stop reading data? If I\n> >> run a simple read() of the file descriptor:\n> >> \n> >> A. If I use reading blocking, the client will wait until new data is available, potentially\n> >> forever.\n> >> B. If I use non-blocking, the client might terminate reading for new data, when in reality new data\n> >> is in transit.\n> >> \n> >> I do not see a mechanism to specify the size or to indicate the end of the pack content. Am I\n> >> missing something?\n> > \n> > Well, I am not very familiar with git-clone internals, but I did some\n> > digging around, and I think I know what answer to your problem is.\n> > \n> > Looking at Documentation/technical/protocol-v2.txt:34, the flush packet\n> > indicates the end of a message. Looking in the output section of the\n> > fetch command (protocol-v2.txt:342), it sends you some optional\n> > sections, and then the packfile and then sends a flush packet.\n> > \n> > So your read should stop reading data when it sees the flush packet.\n> > \n> > Another way would be to look at the packfile contents. Looking at\n> > Documentation/technical/pack-format.txt, the packfile contains the\n> > number of objects in the packfile, and each object entry has the object\n> > size. So you can stop reading after you have received the last object in\n> > the packfile (plus the 20-byte trailer).\n> > \n> > I don't know which is the better way, but the former seems like a better\n> > choice to me.\n> > \n> > --\n> > Regards,\n> > Pratyush Yadav\n> \n> Hi Pratyush,\n> \n> Thanks for your reply!\n> \n> Unless I am mistaken, a pack file does not end in a flush_pkt. I ran some tests and did not see the stream end in \"0000\". Is there is a mistake somewhere on my end?\n \nHm, turns out I was on the pu branch, not master when I looked at \nprotocol-v2.txt. The file was updated about 3 days ago (not in master \nyet) (7ee4ab7e8c3310fc28d9dd7d47da26e497e73556), where it seems to imply \nthat flush-pkt will be sent after the packfile (see excerpt below).\n\n--- protocol-v2.txt ---\n    output = acknowledgements flush-pkt |\n\t     [acknowledgments delim-pkt] [shallow-info delim-pkt]\n\t     [wanted-refs delim-pkt] [packfile-uris delim-pkt]\n\t     packfile flush-pkt\n---\n\nSo either something changed in the protocol with that merge, or there is \na discrepancy in the documentation, because the above output format \nseems to imply the packfile will be followed by a flush packet. I \nhaven't looked at the full contents of the merge, but none of the commit \nmessages mention changing this behaviour.\n\nEither way, you can probably parse the packfile to know how many objects \nyou will get, and stop after the last object. Or like Junio said, just \nwait for an EOF.\n\nSorry for the wrong information.\n\n-- \nRegards,\nPratyush Yadav\n"}]}