git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [RFC 1/4 v2] Implement a basic remote helper for svn in C.

From
Florian Achleitner <florian.achleitner.2.6.31@gmail.com>
Date
Aug 12, 2012, 10:06 UTC
Message-ID
<1636924.tANzCnKezB@flobuntu>
In-Reply-To
<20120801194247.GE24357@copier>
Hi,
back to the pipe-topic.
On Wednesday 01 August 2012 12:42:48 Jonathan Nieder wrote:
Show 22 quoted lines
> Hi again,
> 
> Florian Achleitner wrote:
> > When the first line arrives at the remote-helper, it starts importing one
> > line at a time, leaving the remaining lines in the pipe.
> > For importing it requires the data from fast-import, which would be mixed
> > with import lines or queued at the end of them.
> 
> Oh, good catch.
> 
> The way it's supposed to work is that in a bidi-import, the remote
> helper reads in the entire list of refs to be imported and only once
> the newline indicating that that list is over arrives starts writing
> its fast-import stream.  We could make this more obvious by not
> spawning fast-import until immediately before writing that newline.
> 
> This needs to be clearly documented in the git-remote-helpers(1) page
> if the bidi-import command is introduced.
> 
> If a remote helper writes commands for fast-import before that newline
> comes, that is a bug in the remote helper, plain and simple.  It might
> be fun to diagnose this problem:

This would require all existing remote helpers that use 'import' to be ported to the new concept, right? Probably there is no other..

Show 29 quoted lines
> 
> 	static void pipe_drained_or_die(int fd, const char *msg)
> 	{
> 		char buf[1];
> 		int flags = fcntl(fd, F_GETFL);
> 		if (flags < 0)
> 			die_errno("cannot get pipe flags");
> 		if (fcntl(fd, F_SETFL, flags | O_NONBLOCK))
> 			die_errno("cannot set up non-blocking pipe read");
> 		if (read(fd, buf, 1) > 0)
> 			die("%s", msg);
> 		if (fcntl(fd, F_SETFL, flags))
> 			die_errno("cannot restore pipe flags");
> 	}
> 	...
> 
> 	for (i = 0; i < nr_heads; i++) {
> 		write "import %s\n", to_fetch[i]->name;
> 	}
> 
> 	if (getenv("GIT_REMOTE_HELPERS_SLOW_SANITY_CHECK"))
> 		sleep(1);
> 
> 	pipe_drained_or_die("unexpected output from remote helper before
> fast-import launch");
> 
> 	if (get_importer(transport, &fastimport))
> 		die("couldn't run fast-import");
> 	write_constant(data->helper->in, "\n");

I still don't believe that sharing the input pipe of the remote helper is worth the hazzle. It still requires an additional pipe to be setup, the one from fast-import to the remote-helper, sharing one FD at the remote helper. It still requires more than just stdin, stdout, stderr.

I would suggest to use a fifo. It can be openend independently, after forking 
and on windows they have named pipes with similar semantics, so I think this 
could be easily ported. 
I would suggest the following changes:
- add a capability to the remote helper 'bidi-import', or 'bidi-pipe'. This 
signals that the remote helper requires data from fast-import.
- add a command 'bidi-import', or 'bidi-pipe' that is tells the remote helper 
which filename the fifo is at, so that it can open it and read it when it 
handles 'import' commands.
- transport-helper.c creates the fifo on demand, i.e. on seeing the capability, 
in the gitdir or in /tmp.
- fast-import gets the name of the fifo as a command-line argument. The 
alternative would be to add a command, but that's not allowed, because it 
changes the stream semantics.
Another alternative would be to use the existing --cat-pipe-fd argument. But 
that requires to open the fifo before execing fast-import and makes us 
dependent on the posix model of forking and inheriting file descriptors, while 
opening a fifo in fast-import would not.
Previous: Jonathan NiederNext: Jonathan Nieder
Message 35 of 57 in “[RFC 0/4]”
  1. Florian AchleitnerJun 4, 2012
  2. 1/4 Implement a basic remote helper vor svn in C.Florian Achleitner, Jun 4, 2012
  3. 2/4 Integrate remote-svn into svn-fe/Makefile.Florian Achleitner, Jun 4, 2012
  4. 3/4 Add svndump_init_fd to allow reading dumps from arbitrary FDs.Florian Achleitner, Jun 4, 2012
  5. 4/4 Add cat-blob report pipe from fast-import to remote-helper.Florian Achleitner, Jun 4, 2012
  6. David Michael BarrJun 5, 2012
  7. Jeff KingJun 5, 2012
  8. David Michael BarrJun 5, 2012
  9. Jeff KingJun 5, 2012
  10. Florian AchleitnerJun 5, 2012
  11. Jeff KingJun 6, 2012
  12. Florian AchleitnerJun 6, 2012
  13. Johannes SixtJun 5, 2012
  14. Jeff KingJun 5, 2012
  15. Florian AchleitnerJun 5, 2012
  16. Johannes SixtJun 5, 2012
  17. David Michael BarrJun 5, 2012
  18. 0/4 Florian Achleitner, Jun 29, 2012
  19. 1/4 Implement a basic remote helper for svn in C.Florian Achleitner, Jun 29, 2012
  20. Jonathan NiederJul 2, 2012
  21. Jonathan NiederJul 6, 2012
  22. Florian AchleitnerJul 6, 2012
  23. Florian AchleitnerJul 26, 2012
  24. Jonathan NiederJul 26, 2012
  25. Florian AchleitnerJul 26, 2012
  26. Jonathan NiederJul 28, 2012
  27. Florian AchleitnerJul 30, 2012
  28. Jonathan NiederJul 30, 2012
  29. Florian AchleitnerJul 30, 2012
  30. Jonathan NiederJul 30, 2012
  31. Florian AchleitnerJul 31, 2012
  32. Jonathan NiederJul 31, 2012
  33. Florian AchleitnerAug 1, 2012
  34. Jonathan NiederAug 1, 2012
  35. Florian AchleitnerAug 12, 2012
  36. Jonathan NiederAug 12, 2012
  37. Florian AchleitnerAug 12, 2012
  38. Jonathan NiederAug 12, 2012
  39. Jonathan NiederAug 12, 2012
  40. Junio C HamanoJul 26, 2012
  41. Florian AchleitnerJul 30, 2012
  42. Steven MichalskeJul 26, 2012
  43. 2/4 Integrate remote-svn into svn-fe/Makefile.Florian Achleitner, Jun 29, 2012
  44. 3/4 Add svndump_init_fd to allow reading dumps from arbitrary FDs.Florian Achleitner, Jun 29, 2012
  45. 4/4 Add cat-blob report fifo from fast-import to remote-helper.Florian Achleitner, Jun 29, 2012
  46. Florian AchleitnerJul 21, 2012
  47. Jonathan NiederJul 21, 2012
  48. Florian AchleitnerJul 21, 2012
  49. Jonathan NiederJul 21, 2012
  50. Florian AchleitnerJul 22, 2012
  51. Jonathan NiederJul 22, 2012
  52. Jonathan NiederJul 21, 2012
  53. Jonathan NiederJul 26, 2012
  54. Florian AchleitnerJul 26, 2012
  55. Jonathan NiederJul 26, 2012
  56. Florian AchleitnerJul 27, 2012
  57. Jonathan NiederJul 28, 2012

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.