threads / discuss / 25856

fast-import: multiple processes possible?

Subject: fast-import: multiple processes possible?

## tl;dr

2 messages between Nov 27, 2010 and Nov 27, 2010.

replies: 1people: 2as markdown or json

Ian Zimmerman· Nov 27, 2010, 22:07 UTC · lore

Hi, I have a user question, I hope that's ok. Actually, the end goal is development of a fast-import frontend. The fast-import manpage says that it is safe to run "any other" git operation concurrently with fast-import. It doesn't say if it is possible and safe to run another copy of fast-import itself.

The reason I'd like to do this is speedup. With sequential approach my frontend can only load each core on my system about 50%. If I could run multiple imports concurrently I could make it close to 100%, especially when feeding it blobs (which take most of the time).

Thanks for the advice.

-- 
Ian Zimmerman <itz@buug.org>
gpg public key: 1024D/C6FF61AD 
fingerprint: 66DC D68F 5C1B 4D71 2EE5  BD03 8A00 786C C6FF 61AD
Ham is for reading, not for eating.

Shawn Pearce· Nov 27, 2010, 22:28 UTC · re: Ian Zimmerman · lore

Re: fast-import: multiple processes possible?

On Sat, Nov 27, 2010 at 2:07 PM, Ian Zimmerman <itz@buug.org> wrote:

Show 6 quoted lines

>
> Hi, I have a user question, I hope that's ok.  Actually, the end goal is
> development of a fast-import frontend.  The fast-import manpage says
> that it is safe to run "any other" git operation concurrently with
> fast-import.  It doesn't say if it is possible and safe to run another
> copy of fast-import itself.

Yes, you can.

The only problem is, each fast-import cannot see the objects created by the other concurrent fast-imports until they either terminate, or execute "checkpoint" command received from the frontend.

-- 
Shawn.