Move some files, with all history, from one project into a new one

I would like to move some files, from the project in which they have always resided into a new project. I would like to keep all their history. I don't want to waste space by also moving the rest of the old project's history, or historical file contents.

We have a long-standing project, main-system. A group of files within it are designated as demo-system (sometimes whole folders, sometimes files within folders with other files). All development is done on the master branch. It has now transpired that the demo-system files are necessary, but no longer want to be within main-system but in a new project of their own. So I would like to move them from the main-system repo into a new repo of their own. I do want all their history but I don't want to take any contents from other files from main-system, which is quite big. Please can someone advise if there is a particular method I should best use for this. Do I create a new branch, then delete the rest of main-system leaving only what I want? Surely if I do that then I would end up with all the main-system file data within the new repo, which would expand its size. Is there a recommended way to extract certain files with their histories?

Regards, Richard.

On Wed, Apr 15, 2020 at 08:31:35AM +0000, Kerry, Richard wrote:

Try git-filter-branch's --subdirectory repository, which is designed to do exactly this.

Or the much newer (and faster) git-filter-repo:

  https://github.com/newren/git-filter-repo

Correct; just deleting files and creating a new commit will still carry them in the history. You have to rewrite the old commits.

-Peff

On Wed, Apr 15, 2020 at 11:11:28AM -0400, Jeff King wrote:

For what it's worth, Elijah has provided some excellent documentation on how to use git-filter-repo to do exactly this here:

  https://github.com/newren/git-filter-repo#solving-this-with-filter-repo

Thanks, Taylor

On Wed, Apr 15, 2020 at 8:49 AM Taylor Blau <me@ttaylorr.com> wrote:

That particular example might be for a different case than what Richard requested, though. Let's say the original repo had a file structure like the following:

   module/
      foo.c
      bar.c
   otherDir/
      blah.config
      stuff.txt
   zebra.jpg

If the request is to e.g. take module/ and all files within it with
their history and make a new repository out of it, with module/ being
remapped to the root of the repository, then you would want:
   git filtrer-repo --subdirectory-filter module
and yes, this looks exactly like filter-branch; that's the one flag I
copied from it.  So this one usecase maps directly between the two
tools.

In contrast, if you wanted to keep all files from the original repo but move everything into a subdirectory named "myProject" (so that e.g. module/foo.c became myProject/module/foo.c), possibly in preparation for merging your repo into some larger monorepo, then you'd want to pass `--to-subdirectory-filter myProject` as in the link you pointed out. filter-branch doesn't have an equivalent.

Hope that helps, Elijah

Thanks All. I've cloned git-filter-repo and I shall shortly give it a try.

First step is to get Python 3 installed (I'm on Windows).

Regards, Richard.

-----Original Message-----
From: Elijah Newren <newren@gmail.com> 
Sent: 16 April 2020 07:06
To: Taylor Blau <me@ttaylorr.com>
Cc: Jeff King <peff@peff.net>; Kerry, Richard <richard.kerry@atos.net>; git@vger.kernel.org
Subject: Re: Move some files, with all history, from one project into a new one

Caution! External email. Do not open attachments or click links, unless this email comes from a known sender and you know the content is safe.

On Wed, Apr 15, 2020 at 8:49 AM Taylor Blau <me@ttaylorr.com> wrote:

That particular example might be for a different case than what Richard requested, though.  Let's say the original repo had a file structure like the following:

   module/
      foo.c
      bar.c
   otherDir/
      blah.config
      stuff.txt
   zebra.jpg

If the request is to e.g. take module/ and all files within it with their history and make a new repository out of it, with module/ being remapped to the root of the repository, then you would want:
   git filtrer-repo --subdirectory-filter module and yes, this looks exactly like filter-branch; that's the one flag I copied from it.  So this one usecase maps directly between the two tools.

In contrast, if you wanted to keep all files from the original repo but move everything into a subdirectory named "myProject" (so that e.g. module/foo.c became myProject/module/foo.c), possibly in preparation for merging your repo into some larger monorepo, then you'd want to pass `--to-subdirectory-filter myProject` as in the link you pointed out.  filter-branch doesn't have an equivalent.

Hope that helps, Elijah