threads / discuss / 35304

Selectively commit/publish files to GIT

Subject: Selectively commit/publish files to GIT

## tl;dr

5 messages between Nov 8, 2013 and Nov 8, 2013.

replies: 4people: 3as markdown or json

Thomas Manson· Nov 8, 2013, 10:52 UTC · lore

Hi,

  I've converting my Bazaar repository to GIT.

 I've successfully done this conversion and I now want to publish my
source code to github.

  The problem is that in Bazaar, I've commited some big files (63MB &
173MB), but this files are no longer in my project, only in the
revisions files of Bazaar and now Git.

  I don't need this files to be pushed on Github.

  How can I search git history for big files and remove them, or mark
them to be not published ?

I've tryed this solution found on the link in the error:

git filter-branch --force --index-filter 'git rm --cached --ignore-unmatch giant_file' --prune-empty --tag-name-filter cat -- --all git commit --amend -CHEAD

Then I've tryed to Commit & Push from Github mac application and I had several network error, and finally get the same error on giant files (maybe my multiple commit & push did override something, but I understood that the git rm command would remove things once for good...

can anybody help me ? I'm blocked in my dev because of this, I can't share my project with a friend. I'm publishing here : https://github.com/dev-mansonthomas/crf-rdp.git (paying for storage is an option as I'm quite fed up loosing time for filesize...)

Regards, Thomas.

here is the error I have using the GitHub application on Mac : (after that I intend to use Eclipse)

File Ressources/dwr/dwr-3.0.0.110.dev-src.zip is 67.69 MB; this is
larger than GitHub's recommended maximum file size of 50 MB
GH001: Large files detected.
Trace: 8f0259b29260f0c4d7ae4d4ae70e0306
See http://git.io/iEPt8g for more information.
File .bzr/repository/packs/a7bcd6ba235114ab88c80fe8a97adcfa.pack is
178.76 MB; this exceeds GitHub's file size limit of 100 MB

Magnus Bäck· Nov 8, 2013, 14:13 UTC · re: Thomas Manson · lore

Re: Selectively commit/publish files to GIT

On Friday, November 08, 2013 at 05:52 EST,
     Thomas Manson <dev.mansonthomas@gmail.com> wrote:

[...]

Show 9 quoted lines

>   How can I search git history for big files and remove them, or mark
> them to be not published ?
> 
> I've tryed this solution found on the link in the error:
> 
> git filter-branch --force --index-filter   'git rm --cached
> --ignore-unmatch giant_file'   --prune-empty --tag-name-filter cat --
> --all
> git commit --amend -CHEAD

[...]

Show 10 quoted lines

> here is the error I have using the GitHub application on Mac :
> (after that I intend to use Eclipse)
> 
> File Ressources/dwr/dwr-3.0.0.110.dev-src.zip is 67.69 MB; this is
> larger than GitHub's recommended maximum file size of 50 MB
> GH001: Large files detected.
> Trace: 8f0259b29260f0c4d7ae4d4ae70e0306
> See http://git.io/iEPt8g for more information.
> File .bzr/repository/packs/a7bcd6ba235114ab88c80fe8a97adcfa.pack is
> 178.76 MB; this exceeds GitHub's file size limit of 100 MB

Did you actually replace the 'giant_file' placeholder with the path to *your* giant file? The error message indicates that you didn't, or that something else went wrong during the filtering.

You can use 'git log' to look for commits touching particular pathnames, making it easy to verify that a file has been eradicated.

git log --all -- path/to/big-honking-file.zip

-- 
Magnus Bäck
baeck@google.com

Thomas Manson· Nov 8, 2013, 22:51 UTC · re: Magnus Bäck · lore

Re: Selectively commit/publish files to GIT

Hi Magnus,

 thanks for pointing that out. I must have overlooked the webpage,
when I saw that I thought it was some kind of alias that designated
files greater than a specific files (server side configurated as the
100MB limit)

  I'll try again ;)

Thomas.

On Fri, Nov 8, 2013 at 3:13 PM, Magnus Bäck <baeck@google.com> wrote:

Show 40 quoted lines

> On Friday, November 08, 2013 at 05:52 EST,
>      Thomas Manson <dev.mansonthomas@gmail.com> wrote:
>
> [...]
>
>>   How can I search git history for big files and remove them, or mark
>> them to be not published ?
>>
>> I've tryed this solution found on the link in the error:
>>
>> git filter-branch --force --index-filter   'git rm --cached
>> --ignore-unmatch giant_file'   --prune-empty --tag-name-filter cat --
>> --all
>> git commit --amend -CHEAD
>
> [...]
>
>> here is the error I have using the GitHub application on Mac :
>> (after that I intend to use Eclipse)
>>
>> File Ressources/dwr/dwr-3.0.0.110.dev-src.zip is 67.69 MB; this is
>> larger than GitHub's recommended maximum file size of 50 MB
>> GH001: Large files detected.
>> Trace: 8f0259b29260f0c4d7ae4d4ae70e0306
>> See http://git.io/iEPt8g for more information.
>> File .bzr/repository/packs/a7bcd6ba235114ab88c80fe8a97adcfa.pack is
>> 178.76 MB; this exceeds GitHub's file size limit of 100 MB
>
> Did you actually replace the 'giant_file' placeholder with the path to
> *your* giant file? The error message indicates that you didn't, or that
> something else went wrong during the filtering.
>
> You can use 'git log' to look for commits touching particular pathnames,
> making it easy to verify that a file has been eradicated.
>
> git log --all -- path/to/big-honking-file.zip
>
> --
> Magnus Bäck
> baeck@google.com

Thomas Manson· Nov 8, 2013, 23:01 UTC · re: Thomas Manson · lore

Re: Selectively commit/publish files to GIT

Thanks Magnus, it did solve my issue.

I've noticed by the way that i've accidentally commited the whole .bzr directory (that hold all my commit in Bazaar style).

I've deleted this folder as well using -r

git filter-branch --force --index-filter 'git rm -r --cached --ignore-unmatch .bzr' --prune-empty --tag-name-filter cat -- --all

for reference, an example of working run :

Thomass-MacBook-Pro:crf-rdp tmanson$ git filter-branch --force --index-filter 'git rm --cached --ignore-unmatch Ressources/dwr/dwr-3.0.0.110.dev-src.zip' --prune-empty --tag-name-filter cat -- --all Rewrite 58952de4e10bc1df2f8c324e94d249a0366f7512 (65/158)rm 'Ressources/dwr/dwr-3.0.0.110.dev-src.zip' Rewrite e661d08f1c33d7bd53e2c2231a3296b2ade5a2dd (66/158)rm 'Ressources/dwr/dwr-3.0.0.110.dev-src.zip' Rewrite 8ff3f3f9ffb901b7cdc1dba220a121bab0698121 (67/158)rm 'Ressources/dwr/dwr-3.0.0.110.dev-src.zip' Rewrite cfec6341565c35c410f248e5861543ac01b84f40 (158/158) Ref 'refs/heads/master' was rewritten Ref 'refs/tags/version-0-1_20081120' was rewritten Ref 'refs/tags/version-0-5-1_20120512' was rewritten

On Fri, Nov 8, 2013 at 11:51 PM, Thomas Manson <dev.mansonthomas@gmail.com> wrote:

Show 52 quoted lines

> Hi Magnus,
>
>  thanks for pointing that out. I must have overlooked the webpage,
> when I saw that I thought it was some kind of alias that designated
> files greater than a specific files (server side configurated as the
> 100MB limit)
>
>   I'll try again ;)
>
> Thomas.
>
> On Fri, Nov 8, 2013 at 3:13 PM, Magnus Bäck <baeck@google.com> wrote:
>> On Friday, November 08, 2013 at 05:52 EST,
>>      Thomas Manson <dev.mansonthomas@gmail.com> wrote:
>>
>> [...]
>>
>>>   How can I search git history for big files and remove them, or mark
>>> them to be not published ?
>>>
>>> I've tryed this solution found on the link in the error:
>>>
>>> git filter-branch --force --index-filter   'git rm --cached
>>> --ignore-unmatch giant_file'   --prune-empty --tag-name-filter cat --
>>> --all
>>> git commit --amend -CHEAD
>>
>> [...]
>>
>>> here is the error I have using the GitHub application on Mac :
>>> (after that I intend to use Eclipse)
>>>
>>> File Ressources/dwr/dwr-3.0.0.110.dev-src.zip is 67.69 MB; this is
>>> larger than GitHub's recommended maximum file size of 50 MB
>>> GH001: Large files detected.
>>> Trace: 8f0259b29260f0c4d7ae4d4ae70e0306
>>> See http://git.io/iEPt8g for more information.
>>> File .bzr/repository/packs/a7bcd6ba235114ab88c80fe8a97adcfa.pack is
>>> 178.76 MB; this exceeds GitHub's file size limit of 100 MB
>>
>> Did you actually replace the 'giant_file' placeholder with the path to
>> *your* giant file? The error message indicates that you didn't, or that
>> something else went wrong during the filtering.
>>
>> You can use 'git log' to look for commits touching particular pathnames,
>> making it easy to verify that a file has been eradicated.
>>
>> git log --all -- path/to/big-honking-file.zip
>>
>> --
>> Magnus Bäck
>> baeck@google.com

Keshav Kini· Nov 8, 2013, 23:50 UTC · re: Thomas Manson · lore

Re: Selectively commit/publish files to GIT

Thomas Manson <dev.mansonthomas@gmail.com> writes:

Show 8 quoted lines

>   The problem is that in Bazaar, I've commited some big files (63MB &
> 173MB), but this files are no longer in my project, only in the
> revisions files of Bazaar and now Git.
>
>   I don't need this files to be pushed on Github.
>
>   How can I search git history for big files and remove them, or mark
> them to be not published ?

I see you already solved your problem, but for future reference, there is a nice tool that is more limited in functionality than git filter-branch but also much faster and possibly easier to use:

    http://rtyley.github.io/bfg-repo-cleaner/

-Keshav