{"thread":{"id":"50308","subject":"Removing data from repository","startedAt":"2019-01-24T15:09:53Z","lastAt":"2019-01-24T16:01:56Z","messageCount":5,"participants":["Naum Derzhi","Jeff King","Paul Smith"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"367556","messageId":"7A854577E980BE4FB29FDD041B9B75E701798CEE@NP1EXMB203.corp.halliburton.com","threadId":"50308","inReplyTo":null,"subject":"Removing data from repository","fromName":"Naum Derzhi","fromEmail":"naum.derzhi@halliburton.com","sentAt":"2019-01-24T14:51:50Z","receivedAt":"2019-01-24T15:09:53Z","isPatch":false,"sender":{"key":"naum.derzhi@halliburton.com","avatar":null},"body":"Greetings,\n\nI have this problem: years ago one of our developers committed a large (Gigabytes) piece of binary data into our project repository. This should not have been done, but it happened. (Honest, it was not me). We never needed this data in the repository.\n\nUsing git rm removes these files from the working tree, but they are still somewhere in the repository, so when we clone, we transfer gigabytes of unneeded data. \n\nIs it possible to fix this problem?\n\nThank you\n \n \nNaum Derzhi\nChief Scientific Adviser, Physics\n \n3000 N Sam Houston Pkwy E, Technology Center, Office T1241H\nHouston, TX 77032\n\nOffice: +1 (281) 871 3278\n\n________________________________________\nThis e-mail, including any attached files, may contain confidential and privileged information for the sole use of the intended recipient. Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive information for the intended recipient), please contact the sender by reply e-mail and delete all copies of this message.\n\n"},{"id":"367557","messageId":"20190124151739.GA26616@sigill.intra.peff.net","threadId":"50308","inReplyTo":"7A854577E980BE4FB29FDD041B9B75E701798CEE@NP1EXMB203.corp.halliburton.com","subject":"Re: Removing data from repository","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-01-24T15:17:39Z","receivedAt":"2019-01-24T15:17:42Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Jan 24, 2019 at 02:51:50PM +0000, Naum Derzhi wrote:\n\n> I have this problem: years ago one of our developers committed a large\n> (Gigabytes) piece of binary data into our project repository. This\n> should not have been done, but it happened. (Honest, it was not me).\n> We never needed this data in the repository.\n> \n> Using git rm removes these files from the working tree, but they are\n> still somewhere in the repository, so when we clone, we transfer\n> gigabytes of unneeded data. \n> \n> Is it possible to fix this problem?\n\nYou'll have to rewrite the offending history. You can use\ngit-filter-branch. See especially these sections of the manpage:\n\n https://git-scm.com/docs/git-filter-branch#_examples\n\n https://git-scm.com/docs/git-filter-branch#_checklist_for_shrinking_a_repository\n\nas well as the warning in the DESCRIPTION section:\n\n WARNING! The rewritten history will have different object names for\n all the objects and will not converge with the original branch. You\n will not be able to easily push and distribute the rewritten branch on\n top of the original branch. Please do not use this command if you do\n not know the full implications, and avoid using it anyway, if a simple\n single commit would suffice to fix your problem. (See the \"RECOVERING\n FROM UPSTREAM REBASE\" section in git-rebase(1) for further information\n about rewriting published history.)\n\nYou may also want to check out the BFG repo cleaner[1], as separate\nproject that handles this case a little more simply (it doesn't save you\nfrom dealing with rewritten history, but it does avoid having to learn\nfilter-branch's flexible but confusing syntax).\n\n-Peff\n\n[1] https://rtyley.github.io/bfg-repo-cleaner/\n"},{"id":"367558","messageId":"7A854577E980BE4FB29FDD041B9B75E701799D06@NP1EXMB203.corp.halliburton.com","threadId":"50308","inReplyTo":"20190124151739.GA26616@sigill.intra.peff.net","subject":"RE: [EXTERNAL] Re: Removing data from repository","fromName":"Naum Derzhi","fromEmail":"naum.derzhi@halliburton.com","sentAt":"2019-01-24T15:27:48Z","receivedAt":"2019-01-24T15:28:17Z","isPatch":false,"sender":{"key":"naum.derzhi@halliburton.com","avatar":null},"body":"Thank you very much. We'll try\n\n \n \nNaum Derzhi\nChief Scientific Adviser, Physics\n \n3000 N Sam Houston Pkwy E, Technology Center, Office T1241H\nHouston, TX 77032\n\nOffice: +1 (281) 871 3278\n\nFollow Halliburton: LinkedIn | Facebook | Twitter | YouTube | Blog\n\n \n\n                                                                              \n\n\n\n\nThis e-mail, including any attached files, may contain confidential and privileged information for the sole use of the intended recipient. Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive information for the intended recipient), please contact the sender by reply e-mail and delete all copies of this message.\n\n-----Original Message-----\nFrom: Jeff King \nSent: Thursday, January 24, 2019 9:18 AM\nTo: Naum Derzhi \nCc: git@vger.kernel.org\nSubject: [EXTERNAL] Re: Removing data from repository\n\nExternal Sender: Use caution with links/attachments.\n\n\n\nOn Thu, Jan 24, 2019 at 02:51:50PM +0000, Naum Derzhi wrote:\n\n> I have this problem: years ago one of our developers committed a large\n> (Gigabytes) piece of binary data into our project repository. This \n> should not have been done, but it happened. (Honest, it was not me).\n> We never needed this data in the repository.\n>\n> Using git rm removes these files from the working tree, but they are \n> still somewhere in the repository, so when we clone, we transfer \n> gigabytes of unneeded data.\n>\n> Is it possible to fix this problem?\n\nYou'll have to rewrite the offending history. You can use git-filter-branch. See especially these sections of the manpage:\n\n https://urldefense.proofpoint.com/v2/url?u=https-3A__git-2Dscm.com_docs_git-2Dfilter-2Dbranch-23-5Fexamples&d=DwIBaQ&c=PskvixtEUDK7wuWU-tIg6oKuGYBRbrMXk2FZvF0UfTo&r=osDENe9k4dELyyh8ZmVzyWvVmzDFBB9Z_NHjYAXnqMg&m=090aUSqTq9bhewM2h12aExyV9vDRW60wwTcczIuu7Tg&s=_cdkm_E9dmM7yuNx0_5fpQKrV9wtnG4LPr8uooiV3cI&e=\n\n https://urldefense.proofpoint.com/v2/url?u=https-3A__git-2Dscm.com_docs_git-2Dfilter-2Dbranch-23-5Fchecklist-5Ffor-5Fshrinking-5Fa-5Frepository&d=DwIBaQ&c=PskvixtEUDK7wuWU-tIg6oKuGYBRbrMXk2FZvF0UfTo&r=osDENe9k4dELyyh8ZmVzyWvVmzDFBB9Z_NHjYAXnqMg&m=090aUSqTq9bhewM2h12aExyV9vDRW60wwTcczIuu7Tg&s=iBeuLkNNZbLqcAmKNw6JI7TnSt6KDObIDaBaEgQFcDQ&e=\n\nas well as the warning in the DESCRIPTION section:\n\n WARNING! The rewritten history will have different object names for\n all the objects and will not converge with the original branch. You\n will not be able to easily push and distribute the rewritten branch on\n top of the original branch. Please do not use this command if you do\n not know the full implications, and avoid using it anyway, if a simple\n single commit would suffice to fix your problem. (See the \"RECOVERING\n FROM UPSTREAM REBASE\" section in git-rebase(1) for further information\n about rewriting published history.)\n\nYou may also want to check out the BFG repo cleaner[1], as separate project that handles this case a little more simply (it doesn't save you from dealing with rewritten history, but it does avoid having to learn filter-branch's flexible but confusing syntax).\n\n-Peff\n\n[1] https://urldefense.proofpoint.com/v2/url?u=https-3A__rtyley.github.io_bfg-2Drepo-2Dcleaner_&d=DwIBaQ&c=PskvixtEUDK7wuWU-tIg6oKuGYBRbrMXk2FZvF0UfTo&r=osDENe9k4dELyyh8ZmVzyWvVmzDFBB9Z_NHjYAXnqMg&m=090aUSqTq9bhewM2h12aExyV9vDRW60wwTcczIuu7Tg&s=ADIQXtQrHUv3xXudfCN94eL00x0w0vr8uK2FQMpA2dI&e=\n"},{"id":"367559","messageId":"7A854577E980BE4FB29FDD041B9B75E701799D1C@NP1EXMB203.corp.halliburton.com","threadId":"50308","inReplyTo":"9dce6a3d8ff2f10621ea81fbd9ac9964c10aff9c.camel@mad-scientist.net","subject":"RE: [EXTERNAL] Re: Removing data from repository","fromName":"Naum Derzhi","fromEmail":"naum.derzhi@halliburton.com","sentAt":"2019-01-24T15:28:42Z","receivedAt":"2019-01-24T15:29:29Z","isPatch":false,"sender":{"key":"naum.derzhi@halliburton.com","avatar":null},"body":"Thank you\n\n \n \nNaum Derzhi\nChief Scientific Adviser, Physics\n \n3000 N Sam Houston Pkwy E, Technology Center, Office T1241H\nHouston, TX 77032\n\nOffice: +1 (281) 871 3278\n\nFollow Halliburton: LinkedIn | Facebook | Twitter | YouTube | Blog\n\n \n\n                                                                              \n\n\n\n\nThis e-mail, including any attached files, may contain confidential and privileged information for the sole use of the intended recipient. Any review, use, distribution, or disclosure by others is strictly prohibited. If you are not the intended recipient (or authorized to receive information for the intended recipient), please contact the sender by reply e-mail and delete all copies of this message.\n\n\n-----Original Message-----\nFrom: Paul Smith \nSent: Thursday, January 24, 2019 9:23 AM\nTo: Naum Derzhi ; git@vger.kernel.org\nSubject: [EXTERNAL] Re: Removing data from repository\n\nExternal Sender: Use caution with links/attachments.\n\n\n\nOn Thu, 2019-01-24 at 14:51 +0000, Naum Derzhi wrote:\n> Greetings,\n>\n> I have this problem: years ago one of our developers committed a large \n> (Gigabytes) piece of binary data into our project repository.\n> This should not have been done, but it happened. (Honest, it was not \n> me). We never needed this data in the repository.\n>\n> Using git rm removes these files from the working tree, but they are \n> still somewhere in the repository, so when we clone, we transfer \n> gigabytes of unneeded data.\n>\n> Is it possible to fix this problem?\n\nIt is possible, but it isn't pretty. By waiting so long to try to fix it you've compounded the impacts dramatically unfortunately. By removing that file you will in effect be rewriting the history of your repository starting with the commit that introduced the problematic file, that will be removed, all the way forward.\n\nThat means that every clone of the repository will have to be, at the very least, \"hard-reset\" and preferably (to avoid accidentally re- introducing the bad file) re-cloned from scratch.\n\nAs for actually removing the file, you can find information on how to do this all over the Google. Here's some reasonable advice from\nStackOverflow:\n\n\nhttps://urldefense.proofpoint.com/v2/url?u=https-3A__stackoverflow.com_questions_2100907_how-2Dto-2Dremove-2Ddelete-2Da-2Dlarge-2Dfile-2Dfrom-2Dcommit-2Dhistory-2Din-2Dgit-2Drepository&d=DwICaQ&c=PskvixtEUDK7wuWU-tIg6oKuGYBRbrMXk2FZvF0UfTo&r=osDENe9k4dELyyh8ZmVzyWvVmzDFBB9Z_NHjYAXnqMg&m=dmUTLEwyrdm6hRWOzlTFqBaVv4_WJoUWDed9sG4uDQ0&s=QrNGuuWuNZQqkSksRUK9sfcG5iwVZIxmE6UZcneRxHg&e=\n\n"},{"id":"367560","messageId":"9dce6a3d8ff2f10621ea81fbd9ac9964c10aff9c.camel@mad-scientist.net","threadId":"50308","inReplyTo":"7A854577E980BE4FB29FDD041B9B75E701798CEE@NP1EXMB203.corp.halliburton.com","subject":"Re: Removing data from repository","fromName":"Paul Smith","fromEmail":"paul@mad-scientist.net","sentAt":"2019-01-24T15:22:42Z","receivedAt":"2019-01-24T16:01:56Z","isPatch":false,"sender":{"key":"paul@mad-scientist.net","avatar":"https://avatars.githubusercontent.com/u/109636?v=4"},"body":"On Thu, 2019-01-24 at 14:51 +0000, Naum Derzhi wrote:\n> Greetings,\n> \n> I have this problem: years ago one of our developers committed a\n> large (Gigabytes) piece of binary data into our project repository.\n> This should not have been done, but it happened. (Honest, it was not\n> me). We never needed this data in the repository.\n> \n> Using git rm removes these files from the working tree, but they are\n> still somewhere in the repository, so when we clone, we transfer\n> gigabytes of unneeded data. \n> \n> Is it possible to fix this problem?\n\nIt is possible, but it isn't pretty. By waiting so long to try to fix\nit you've compounded the impacts dramatically unfortunately. By\nremoving that file you will in effect be rewriting the history of your\nrepository starting with the commit that introduced the problematic\nfile, that will be removed, all the way forward.\n\nThat means that every clone of the repository will have to be, at the\nvery least, \"hard-reset\" and preferably (to avoid accidentally re-\nintroducing the bad file) re-cloned from scratch.\n\nAs for actually removing the file, you can find information on how to\ndo this all over the Google. Here's some reasonable advice from\nStackOverflow:\n\n\nhttps://stackoverflow.com/questions/2100907/how-to-remove-delete-a-large-file-from-commit-history-in-git-repository\n\n"}]}