{"thread":{"id":"60995","subject":"git mv fails to deduplicate blob objects on transfer","startedAt":"2024-02-25T18:34:48Z","lastAt":"2024-02-25T18:34:48Z","messageCount":1,"participants":["Milan Hauth"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"489322","messageId":"CAGiEHCub4H7ZCV3CqfFaCRTOhN5A=qy7G_p1pVQw_puyAgjM8w@mail.gmail.com","threadId":"60995","inReplyTo":null,"subject":"git mv fails to deduplicate blob objects on transfer","fromName":"Milan Hauth","fromEmail":"milahu@gmail.com","sentAt":"2024-02-25T18:34:36Z","receivedAt":"2024-02-25T18:34:48Z","isPatch":false,"sender":{"key":"milahu@gmail.com","avatar":"https://gravatar.com/avatar/68beefdc29b5666a84773ff8094481ddd56ff3d2f997948c05dedb586574afa9?d=mp&s=160"},"body":"\"git mv\" followed by \"git push\" or \"git pull\"\ncan produce wasteful transfers of blob objects\n\nthese transfers are wasteful\nbecause the blob object already exists in the destination repo\nbut \"git push\" or \"git pull\" fail to see that\n\nthis affects only some cases of \"git mv\"\nin some cases, the deduplication works as expected\nin other cases, dedup fails\n\nthis is neglegible for small files, but noticable with large files\n\nin my case, i tried to move 5GB of files (250 x 20MB)\nand i was surprised as \"git push\" wanted to transfer 5GB\ninstead of a few bytes for the tree and commit objects\n\nto reproduce: see repro.sh in\nhttps://github.com/milahu/git-bug-git-mv-wasteful-transfer\n\n\n\noutput of repro.sh\n\nthe first size is the transfer size before \"git mv\"\nthe second size is the transfer size after \"git mv\"\n\npass: 1.00 MiB != 288 bytes # path_a=file_a; path_b=file_b\npass: 1.00 MiB != 331 bytes # path_a=dir/file_a; path_b=dir/file_b\npass: 1.00 MiB != 286 bytes # path_a=dir_a/file; path_b=dir_b/file\npass: 1.00 MiB != 284 bytes # path_a=file; path_b=dir/file\npass: 1.00 MiB != 329 bytes # path_a=file; path_b=dir1/dir2/file\npass: 1.00 MiB != 373 bytes # path_a=file; path_b=dir1/dir2/dir3/file\npass: 1.00 MiB != 331 bytes # path_a=file_a; path_b=dir/file_b\npass: 1.00 MiB != 376 bytes # path_a=file_a; path_b=dir1/dir2/file_b\npass: 1.00 MiB != 420 bytes # path_a=file_a; path_b=dir1/dir2/dir3/file_b\npass: 1.00 MiB != 241 bytes # path_a=dir/file; path_b=file\nFAIL: 1.00 MiB == 1.00 MiB # path_a=dir1/dir2/file; path_b=file\nFAIL: 1.00 MiB == 1.00 MiB # path_a=dir1/dir2/dir3/file; path_b=file\nFAIL: 1.00 MiB == 1.00 MiB # path_a=dir/file_a; path_b=file_b\nFAIL: 1.00 MiB == 1.00 MiB # path_a=dir1/dir2/file_a; path_b=file_b\nFAIL: 1.00 MiB == 1.00 MiB # path_a=dir1/dir2/dir3/file_a; path_b=file_b\nFAIL: 1.00 MiB == 1.00 MiB # path_a=dir1a/dir2a/file; path_b=dir1b/dir2b/file\nFAIL: 1.00 MiB == 1.00 MiB # path_a=dir1a/file_a; path_b=dir1b/file_b\n\n\n\nsee also\n\nhttps://colabti.org/ircloggy/git/2024-02-24#l704\n\nhttps://colabti.org/ircloggy/git/2024-02-25#l218\n\nhttps://colabti.org/ircloggy/git/2024-02-25#l269\n\n> I have a strong déjà vu about this also; I think we talked about this exact thing a while ago\n\n> this, same: https://colabti.org/ircloggy/git/2023-09-13#l912\n\nhttps://colabti.org/ircloggy/git/2024-02-25#l404\n\nhttps://colabti.org/ircloggy/git/2024-02-25#l433\n\n> reading that SO answer by jthill but can't quite get the whole picture from it -- is it saying that it's a trade-off in sending all the objects vs. spending resources trying to figure out what to send?\n\n> I bet there's an opportunity for optimization here; Git could probably figure out a good balance based on how much data it is about to send\n\nhttps://stackoverflow.com/questions/48228425/git-push-new-branch-with-same-files-uploads-all-files-again\n"}]}