{"thread":{"id":"53336","subject":"Fwd: Repack memory usage","startedAt":"2020-04-28T05:16:05Z","lastAt":"2020-04-28T05:16:05Z","messageCount":1,"participants":["Bryan Turner"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"396433","messageId":"CAGyf7-FZ5vUCgs+DY-GZUb+zNNBPQmVgLzOLf+wqgO7PpHikDw@mail.gmail.com","threadId":"53336","inReplyTo":"CAGyf7-FWssYXkJQ_LignBPRuVchrAR9MFE7rX5d+vy8PLCY5Mw@mail.gmail.com","subject":"Fwd: Repack memory usage","fromName":"Bryan Turner","fromEmail":"bturner@atlassian.com","sentAt":"2020-04-28T05:15:51Z","receivedAt":"2020-04-28T05:16:05Z","isPatch":false,"sender":{"key":"bturner@atlassian.com","avatar":"https://gravatar.com/avatar/16bcf3167981c1ef7c804e502642366d888a35b0d0b0a4ca01fdc442aa1acb1e?d=mp&s=160"},"body":"I'm trying to help a user who has run into a situation where their\nrepository has gotten so \"big\" (to some dimension of \"big\"ness) that\nthey can no longer repack it on their hardware. Every time repack\nruns, it ends up consuming so much RAM the Linux OOM killer kills it\nbefore it can complete.\n\nThe repack command being used is: git repack -Adln\n--unpack-unreachable=72.hours.ago\n\nA count-objects on the repository (without -h, so these sizes are in\nK) showed this:\n\ncount: 346911\nsize: 18367032\nin-pack: 45117481\npacks: 27\nsize-pack: 9462621\nprune-packable: 22584\ngarbage: 0\nsize-garbage: 0\n\nAfter the next repack was killed by the OOM killer, the numbers changed to this:\n\ncount: 427870\nsize: 22559712\nin-pack: 90554922\npacks: 37\nsize-pack: 18953693\nprune-packable: 371285\ngarbage: 0\nsize-garbage: 0\n\nThe machine in question has 12 cores and 32GB of RAM. I've tried\nsetting pack.threads=1, but still hit the OOM killer. Bumping the RAM\nfor the machine to 64GB, paired with limiting repacking to a single\nthread, finally produced a successful repack. (But it appears to be\nunreliable, with some runs still failing. By the time the successful\nrepack completed the git process was showing over 60GB of RAM in\nuse--though that number is likely misleading.) There’s no setting\napplied for pack.windowMemory, but my local testing (on a different\nrepository, I should note) didn’t show much change to repack’s overall\nmemory usage even with that set. Repacking a 500MB repository with\npack.windowMemory set to “100m” (and pack.threads set to 1) had still\nallocated far more than 100MB of RAM before the process completed.\n\nUnfortunately, I don't really know too much about the contents of the\nrepository. I _do_ know that it contains binaries, some of which may\nbe large, but that's all I can find out; the user is unable to share\nthe actual repository with me.\n\nDoes anyone have any suggestions on how I can constrain repack's\nmemory usage, so this user can get a successful repack without\nrequiring all the memory in the system to run (and bringing the system\nto a complete standstill for all other processing while it’s running)?\n\nBest regards,\nBryan Turner\n"}]}