# [bug] user may be cornered into delete files #9901 7 messages from 2024-11-14 to 2024-11-18. Participants: A bughunter, brian m. carlson, Jeff King. Thread: https://gitlist.dev/t/62491 ## A bughunter, 2024-11-14 17:53 Subject: [bug] user may be cornered into delete files #9901 Message-ID: URL: https://gitlist.dev/e/P_ttzPnNTtUgpAy8cg_ntLvwskJhz7cmv-fFgDw0K4Y2vOCx8v_AXxHtHEuBtpREOQu9pxpvK6JgLusc5Uyn2wDZ4HZvC3ttM4k-Ry6jBs8%3D%40proton.me ``` -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 [bug] user may be cornered into delete files #9901 pls see https://github.com/cli/cli/issues/9901 and look into all of the links. from A_bughunter@proton.me Sent with Proton Mail secure email. -----BEGIN PGP SIGNATURE----- Version: ProtonMail wnUEARYKACcFgmc2OQgJkKkWZTlQrvKZFiEEZlQIBcAycZ2lO9z2qRZlOVCu 8pkAAMpYAP92jH1GmK/fhQDScPH7WylcYaYatoIkZGTKhYGtvAcWdQEAqa9I 34yBlDPlbMPExwW6vNJEk0gzwHLKbZjgXO2WrAA= =0D8V -----END PGP SIGNATURE----- ``` ## A bughunter, 2024-11-14 17:56 Subject: Fw: [bug] user may be cornered into delete files #9901 Message-ID: URL: https://gitlist.dev/e/ZJaaSrglj_jUGE-7IfLWHaUIKKLGlw1SJfPP8MeyvdVKO7iMFhbTQaBKeJVLePFGkw3Dvo1co7B6MBKkTN7v5J6hRqjb8yHiB6_-4H8PnAs%3D%40proton.me In-Reply-To: ``` from A_bughunter@proton.me Sent with Proton Mail secure email. ------- Forwarded Message ------- From: A bughunter Date: On Thursday, November 14th, 2024 at 17:53 Subject: [bug] user may be cornered into delete files #9901 To: git@vger.kernel.org > [bug] user may be cornered into delete files #9901 pls see https://github.com/cli/cli/issues/9901 and look into all of the links. > > > > from A_bughunter@proton.me > > Sent with Proton Mail secure email. ``` ## A bughunter, 2024-11-14 18:03 Subject: Fw: [bug] user may be cornered into delete files #9901 Message-ID: URL: https://gitlist.dev/e/XdBUcr7M60BIudOE7vay5rAdPqP06WOrPnt0ORM8QtlkEr980kTDIxtCJUySclvxfuFiyinGtvTmCBrhe4Y4mtwZcNgseoTb6JtzgIjrwWI%3D%40proton.me In-Reply-To: ``` from A_bughunter@proton.me Sent with Proton Mail secure email. ------- Forwarded Message ------- From: A bughunter Date: On Thursday, November 14th, 2024 at 17:56 Subject: Fw: [bug] user may be cornered into delete files #9901 To: git@vger.kernel.org > > > > > from A_bughunter@proton.me > > Sent with Proton Mail secure email. > > > ------- Forwarded Message ------- > From: A bughunter A_bughunter@proton.me > > Date: On Thursday, November 14th, 2024 at 17:53 > Subject: [bug] user may be cornered into delete files #9901 > To: git@vger.kernel.org git@vger.kernel.org > > > > > [bug] user may be cornered into delete files #9901 pls see https://github.com/cli/cli/issues/9901 and look into all of the links. > > > > from A_bughunter@proton.me > > > > Sent with Proton Mail secure email. ``` ## brian m. carlson, 2024-11-14 23:37 Subject: Re: [bug] user may be cornered into delete files #9901 Message-ID: URL: https://gitlist.dev/e/ZzaJzm4kyYbcDSgm%40tapette.crustytoothpaste.net In-Reply-To: ``` On 2024-11-14 at 17:53:15, A bughunter wrote: > [bug] user may be cornered into delete files #9901 pls see https://github.com/cli/cli/issues/9901 and look into all of the links. What you're describing sounds like a network problem of some sort. I don't believe this is a bug in Git because Git handles this kind of thing on a regular basis without a problem. The only thing which might potentially be a problem on the Git side is that I don't know if we try to hold the connection open without sending a sideband during pack generation, in which case if the client side doesn't send anything at all, then the connection might be closed by the server. I'll point out that GitHub sends SSH keepalives, so typically the connection should not be reset unless the connection actually drops. However, you could try pushing over HTTPS instead, which won't try to make the connection until the pack is generated, so even if we do what I suggested might be a problem above, that wouldn't affect HTTPS. I will mention that the repository you mentioned there contains a large number of TIFF images about 1–2 MB in size. These TIFFs appear to be using JPEG compression, so when Git tries to deltify them (during the "Compressing objects" stage), that step is just going to waste a lot of CPU and take a long time, since trying to deltify compressed objects doesn't work and is just slow. That's contributing to the slow push performance. In general, Git is not a great way to store large numbers of compressed images. If you're going to store images in your repository, you should make them uncompressed (which you can do with TIFFs) so that Git can store them more efficiently. What would be even better, since your data appears to be effectively a book, is to store the data as text, possibly with a small set of images for illustrations, instead of storing the book as a set of images. That would be much more searchable and accessible as well. The Git FAQ[0] mentions this and makes recommendations on approach. Overall, I would not say this is a bug in Git. Pushing over HTTPS may help you get your pushes working in a more robust way, but in general, I'd recommend storing the data in your repository differently. [0] https://git-scm.com/docs/gitfaq#recommended-storage-settings -- brian m. carlson (they/them or he/him) Toronto, Ontario, CA ``` ## A bughunter, 2024-11-15 05:15 Subject: Fw: Re: [bug] user may be cornered into delete files #9901 Message-ID: URL: https://gitlist.dev/e/TOa5gGm8tmncWfFotfGhpWsRxDWvHH74skCKqc0_QPbRuWLvm-JFnDK8Y6SQxWooczbInHWtDbmmznLrDbr1pPSZ7vvPIlUAP9dSZcydRpg%3D%40proton.me In-Reply-To: ``` My reply for git mailing list about the bug and more for brian is here forwarded because the reply button reply's to the sender and not the mailing list. See more for brian after the reply. from A_bughunter@proton.me Sent with Proton Mail secure email. ------- Forwarded Message ------- From: A bughunter Date: On Friday, November 15th, 2024 at 04:31 Subject: Re: [bug] user may be cornered into delete files #9901 To: brian m. carlson > My reply to brian m. carlson. > > No, you brian seem to be missing what the bug is my complaint is this: > if the initial push fails (for any reason) then the user must reset thus deleting files. My capacity is new to git as you would see on my github.com/freedom-foundation profile. You can run the problem again and see if a --soft reset would evade having to delete files. Help resources instructed hard reset when asking how to undo a commit because push fails. Can a --soft reset undo add without deleting the files which initially have no backup. Bear in mind the files could be anything, text, tiff is not part of the bugreport. Any initial files would have to be deleted if push fails for any reason is the bugreport. > brian, I was not impressed that you could not gather what my bugreport was before I rephrased it. brian, For years my files have been the victim of vandalism and hex editors, and likely everybody's files are because I have yet to meet anybody whome can describe how to perfectly checksum text. I get what you say brian about a text file instead of TIFF but the conversion to text from TIFF is a long drawn out process which may take a while so I have stored the original TIFF which is further necesarry to authenticate the sources of The Book. Unicode is a problem see here github.com/freedom-foundation/unicode-map . Can you help to build an ANSI ASCII only Linux kernel because the Unicode is a problem and Unicode is in the Linux kernel. In order to convert The Book to text I need 7-bit ASCII ( see here https://github.com/freedom-foundation/ASCII-format-for-Network-Interchange ) and for checksums to remain stable either BOM or a BOM notation along with the checksum. Unicode is a problem because you cannot checksum text while fighting against vandalism and binary/hex editors.You may know also calibrating OCR such as tesseract is not a snap and may need about ten percent of total size to be corrected which is effectually dilligently re-reading The Book that being the case it is about as easy to type it by hand instead of using OCR such as tesseract while fighting vandalism and binary/hex editors which impeade the process. The process cannot be verified without checksums while protecting against corruption. While. > > from A_bughunter@proton.me > > Sent with Proton Mail secure email. > > > On Thursday, November 14th, 2024 at 23:37, brian m. carlson sandals@crustytoothpaste.net wrote: > > > On 2024-11-14 at 17:53:15, A bughunter wrote: > > > > > [bug] user may be cornered into delete files #9901 pls see https://github.com/cli/cli/issues/9901 and look into all of the links. > > > > What you're describing sounds like a network problem of some sort. I > > don't believe this is a bug in Git because Git handles this kind of > > thing on a regular basis without a problem. > > > > The only thing which might potentially be a problem on the Git side is > > that I don't know if we try to hold the connection open without sending > > a sideband during pack generation, in which case if the client side > > doesn't send anything at all, then the connection might be closed by the > > server. I'll point out that GitHub sends SSH keepalives, so typically > > the connection should not be reset unless the connection actually > > drops. > > > > However, you could try pushing over HTTPS instead, which won't try to > > make the connection until the pack is generated, so even if we do what I > > suggested might be a problem above, that wouldn't affect HTTPS. > > > > I will mention that the repository you mentioned there contains a large > > number of TIFF images about 1–2 MB in size. These TIFFs appear to be > > using JPEG compression, so when Git tries to deltify them (during the > > "Compressing objects" stage), that step is just going to waste a lot of > > CPU and take a long time, since trying to deltify compressed objects > > doesn't work and is just slow. That's contributing to the slow push > > performance. > > > > In general, Git is not a great way to store large numbers of compressed > > images. If you're going to store images in your repository, you should > > make them uncompressed (which you can do with TIFFs) so that Git can > > store them more efficiently. What would be even better, since your data > > appears to be effectively a book, is to store the data as text, possibly > > with a small set of images for illustrations, instead of storing the > > book as a set of images. That would be much more searchable and > > accessible as well. The Git FAQ[0] mentions this and makes > > recommendations on approach. > > > > Overall, I would not say this is a bug in Git. Pushing over HTTPS may > > help you get your pushes working in a more robust way, but in general, > > I'd recommend storing the data in your repository differently. > > > > [0] https://git-scm.com/docs/gitfaq#recommended-storage-settings > > -- > > brian m. carlson (they/them or he/him) > > Toronto, Ontario, CA ``` ## Jeff King, 2024-11-15 09:32 Subject: Re: [bug] user may be cornered into delete files #9901 Message-ID: <20241115093214.GA1749331@coredump.intra.peff.net> URL: https://gitlist.dev/e/20241115093214.GA1749331%40coredump.intra.peff.net In-Reply-To: ``` On Thu, Nov 14, 2024 at 11:37:50PM +0000, brian m. carlson wrote: > The only thing which might potentially be a problem on the Git side is > that I don't know if we try to hold the connection open without sending > a sideband during pack generation, in which case if the client side > doesn't send anything at all, then the connection might be closed by the > server. I'll point out that GitHub sends SSH keepalives, so typically > the connection should not be reset unless the connection actually > drops. Yes, we'll hold the connection open for git-over-ssh. We can't generate the pack until we've seen what the other side advertises, and it's probably too heavy-weight to start a second ssh connection. Even with ssh keepalives, I would not be surprised if the process terminating the Git-level protocol conversation on the server side had some internal timeouts. There are Git-level keepalives during the similar compression operation of a clone/fetch, as well as the delta resolution for the server side of a push. But there's nothing during the client-side compression. I know we've discussed this on the list before, but I couldn't find anything substantive, and certainly not patches. I think it would _probably_ work for the client to send 0-length pktlines (actual "0004", not "0000" flushes) every few seconds while it's waiting. But it would be the first time we've done so from the client side, and the first time we've done it outside of sideband framing. So it's possible a server might not like it (in which case we'd probably need a new protocol extension). > Overall, I would not say this is a bug in Git. Pushing over HTTPS may > help you get your pushes working in a more robust way, but in general, > I'd recommend storing the data in your repository differently. I agree with everything you said, but I wanted to add one more workaround: running "git gc" locally will pack all of those objects into a single pack. And then the subsequent push should be fast, because we'll already have done the delta search. -Peff ``` ## A bughunter, 2024-11-18 22:04 Subject: Re: [bug] user may be cornered into delete files #9901 Message-ID: URL: https://gitlist.dev/e/kQjk0kPkRFcFhFJzAV12mEO_0RDHGLeoWoxbFVwNfXa_i5MJ4o37D-BHTdCnIS6a0EwTKprHrAY3tHl8OPZ_9wpXByHFk0gP8JUlpUmJuIg%3D%40proton.me In-Reply-To: ``` -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 BROAD OVERVIEW bugreport[A], question[B] and use-case[C]. These are all related but separate threads for the purposes as labled. Please do not cross-post or cross-quote. Focus on productivity and solving these. You are welcomed to view and participate in all of these as I contribute more. use-case[C] - git question (short) rephrase with use-case added. ADD, ADD, ADD why cant they get that: ADD. ADD tracks files for commit. It's already been commit : push failed. Failed pushes piled up. I need them untracked. How do you undo an add ( many adds): simple question. Without deleting any files, to repush 1 by 1. I suspect the answer is: "you can't - git provides no means to do so" - a defect, bug as we call it. The answer to this use-case question likely will confirm my bugreport and I get the feeling this is why they will not answer the question they do not want to accept my bugreport and have a bad spirit to argue. The excuses they give for this will be that it is not typical: This can normally be solved by repushing. However it is still a defect and in my use-case there is a slow or intermittent connection and no sha-backups yet because this is the initial creation of a repo. The above question is a use-case[C] seeking how to undo without deleting files. There shouldn't be any need to know a use-case in order to answer a technical question. I am open to discuss my use-case on this thread. The use-case if of this repo github.com/freedom-foundation/kjbible My original bug report is that the user " may be cornered into delete files" bugreport[A] Summary as of 20241117 brian does not believe it's a bug and begins chattering to Peff about how to fix the bug "There are Git-level keepalives during the similar compression operation". The full question[B] which is a child of the bugreport[A] is here. Summary as of 20241117 my question was mostly sidestepped and spammed whilst the thread devolved into a political drama about code_of_conduct. I am not open to discuss my use-case on this thread. [A]: https://lore.kernel.org/git/20241115093214.GA1749331@coredump.intra.peff.net/#r [B]: https://lore.kernel.org/git/xmqqr079xty7.fsf@gitster.g/T/#t [C]: https://lore.kernel.org/all/4hiTc8Kx5yNhYuN8abv3QFJBuptM6VWZ9OKvkdZFlSI5y0zoK-lN_VHf-QCSEjllmSWvu9V-tbrvFOx17_P0Nq8UKxEcK3Rs2d02FjbYuUc=@proton.me/#r -----BEGIN PGP SIGNATURE----- Version: ProtonMail wnUEARYKACcFgmc7uf0JkKkWZTlQrvKZFiEEZlQIBcAycZ2lO9z2qRZlOVCu 8pkAABh9AQDQxbp7CHRFU0poLOuwPdICYc07bOUMHu/EXC4SweKiPAEA6fiT yPK2ueIqGYTPzGYtEHLxw7wsV8ZuXS7mE6HseAA= =/xm/ -----END PGP SIGNATURE----- ```