{"thread":{"id":"55417","subject":"Detecting when bulk file-system operations complete","startedAt":"2021-03-31T03:40:54Z","lastAt":"2021-04-02T15:02:16Z","messageCount":4,"participants":["Drew Noakes","Philip Oakley","Derrick Stolee","Jeff Hostetler"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"420681","messageId":"CAJd66x6XT7m5Njg2kRyGJ80rU6WNcLifijS98=onJeBz+74rrA@mail.gmail.com","threadId":"55417","inReplyTo":null,"subject":"Detecting when bulk file-system operations complete","fromName":"Drew Noakes","fromEmail":"drew@drewnoakes.com","sentAt":"2021-03-31T03:39:37Z","receivedAt":"2021-03-31T03:40:54Z","isPatch":false,"sender":{"key":"drew@drewnoakes.com","avatar":null},"body":"Hi,\n\nI develop IDE tooling that watches a repo's workspace and reacts to changes.\n\nBulk file-system changes (i.e. branch switch, rebase, merge,\ncherry-pick) trigger lots of file system events, and my tooling should\nignore intermediary updates. Currently I debounce events with a fixed\ntime span, but would like a more reliable and performant approach to\nscheduling this reactive work.\n\nCan this be done by monitoring the GITDIR in some way? For example, is\nthere a file that's present when these operations are in flight, and\nwhich is removed when they complete?\n\nIf an operation is interrupted (i.e. merge or rebase that hits a\nconflict) my tooling should consider the bulk operation as complete.\nThis means that detecting a git-rebase-todo file or\nrebase-merge/rebase-apply folder is not adequate.\n\nAny help appreciated. Thanks!\n\nDrew.\n"},{"id":"420711","messageId":"5a1abc6b-52a3-fc12-166f-8af5e7bdff48@iee.email","threadId":"55417","inReplyTo":"CAJd66x6XT7m5Njg2kRyGJ80rU6WNcLifijS98=onJeBz+74rrA@mail.gmail.com","subject":"Re: Detecting when bulk file-system operations complete","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.email","sentAt":"2021-03-31T16:03:21Z","receivedAt":"2021-03-31T16:03:59Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"Hi Drew,\n\nOn 31/03/2021 04:39, Drew Noakes wrote:\n> Hi,\n>\n> I develop IDE tooling that watches a repo's workspace and reacts to changes.\n>\n> Bulk file-system changes (i.e. branch switch, rebase, merge,\n> cherry-pick) trigger lots of file system events, and my tooling should\n> ignore intermediary updates. Currently I debounce events with a fixed\n> time span, but would like a more reliable and performant approach to\n> scheduling this reactive work.\n>\n> Can this be done by monitoring the GITDIR in some way? For example, is\n> there a file that's present when these operations are in flight, and\n> which is removed when they complete?\n>\n> If an operation is interrupted (i.e. merge or rebase that hits a\n> conflict) my tooling should consider the bulk operation as complete.\n> This means that detecting a git-rebase-todo file or\n> rebase-merge/rebase-apply folder is not adequate.\n>\n> Any help appreciated. Thanks!\n>\n> Drew.\nYou may want to look at the various bits of work on `fsmonitor` etc on\nthe mailing list archive\n\nhttps://lore.kernel.org/git/?q=fsmonitor\n\nto ensure that all the different approaches inter-operate with\nreasonable efficiency..\n\n-- \nPhilip\n"},{"id":"420815","messageId":"9c09459b-2d27-6997-63c1-46c8df52eda7@gmail.com","threadId":"55417","inReplyTo":"5a1abc6b-52a3-fc12-166f-8af5e7bdff48@iee.email","subject":"Re: Detecting when bulk file-system operations complete","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-04-01T13:10:55Z","receivedAt":"2021-04-01T17:48:00Z","isPatch":false,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/31/2021 12:03 PM, Philip Oakley wrote:\n> Hi Drew,\n> \n> On 31/03/2021 04:39, Drew Noakes wrote:\n>> Hi,\n>>\n>> I develop IDE tooling that watches a repo's workspace and reacts to changes.\n>>\n>> Bulk file-system changes (i.e. branch switch, rebase, merge,\n>> cherry-pick) trigger lots of file system events, and my tooling should\n>> ignore intermediary updates. Currently I debounce events with a fixed\n>> time span, but would like a more reliable and performant approach to\n>> scheduling this reactive work.\n>>\n>> Can this be done by monitoring the GITDIR in some way? For example, is\n>> there a file that's present when these operations are in flight, and\n>> which is removed when they complete?\n>>\n>> If an operation is interrupted (i.e. merge or rebase that hits a\n>> conflict) my tooling should consider the bulk operation as complete.\n>> This means that detecting a git-rebase-todo file or\n>> rebase-merge/rebase-apply folder is not adequate.\n>>\n>> Any help appreciated. Thanks!\n>>\n>> Drew.\n> You may want to look at the various bits of work on `fsmonitor` etc on\n> the mailing list archive\n> \n> https://lore.kernel.org/git/?q=fsmonitor\n> \n> to ensure that all the different approaches inter-operate with\n> reasonable efficiency..\n\nThis is an important suggestion. There is an issue with the current\napproach of using FS Monitor with Watchman along with Visual Studio\nCode and certain Git plugins:\n\n1. When \"git status\" runs, the FS Monitor hook asks Watchman for\n   changes. Watchman puts a \"cookie file\" in the working directory\n   so it knows when the file system events are flushed.\n\n2. VS Code notices this cookie file was written, so it interprets\n   that a file was changed in the working directory. It calls\n   \"git status\" to update its markers on the modified files.\n\nThis loops forever.\n\nThe new version of FS Monitor that Jeff Hostetler is working on\nwrites the cookie file into the .git directory, which VS Code (and\nhopefully other IDEs) do not consider a trigger for running commands\nlike \"git status\".\n\nThis is just one example of the trickiness that ensues when using\nfilesystem events.\n\nThanks,\n-Stolee\n"},{"id":"420886","messageId":"365c1b90-73f4-18de-ae37-a13e52104e9c@jeffhostetler.com","threadId":"55417","inReplyTo":"9c09459b-2d27-6997-63c1-46c8df52eda7@gmail.com","subject":"Re: Detecting when bulk file-system operations complete","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-04-02T15:02:13Z","receivedAt":"2021-04-02T15:02:16Z","isPatch":false,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 4/1/21 9:10 AM, Derrick Stolee wrote:\n> On 3/31/2021 12:03 PM, Philip Oakley wrote:\n>> Hi Drew,\n>>\n>> On 31/03/2021 04:39, Drew Noakes wrote:\n>>> Hi,\n>>>\n>>> I develop IDE tooling that watches a repo's workspace and reacts to changes.\n>>>\n>>> Bulk file-system changes (i.e. branch switch, rebase, merge,\n>>> cherry-pick) trigger lots of file system events, and my tooling should\n>>> ignore intermediary updates. Currently I debounce events with a fixed\n>>> time span, but would like a more reliable and performant approach to\n>>> scheduling this reactive work.\n>>>\n>>> Can this be done by monitoring the GITDIR in some way? For example, is\n>>> there a file that's present when these operations are in flight, and\n>>> which is removed when they complete?\n>>>\n>>> If an operation is interrupted (i.e. merge or rebase that hits a\n>>> conflict) my tooling should consider the bulk operation as complete.\n>>> This means that detecting a git-rebase-todo file or\n>>> rebase-merge/rebase-apply folder is not adequate.\n>>>\n>>> Any help appreciated. Thanks!\n>>>\n>>> Drew.\n>> You may want to look at the various bits of work on `fsmonitor` etc on\n>> the mailing list archive\n>>\n>> https://lore.kernel.org/git/?q=fsmonitor\n>>\n>> to ensure that all the different approaches inter-operate with\n>> reasonable efficiency..\n> \n> This is an important suggestion. There is an issue with the current\n> approach of using FS Monitor with Watchman along with Visual Studio\n> Code and certain Git plugins:\n> \n> 1. When \"git status\" runs, the FS Monitor hook asks Watchman for\n>     changes. Watchman puts a \"cookie file\" in the working directory\n>     so it knows when the file system events are flushed.\n> \n> 2. VS Code notices this cookie file was written, so it interprets\n>     that a file was changed in the working directory. It calls\n>     \"git status\" to update its markers on the modified files.\n> \n> This loops forever.\n> \n> The new version of FS Monitor that Jeff Hostetler is working on\n> writes the cookie file into the .git directory, which VS Code (and\n> hopefully other IDEs) do not consider a trigger for running commands\n> like \"git status\".\n> \n> This is just one example of the trickiness that ensues when using\n> filesystem events.\n\nIt sounds like you want to watch the contents of the .git directory\nfor creation/deletion of various control files.  Something like a\nbegin- and end-transaction marker.  I'm not sure that Git has such\na clear marker.  Git commands often invoke child commands, rebase\nmight invoke checkout and then run some other commands for example,\nso the marker would need to be recursive.  And some commands can be\npaused, an interactive rebase can stop and let the user edit a command\nand then continue with the next step in the rebase.\n\nSo you would have to decide what the boundaries are that you want\nto bracket between your GUI refreshes -- do you freeze during the\nwhole rebase, or do you incrementally update the display after each\ncheckout or merge within the rebase?\n\nHaving said that, you might look at the `.git/index.lock`.  That file\nis present when a particular command is running and modifying things.\nIt is a start, but probably not sufficient.  And it is possible for\ncommands to abort (crash) and fail to delete that file, so be careful.\n\nYou could have a FS watcher to keep track of the contents of the .git\ndirectory.  But you'd have to do some research on which control files\nare created and when and see what makes sense for your application.\nI'm not saying it is impossible, but it probably won't be easy to\nget right.\n\nJeff\n"}]}