git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v5 04/16] promisor-remote: implement promisor_remote_get_direct()

From
Christian Couder <christian.couder@gmail.com>
Date
Jun 25, 2019, 13:50 UTC
Message-ID
<CAP8UFD2-BUVsaVcO-pRNY+qc8WBZC=NUOQHnNYbBVKc=q_aOSg@mail.gmail.com>
In-Reply-To
<CAP8UFD0zvqC2s3_RsWfKwoWQL=2iV6PbaL8+hwUXrWBVe+6sgA@mail.gmail.com>

On Fri, May 31, 2019 at 7:10 AM Christian Couder <christian.couder@gmail.com> wrote:

> On Thu, May 30, 2019 at 7:21 PM Derrick Stolee <stolee@gmail.com> wrote:
> >
> > On 4/9/2019 12:11 PM, Christian Couder wrote:
Show 19 quoted lines
> > > +{
> > > +     int i, missing_nr = 0;
> > > +     int *missing = xcalloc(oid_nr, sizeof(*missing));
> > > +     struct object_id *old_oids = *oids;
> > > +     struct object_id *new_oids;
> > > +     int old_fetch_if_missing = fetch_if_missing;
> > > +
> > > +     fetch_if_missing = 0;
> >
> > This global 'fetch_if_missing' swap seems very fragile. I'm guessing you are using
> > it to prevent a loop when calling oid_object_info_extended() below. Can you instead
> > pass a flag to the method that disables the fetch_if_missing behavior?
>
> If such a flag existed when I wrote the function I would certainly
> have used it, as I also dislike this kind of messing with a global
> (and globals in general).
>
> I will see if I can do something about it according to what you
> suggest later in this thread.

In the V6 patch series I just sent, the new OBJECT_INFO_SKIP_FETCH_OBJECT flag that you introduced is used.

Show 8 quoted lines
> > > +     for (i = 0; i < oid_nr; i++)
> > > +             if (oid_object_info_extended(the_repository, &old_oids[i], NULL, 0)) {
> >
> > A use of "the_repository" this deep in new code is asking for a refactor later to remove it.
> > Please try to pass a "struct repository *r" through your methods so we minimize references
> > to the_repository (and the amount of work required to remove them later).
>
> Ok, I will take a look at that.

A "struct repository *r" is passed in V6. I forgot to mention that in the cover letter.

Show 6 quoted lines
> > > +                     missing_nr = remove_fetched_oids(&missing_oids, missing_nr, to_free);
> >
> > Here is the one call, and after this assignment "missing_nr" does mean the number of missing objects.
> > However, I do think this could be clarified by using remaining_nr and remaining_oids.
>
> Ok, I will take a look at using "remaining_nr" and "remaining_oids".
Done in V6 too.
Show 13 quoted lines
> > > +                     if (missing_nr) {
> > > +                             to_free = 1;
> > > +                             continue;
> > > +                     }
> >
> > Now this block took a bit to grok. You use to_free in the if(to_free) free(missing_oids); below.
> > But it also changes the behavior of remove_fetched_oids(). This means that the first time
> > remove_fetched_oids() will preserve the list (because it is the input list) but all later
> > calls will free the newly-created intermediate list. This checks out.
> >
> > What is confusing to me: is there any reason that missing_nr would be zero in this situation?
>
> I don't think so but I will check again, and maybe add a comment.

Actually missing_nr, or now remaining_nr, would be 0 if all the promised objects have been fetched.

Show 19 quoted lines
> > > +             }
> > > +             res = 0;
> > > +             break;
> > > +     }
> > > +
> > > +     if (to_free)
> > > +             free(missing_oids);
> > > +
> > > +     return res;
> > > +}
> >
> > While the test coverage report brought this patch to my attention, it does seem correct.
> > I still think a test exposing this method would be good, especially one that requires
> > a fetch_objects() call to multiple remotes to really exercise the details of remove_fetched_oids().
>
> Yeah, I would like to actually test it. I will take another look at
> what can be done to test this. Perhaps I will look at what can be done
> to still get some objects when fetching from a promisor/partial clone
> remote even when it doesn't have all of the objects we request.

I haven't improved test coverage or looked at how we could better handle a partial fetch. I plan to look at that soon.

Thanks, Christian.

Previous: Christian CouderNext: Christian Couder
Message 14 of 28 in “Many promisor remotes”
  1. 00/16 Many promisor remotesChristian Couder, Apr 9, 2019
  2. 01/16 t0410: remove pipes after git commandsChristian Couder, Apr 9, 2019
  3. 02/16 fetch-object: make functions return an error codeChristian Couder, Apr 9, 2019
  4. 03/16 Add initial support for many promisor remotesChristian Couder, Apr 9, 2019
  5. 05/16 promisor-remote: add promisor_remote_reinit()Christian Couder, Apr 9, 2019
  6. 06/16 promisor-remote: use repository_format_partial_cloneChristian Couder, Apr 9, 2019
  7. 04/16 promisor-remote: implement promisor_remote_get_direct()Christian Couder, Apr 9, 2019
  8. Derrick StoleeMay 30, 2019
  9. Johannes SchindelinMay 30, 2019
  10. Derrick StoleeMay 30, 2019
  11. Johannes SchindelinMay 31, 2019
  12. Junio C HamanoMay 31, 2019
  13. Christian CouderMay 31, 2019
  14. Christian CouderJun 25, 2019
  15. 07/16 Use promisor_remote_get_direct() and has_promisor_remote()Christian Couder, Apr 9, 2019
  16. 08/16 diff: use promisor-remote.h instead of fetch-object.hChristian Couder, Apr 9, 2019
  17. 09/16 promisor-remote: parse remote.*.partialclonefilterChristian Couder, Apr 9, 2019
  18. 10/16 builtin/fetch: remove unique promisor remote limitationChristian Couder, Apr 9, 2019
  19. 12/16 partial-clone: add multiple remotes in the docChristian Couder, Apr 9, 2019
  20. 11/16 t0410: test fetching from many promisor remotesChristian Couder, Apr 9, 2019
  21. 13/16 remote: add promisor and partial clone config to the docChristian Couder, Apr 9, 2019
  22. 14/16 Remove fetch-object.{c,h} in favor of promisor-remote.{c,h}Christian Couder, Apr 9, 2019
  23. 15/16 Move repository_format_partial_clone to promisor-remote.cChristian Couder, Apr 9, 2019
  24. 16/16 Move core_partial_clone_filter_default to promisor-remote.cChristian Couder, Apr 9, 2019
  25. Junio C HamanoApr 15, 2019
  26. Junio C HamanoApr 15, 2019
  27. Christian CouderApr 15, 2019
  28. Christian CouderApr 15, 2019

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.