usePaginatedQuery makes thousands of requests when using streams and filterwith together

[ignaciano3]

Summary

When using complex convex paginated queries in which there are streams using .filterWith(...) the hook usePaginatedQuery (from convex-helpers or convex) doesn't know when to stop and gets data until there are no more rows to fetch.

Steps To Reproduce

1. Make a table with an index and add generous sample data.
2. Create a paginated query that fetches data from that index merging many streams that also does some other filtering using filterWith.
3. Add a console log somewhere in the query optionally.
4. In a React app, use the usePaginatedQuery hook to get the first data available.
5. You should see that it fetches many more times than requested (and we are not using the loadMore)

Expected behaviour

Only one fetch to the backend

Actual behaviour

Thousands of fetches, making it very expensive.

Notes

I have other queries where there are no filterWith and it works fine.

Example of a query

export const listarAfiliadosPorEtapaDelegacion = query({
  args: {
    paginationOpts: paginationOptsValidator,
    etapa: etapaAfiliacionValidator,
    delegaciones: v.optional(v.array(v.string())),
    comercializadora: v.string(),
    vendedorEmail: v.string(),
    filters: listarAfiliadosFiltersValidator,
  },
  handler: async (ctx, args) => {
    const {
      etapa,
      delegaciones,
      comercializadora,
      vendedorEmail,
      paginationOpts,
      filters,
    } = args;
    const {
      texto,
      uid,
      control,
      categoria,
      regimen,
      auditoriaMedica,
      delegacion,
    } = filters;

    // Some parts of the query are omitted to simplify

    console.log(args)
    const db = stream(ctx.db, schema);

    // One indexed stream per delegacion
    const delegacionStreams = (delegaciones ?? []).map((d) =>
      db
        .query("afiliados")
        .withIndex("by_etapa_delegacion_completado", (q) =>
          q.eq("etapa", etapa).eq("delegacion", d),
        )
        .order("desc")
        .filterWith((q) =>
          Promise.resolve(matchesListarAfiliadosFilters(q, filters)),
        ),
    );

    // One indexed stream for vendedorEmail
    const vendedorStream = db
      .query("afiliados")
      .withIndex("by_etapa_vendedorEmail_completado", (q) =>
        q.eq("etapa", etapa).eq("vendedorEmail", vendedorEmail),
      )
      .order("desc")
      .filterWith((q) =>
        Promise.resolve(
          matchesListarAfiliadosFilters(q, filters) &&
            !delegaciones?.includes(q.delegacion),
        ),
      );

    const mergedStreams = mergedStream(
      [...delegacionStreams, vendedorStream],
      ["completado"],
    );

    return await mergedStreams.paginate(paginationOpts);
  },
});

const matchesListarAfiliadosFilters = (
  afiliado: {
    afiliadoUid?: string;
    delegacion?: string;
    control: boolean;
    categoria?: Categoria | "Desconocida";
    regimen?: Regimen;
    auditoriaMedica?: EstadoAuditoria;
    vendedorEmail?: string;
  },
  filters: ListarAfiliadosFilters,
) => {
  if (filters.uid && afiliado.afiliadoUid !== filters.uid) {
    return false;
  }

 ...
 // Similar filters

  return true;
};