Search Pagination Plan

[SeanTAllen]

Context

GitHub search endpoints return paginated results via Link headers, just like list
endpoints. But the JSON shape differs:

• List endpoints (labels, etc.) return a top-level JSON array: [...]
• Search endpoints return an object with metadata: {"total_count": N, "incomplete_results": bool, "items": [...]}

The library's current PaginatedJsonRequester → PaginatedResultReceiver →
PaginatedListJsonConverter chain handles the list shape. SearchIssues currently
uses the non-paginated JsonRequester → ResultReceiver chain, so search results
have no next_page()/prev_page().

The existing TODO in search.pony:9-21 outlines the intent.

Approach

Duplicate the paginated HTTP plumbing for search rather than trying to share it.

The PaginatedJsonRequester/handler factory/handler chain is parameterized on A
because it holds a concrete PaginatedResultReceiver[A]. This type parameter
carries real type information through the pipeline to PaginatedList[A] — it's
how Pony knows what the list contains. Erasing it via an unparameterized interface
would lose compile-time type safety. A parameterized interface doesn't help either
— A would be phantom since success and failure signatures don't mention it.

Duplicating the ~100 lines of mechanical HTTP code (requester, handler factory,
handler) is the cleanest approach. This duplication can be consolidated later
if a good shared abstraction emerges.

Steps

Step 1: Add pagination to SearchResults[A] in search.pony

Modify SearchResults[A] to hold pagination state and expose navigation:

• Add fields: _creds, _converter, _prev_link, _next_link
• Change constructor from public create to private _create (instances are
  always built by the converter, never by end users — matches PaginatedList's
  _from_array pattern)
• Add prev_page() and next_page() methods returning
  (Promise[(SearchResults[A] | RequestError)] | None)
• Add _retrieve_link() private helper (same pattern as PaginatedList._retrieve_link)
• Keep items as a public field (consistent with PaginatedList.results — making
  both methods is a separate refactor)

Step 2: Add PaginatedSearchJsonConverter[A] in search.pony

A new converter class (analogous to PaginatedListJsonConverter[A]) that:

1. Parses the search JSON envelope (extracts total_count, incomplete_results, items)
2. Converts each item using the inner JsonConverter[A]
3. Parses the Link header for prev/next links
4. Constructs SearchResults[A] via _create

This replaces IssueSearchResultsJsonConverter.

Step 3: Add SearchResultReceiver[A] actor in search.pony

A new actor (analogous to PaginatedResultReceiver[A]) that:

• Holds a Promise[(SearchResults[A] | RequestError)]
• Holds a PaginatedSearchJsonConverter[A]
• On success(json, link_header): converts and fulfills the promise
• On failure(...): wraps in RequestError and fulfills

Step 4: Add search HTTP plumbing in search.pony

Duplicate the paginated HTTP infrastructure for search use:

• SearchJsonRequester: same as PaginatedJsonRequester, but apply[A] takes
  SearchResultReceiver[A] instead of PaginatedResultReceiver[A]
• SearchJsonRequesterHandlerFactory[A]: same as PaginatedJsonRequesterHandlerFactory[A],
  stores SearchResultReceiver[A]
• SearchJsonRequesterHandler[A]: same as PaginatedJsonRequesterHandler[A],
  stores SearchResultReceiver[A]

These are mechanical duplicates (~100 lines) of the corresponding paginated types.
The only difference is the receiver type. This duplication can be cleaned up later
if a good shared abstraction is found.

Note: The paginated handler has a bug on line 236 — (_status != 301) or (_status != 307)
is always true (should be and). The search duplicate should use the correct logic.

Step 5: Update SearchIssues primitive in search.pony

Change from non-paginated to paginated infrastructure:

• Use PaginatedSearchJsonConverter[Issue] instead of IssueSearchResultsJsonConverter
• Use SearchResultReceiver[Issue] instead of ResultReceiver[SearchResults[Issue]]
• Use SearchJsonRequester instead of JsonRequester

Step 6: Clean up search.pony

• Remove IssueSearchResultsJsonConverter (replaced by generic converter)
• Remove the TODO block (lines 9-21)
• Add use plp = "pagination_link_parser" import

Files changed

File	What
github_rest_api/search.pony	Bulk of the work: paginated SearchResults, new converter, new receiver, search HTTP plumbing, updated SearchIssues

No changes to paginated_list.pony or repository.pony — the existing paginated
infrastructure is left untouched.

Verification

make

The existing unit tests cover SimpleURITemplate and PaginationLinkParser which
are infrastructure this change depends on. There are no integration tests that hit
the live GitHub API, so verification that the search pagination actually works
end-to-end would require manual testing with a real token, or adding a test that
mocks the HTTP layer (which doesn't exist yet).

The key compile-time checks:

• SearchResultReceiver[A] type-checks with SearchJsonRequester.apply[A]
• PaginatedResultReceiver[A] continues to type-check with PaginatedJsonRequester.apply[A]
  (no changes to existing paginated infrastructure)
• All existing call sites compile unchanged

Manual testing

The examples/search-issues/ example is the natural test harness. After the change,
update it to exercise pagination — following the same recursive-chaining pattern
used by examples/get-repository-labels/main.pony:

// In PrintResults, after printing the first page:
match results.next_page()
| let promise: Promise[IssueSearchResultsOrError] =>
  promise.next[None](PrintResults~apply(out))
else
  out.print("------- No more results")
end

Requirements for manual testing:

• A GitHub personal access token (passed via --token). Public search works
  without a token but is rate-limited to 10 requests/minute.
• A query broad enough to produce multiple pages. GitHub search returns 30 items
  per page by default. Example: --query "repo:ponylang/ponyc is:issue" (ponyc
  has thousands of issues, so this will produce many pages).

What to verify:

1. First page prints results and total_count
2. Second page prints additional results (not the same ones)
3. Last page prints "No more results" (i.e. next_page() returns None)
4. total_count and incomplete_results are consistent across pages
5. Existing get-repository-labels example still works (regression check — no
   changes to paginated infrastructure)

[SeanTAllen]

Implemented in #50