Skip to content

Create OpenAI API-compatible HTTP interface for mellea #521

@psschwei

Description

@psschwei

Description:
Create an OpenAI API-compatible HTTP server that wraps mellea, allowing any OpenAI-compatible client to use mellea backends.

Key deliverables:

  • using m serve serves an OpenAI API compatible endpoints
  • FastAPI/Starlette server implementing OpenAI chat completions endpoint
  • /v1/chat/completions endpoint with streaming support
  • Model listing endpoint
  • Configuration for backend selection

Acceptance criteria:

  • OpenAI Python SDK works against the server
  • Streaming works correctly
  • Tool calling works
  • Can be deployed as standalone service

Notes:

  • This enables "any framework that supports OpenAI" to use mellea
  • Consider using existing libraries like litellm proxy as reference

Task breakdown

P0 — Breaks Compatibility
These are things that cause OpenAI SDK clients to fail or return incorrect results.

  • Add finish_reason to response choices. Default to "stop"; set to "length" if max_tokens was hit; set to "tool_calls" if tool calls are present.
  • Add usage to response.
  • Map standard parameters to ModelOption sentinels.
  • Fix sync/async mismatch. Wrap module.serve() in asyncio.to_thread() or detect and await async serve functions.
  • Add OpenAI error response format. Return {"error": {"message":..., "type":..., "code":...}} on errors instead of FastAPI's default HTML/JSON.

P1 — Core Features Expected by Clients

  • see sub-issues
  • /v1/models endpoint. Return the model(s) configured in the serve script.
  • Health check endpoint. GET /health returning {"status": "ok"}.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions