fix(aws): add dummy user message for Claude 4.5+/Opus 4.6+ trailing assistant turns#6008
Conversation
…ssistant turns Mirror PR livekit#4973's Anthropic-plugin fix on the AWS Bedrock plugin so Claude Sonnet 4.5+ and Opus 4.6+ requests don't fail with ValidationException ("This model does not support assistant message prefill. The conversation must end with a user message.") when the final message in `chat_ctx` is an assistant turn. Changes: - `_provider_format/aws.py::to_chat_ctx` — add `inject_trailing_user_message: bool = False`. When True and the messages list ends on `assistant`, append a minimal `{"role": "user", "content": [{"text": " "}]}` so the request ends on user. - `livekit-plugins-aws/.../llm.py` — add `_NO_PREFILL_PATTERNS` and `_model_disables_prefill(model)`; pass `inject_trailing_user_message` to `to_provider_format` when the active model is in the affected Claude family. - `tests/test_chat_ctx.py` — cover the new parameter (append when last is assistant; default-off keeps assistant last; idempotent when last is already user) and the model gate (matches plain IDs, snapshots, and cross-region inference profiles; excludes Nova/Llama/Mistral/older Claude/empty/None). Why substring matching instead of `startswith` like livekit#4973: Bedrock model IDs vary in shape — plain (`claude-sonnet-4-6`), date-stamped snapshots (`anthropic.claude-sonnet-4-5-20250929-v1:0`), and cross-region inference profiles (`global.anthropic.claude-sonnet-4-6`, `us.anthropic.claude-opus-4-7-v1:0`). The Anthropic plugin only sees short IDs from Anthropic's native API, so `startswith` works there; Bedrock needs substring matching to handle the prefixes. Closes livekit#6007 Refs livekit#4907, livekit#4973
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| _NO_PREFILL_PATTERNS = ( | ||
| "claude-sonnet-4-5", | ||
| "claude-sonnet-4-6", | ||
| "claude-sonnet-4-7", | ||
| "claude-opus-4-6", | ||
| "claude-opus-4-7", | ||
| "claude-opus-4-8", | ||
| ) |
There was a problem hiding this comment.
🚩 AWS _NO_PREFILL_PATTERNS is broader than Anthropic's equivalent
The AWS plugin's _NO_PREFILL_PATTERNS at livekit-plugins/livekit-plugins-aws/livekit/plugins/aws/llm.py:46-53 includes "claude-sonnet-4-5" and several future model versions (4-7, 4-8), while the Anthropic plugin at livekit-plugins/livekit-plugins-anthropic/livekit/plugins/anthropic/llm.py:41 only has ("claude-sonnet-4-6", "claude-opus-4-6"). The AWS list explicitly covers "Claude 4.5+" per the comment, but there is no model named claude-sonnet-4-5 in the Anthropic models list (livekit-plugins/livekit-plugins-anthropic/livekit/plugins/anthropic/models.py:6-17). This is likely intentional future-proofing, but means the two plugins have divergent prefill-disable thresholds if a Claude Sonnet 4.5 ever ships. Worth confirming this asymmetry is deliberate.
Was this helpful? React with 👍 or 👎 to provide feedback.
2 participants
Summary
Mirror #4973's Anthropic-plugin fix on the AWS Bedrock plugin. When the active model is in the Claude 4.5+/Opus 4.6+ family (which no longer supports assistant-message prefill), append a minimal dummy user message if the conversation would otherwise end on
assistant. This preventsbotocore.errorfactory.ValidationException: This model does not support assistant message prefill. The conversation must end with a user message.at the BedrockConverseStreamvalidation layer.Closes #6007. Anthropic-plugin precedent: #4973 (merged 2026-03-03), #4907.
Motivation
Anthropic removed assistant-message prefill starting in Claude Sonnet 4.5 and Opus 4.6 (carried forward to Sonnet 4.6/4.7, Opus 4.7/4.8). The same removal applies on Bedrock, where it surfaces as a
ConverseStreamValidationException. PR #4973 added end-side dummy-user injection to the Anthropic plugin gated on model ID, but the equivalent fix was never applied to the AWS Bedrock plugin — so anyone routing affected Claude models (e.g.,global.anthropic.claude-sonnet-4-6,us.anthropic.claude-opus-4-7-v1:0,anthropic.claude-sonnet-4-5-20250929-v1:0) throughlivekit.plugins.aws.llm.LLMhits the same wall.The AWS provider format already has start-side
inject_dummy_user_message=True(prepends a dummy whenmessages[0]["role"] != "user"); this PR adds the symmetric end-side handling.Changes
livekit-agents/livekit/agents/llm/_provider_format/aws.py— addinject_trailing_user_message: bool = Falseparameter toto_chat_ctx(). When True andmessages[-1]["role"] == "assistant", append{"role": "user", "content": [{"text": " "}]}. Defaults to False, so existing callers see no behavior change.livekit-plugins/livekit-plugins-aws/livekit/plugins/aws/llm.py— add_NO_PREFILL_PATTERNSand_model_disables_prefill(model), and passinject_trailing_user_message=_model_disables_prefill(self._opts.model)tochat_ctx.to_provider_format. Models in the affected Claude family get the injection; everything else (Nova, Llama, Mistral, older Claude) is unaffected.tests/test_chat_ctx.py— four new test cases:test_aws_inject_trailing_user_message_appends_when_last_is_assistant— happy path.test_aws_inject_trailing_user_message_default_off_keeps_assistant_last— backward compat for callers not opting in.test_aws_inject_trailing_user_message_idempotent_when_last_is_user— no extra dummy when already user-terminated.test_aws_model_disables_prefill_matches_inference_profiles— gate matches plain IDs, date-stamped snapshots, and cross-region inference profile prefixes; excludes Nova, Llama, Mistral, older Claude, and empty/None.Why substring matching instead of
startswithPR #4973 used
_NO_PREFILL_PATTERNS = ("claude-sonnet-4-6", "claude-opus-4-6")withmodel.startswith(p). That works for the Anthropic plugin because its model IDs are short and unprefixed (claude-sonnet-4-6).Bedrock model IDs vary in shape:
claude-sonnet-4-6anthropic.claude-sonnet-4-5-20250929-v1:0global.anthropic.claude-sonnet-4-6,us.anthropic.claude-opus-4-7-v1:0,eu.anthropic.claude-opus-4-6-v1:0startswith("claude-sonnet-4-6")would miss the prefixed forms, so this PR uses substring matching (any(p in model for p in _NO_PREFILL_PATTERNS)). If a future change unifies the two plugins' detection logic, that's a reasonable follow-up — keeping them separate for now mirrors #4973's local-to-the-plugin shape.Pattern list scope
Includes Sonnet 4.5/4.6/4.7 and Opus 4.6/4.7/4.8. Sonnet 4.5 is included (it's the actual cut-line per Anthropic's migration guide); #4973's list omitted it. Future versions will need additions to this tuple; the doc comment above it points at the Anthropic migration guide as the source of truth.
Verification
make format: clean (848 files unchanged).make lint: clean.make type-check: clean.tests/test_chat_ctx.py: 23 passed, 1 skipped (pre-existing), 0 failures.