Add custom Qwen3 model with configurable attention and latentMoE.#3613
Merged
copybara-service[bot] merged 1 commit intomainfrom Apr 17, 2026
Merged
Add custom Qwen3 model with configurable attention and latentMoE.#3613copybara-service[bot] merged 1 commit intomainfrom
copybara-service[bot] merged 1 commit intomainfrom
Conversation
copybara-service
bot
requested review from
A9isha,
NicoGrande,
NuojCheng,
RissyRan,
SurbhiJainUSC,
aireenmei,
bvandermoon,
dipannita08,
gagika,
gobbleturk,
hengtaoguo,
igorts-git,
jesselu-google,
jiangjy1982,
khatwanimohit,
michelle-yooh,
parambole,
richjames0,
shralex,
shuningjin,
suexu1025 and
vipannalla
as code owners
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
copybara-service
bot
force-pushed
the
test_896749554
branch
3 times, most recently
from
86b6433 to
4d67a7d
Compare
copybara-service
bot
force-pushed
the
test_896749554
branch
2 times, most recently
from
8216eff to
a1a0123
Compare
copybara-service
bot
force-pushed
the
test_896749554
branch
2 times, most recently
from
b21b20a to
f679f0a
Compare
copybara-service
bot
changed the title
copybara-service
bot
force-pushed
the
test_896749554
branch
15 times, most recently
from
9350e8a to
3cf64af
Compare
Specifically, this introduces: * `attention_output_dim` and `moe_expert_input_dim` to allow the attention block output and the MoE expert input to have different dimensionalities than the base embedding dimension. * A `dense_init_scale` config to allow configuring the initialization scale for dense layers across all models (replacing the hardcoded 1.0). PiperOrigin-RevId: 901021328
copybara-service
bot
force-pushed
the
test_896749554
branch
from
3cf64af to
7f78228
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add custom Qwen3 model with configurable attention and latentMoE.
Specifically, this introduces:
attention_output_dimandmoe_expert_input_dimto allow the attentionblock output and the MoE expert input to have different dimensionalities
than the base embedding dimension.
dense_init_scaleconfig to allow configuring the initialization scalefor dense layers across all models (replacing the hardcoded 1.0).