Skip to content

[draft] wide metrics schema #6233

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
mattmkim wants to merge 27 commits into
base: main
Choose a base branch
from
Draft

[draft] wide metrics schema #6233

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
65372a2
PR1: adding quickwit-parquet-engine-crate, defining parquet schema, p…
mattmkim Mar 10, 2026
6cb7212
PR2: add ingest and indexing logic to quick-parquet-engine
mattmkim Mar 10, 2026
6b45aed
PR3: Add proto definitions for metrics ingest and split management
mattmkim Mar 10, 2026
7ef9d29
PR4: Metrics ingestion - OTel metrics parsing and Arrow conversion
mattmkim Mar 10, 2026
c5fd2a1
PR5: Add metastore support for metrics
mattmkim Mar 10, 2026
8914987
PR6: Generalize Source trait and SourceActor over Processor type
mattmkim Mar 10, 2026
e998596
PR7: Add parquet indexing pipeline actors and wiring
mattmkim Mar 10, 2026
8b2be61
minimal cargo lock and toml
mattmkim Mar 10, 2026
86415a3
cargo clippy
mattmkim Mar 10, 2026
d89a6c9
rustfmt, cargo machete
mattmkim Mar 10, 2026
c8f93cd
fix unit tests
mattmkim Mar 10, 2026
661ce42
fix accidental rename of index_id --> index_uid in build_index_id_pat…
mattmkim Mar 10, 2026
1c3605a
old test was asserting buggy sql
mattmkim Mar 10, 2026
e15e445
address codex comments
mattmkim Mar 10, 2026
8f5548b
address pr1 comments
mattmkim Mar 13, 2026
0b3fea8
minor change
fulmicoton-dd Mar 19, 2026
c183c6e
remove datadog specific stuff, remove otlp metrics endpoint
mattmkim Mar 27, 2026
86071ec
license, fix tests
mattmkim Mar 27, 2026
1e11160
license check
mattmkim Mar 27, 2026
6c5255f
actually fix license, formatting changes, remove expectation of otel …
mattmkim Mar 27, 2026
5f951ae
feat: dynamic MetricDataPoint with tag HashMap; rewrite Arrow batch b…
mattmkim Mar 18, 2026
6f86e8b
feat: replace ParquetField enum with SORT_ORDER/REQUIRED_FIELDS const…
mattmkim Mar 18, 2026
b41b564
feat: dynamic sort order and bloom filter configuration from batch sc…
mattmkim Mar 18, 2026
b3947a3
feat: schema-agnostic ingest validation and incremental union schema …
mattmkim Mar 18, 2026
34c9e6e
feat: dynamic column lookup in split writer
mattmkim Mar 18, 2026
23b4049
feat: update indexing actors to remove fixed ParquetSchema dependency
mattmkim Mar 18, 2026
92ebd86
refactor: extract shared test batch helpers to reduce duplication
mattmkim Mar 18, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions LICENSE-3rdparty.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,18 @@ anstyle-wincon,https://github.com/rust-cli/anstyle,MIT OR Apache-2.0,The anstyle
anyhow,https://github.com/dtolnay/anyhow,MIT OR Apache-2.0,David Tolnay <[email protected]>
arc-swap,https://github.com/vorner/arc-swap,MIT OR Apache-2.0,Michal 'vorner' Vaner <[email protected]>
arrayvec,https://github.com/bluss/arrayvec,MIT OR Apache-2.0,bluss
arrow,https://github.com/apache/arrow-rs,Apache-2.0,Apache Arrow <[email protected]>
arrow-arith,https://github.com/apache/arrow-rs,Apache-2.0,Apache Arrow <[email protected]>
arrow-array,https://github.com/apache/arrow-rs,Apache-2.0,Apache Arrow <[email protected]>
arrow-buffer,https://github.com/apache/arrow-rs,Apache-2.0,Apache Arrow <[email protected]>
arrow-cast,https://github.com/apache/arrow-rs,Apache-2.0,Apache Arrow <[email protected]>
arrow-data,https://github.com/apache/arrow-rs,Apache-2.0,Apache Arrow <[email protected]>
arrow-ipc,https://github.com/apache/arrow-rs,Apache-2.0,Apache Arrow <[email protected]>
arrow-ord,https://github.com/apache/arrow-rs,Apache-2.0,Apache Arrow <[email protected]>
arrow-row,https://github.com/apache/arrow-rs,Apache-2.0,Apache Arrow <[email protected]>
arrow-schema,https://github.com/apache/arrow-rs,Apache-2.0,Apache Arrow <[email protected]>
arrow-select,https://github.com/apache/arrow-rs,Apache-2.0,Apache Arrow <[email protected]>
arrow-string,https://github.com/apache/arrow-rs,Apache-2.0,Apache Arrow <[email protected]>
ascii-canvas,https://github.com/lalrpop/ascii-canvas,Apache-2.0 OR MIT,Niko Matsakis <[email protected]>
assert-json-diff,https://github.com/davidpdrsn/assert-json-diff,MIT,David Pedersen <[email protected]>
async-channel,https://github.com/smol-rs/async-channel,Apache-2.0 OR MIT,Stjepan Glavina <[email protected]>
Expand Down Expand Up @@ -142,6 +154,8 @@ console,https://github.com/console-rs/console,MIT,The console Authors
console-api,https://github.com/tokio-rs/console,MIT,"Eliza Weisman <[email protected]>, Tokio Contributors <[email protected]>"
console-subscriber,https://github.com/tokio-rs/console,MIT,"Eliza Weisman <[email protected]>, Tokio Contributors <[email protected]>"
const-oid,https://github.com/RustCrypto/formats/tree/master/const-oid,Apache-2.0 OR MIT,RustCrypto Developers
const-random,https://github.com/tkaitchuck/constrandom,MIT OR Apache-2.0,Tom Kaitchuck <[email protected]>
const-random-macro,https://github.com/tkaitchuck/constrandom,MIT OR Apache-2.0,Tom Kaitchuck <[email protected]>
const_fn,https://github.com/taiki-e/const_fn,Apache-2.0 OR MIT,The const_fn Authors
convert_case,https://github.com/rutrum/convert-case,MIT,rutrum <[email protected]>
core-foundation,https://github.com/servo/core-foundation-rs,MIT OR Apache-2.0,The Servo Project Developers
Expand Down Expand Up @@ -231,6 +245,7 @@ fiat-crypto,https://github.com/mit-plv/fiat-crypto,MIT OR Apache-2.0 OR BSD-1-Cl
find-msvc-tools,https://github.com/rust-lang/cc-rs,MIT OR Apache-2.0,The find-msvc-tools Authors
findshlibs,https://github.com/gimli-rs/findshlibs,MIT OR Apache-2.0,The findshlibs Authors
fixedbitset,https://github.com/petgraph/fixedbitset,MIT OR Apache-2.0,bluss
flatbuffers,https://github.com/google/flatbuffers,Apache-2.0,"Robert Winslow <[email protected]>, FlatBuffers Maintainers"
flate2,https://github.com/rust-lang/flate2-rs,MIT OR Apache-2.0,"Alex Crichton <[email protected]>, Josh Triplett <[email protected]>"
float-cmp,https://github.com/mikedilger/float-cmp,MIT,Mike Dilger <[email protected]>
fluent-uri,https://github.com/yescallop/fluent-uri-rs,MIT,Scallop Ye <[email protected]>
Expand Down Expand Up @@ -322,6 +337,7 @@ influxdb-line-protocol,https://github.com/influxdata/influxdb_iox/tree/main/infl
inherent,https://github.com/dtolnay/inherent,MIT OR Apache-2.0,David Tolnay <[email protected]>
inout,https://github.com/RustCrypto/utils,MIT OR Apache-2.0,RustCrypto Developers
instant,https://github.com/sebcrozet/instant,BSD-3-Clause,sebcrozet <[email protected]>
integer-encoding,https://github.com/dermesser/integer-encoding-rs,MIT,Lewin Bormann <[email protected]>
inventory,https://github.com/dtolnay/inventory,MIT OR Apache-2.0,David Tolnay <[email protected]>
ipcrypt-rs,https://github.com/jedisct1/rust-ipcrypt2,ISC,Frank Denis <[email protected]>
ipnet,https://github.com/krisprice/ipnet,MIT OR Apache-2.0,Kris Price <[email protected]>
Expand All @@ -347,6 +363,12 @@ lambda_runtime,https://github.com/awslabs/aws-lambda-rust-runtime,Apache-2.0,"Da
lambda_runtime_api_client,https://github.com/awslabs/aws-lambda-rust-runtime,Apache-2.0,"David Calavera <[email protected]>, Harold Sun <[email protected]>"
lazy_static,https://github.com/rust-lang-nursery/lazy-static.rs,MIT OR Apache-2.0,Marvin Löbel <[email protected]>
levenshtein_automata,https://github.com/tantivy-search/levenshtein-automata,MIT,Paul Masurel <[email protected]>
lexical-core,https://github.com/Alexhuszagh/rust-lexical,MIT OR Apache-2.0,Alex Huszagh <[email protected]>
lexical-parse-float,https://github.com/Alexhuszagh/rust-lexical,MIT OR Apache-2.0,Alex Huszagh <[email protected]>
lexical-parse-integer,https://github.com/Alexhuszagh/rust-lexical,MIT OR Apache-2.0,Alex Huszagh <[email protected]>
lexical-util,https://github.com/Alexhuszagh/rust-lexical,MIT OR Apache-2.0,Alex Huszagh <[email protected]>
lexical-write-float,https://github.com/Alexhuszagh/rust-lexical,MIT OR Apache-2.0,Alex Huszagh <[email protected]>
lexical-write-integer,https://github.com/Alexhuszagh/rust-lexical,MIT OR Apache-2.0,Alex Huszagh <[email protected]>
libc,https://github.com/rust-lang/libc,MIT OR Apache-2.0,The Rust Project Developers
libloading,https://github.com/nagisa/rust_libloading,ISC,Simonas Kazlauskas <[email protected]>
libm,https://github.com/rust-lang/compiler-builtins,MIT,Jorge Aparicio <[email protected]>
Expand Down Expand Up @@ -454,6 +476,10 @@ parking_lot,https://github.com/Amanieu/parking_lot,Apache-2.0 OR MIT,Amanieu d'A
parking_lot,https://github.com/Amanieu/parking_lot,MIT OR Apache-2.0,Amanieu d'Antras <[email protected]>
parking_lot_core,https://github.com/Amanieu/parking_lot,Apache-2.0 OR MIT,Amanieu d'Antras <[email protected]>
parking_lot_core,https://github.com/Amanieu/parking_lot,MIT OR Apache-2.0,Amanieu d'Antras <[email protected]>
parquet,https://github.com/apache/arrow-rs,Apache-2.0,Apache Arrow <[email protected]>
parquet-variant,https://github.com/apache/arrow-rs,Apache-2.0,Apache Arrow <[email protected]>
parquet-variant-compute,https://github.com/apache/arrow-rs,Apache-2.0,Apache Arrow <[email protected]>
parquet-variant-json,https://github.com/apache/arrow-rs,Apache-2.0,Apache Arrow <[email protected]>
parse-size,https://github.com/kennytm/parse-size,MIT,kennytm <[email protected]>
paste,https://github.com/dtolnay/paste,MIT OR Apache-2.0,David Tolnay <[email protected]>
pbkdf2,https://github.com/RustCrypto/password-hashes/tree/master/pbkdf2,MIT OR Apache-2.0,RustCrypto Developers
Expand Down Expand Up @@ -603,6 +629,7 @@ security-framework,https://github.com/kornelski/rust-security-framework,MIT OR A
security-framework-sys,https://github.com/kornelski/rust-security-framework,MIT OR Apache-2.0,"Steven Fackler <[email protected]>, Kornel <[email protected]>"
semver,https://github.com/dtolnay/semver,MIT OR Apache-2.0,David Tolnay <[email protected]>
separator,https://github.com/saghm/rust-separator,MIT,Saghm Rossi <[email protected]>
seq-macro,https://github.com/dtolnay/seq-macro,MIT OR Apache-2.0,David Tolnay <[email protected]>
serde,https://github.com/serde-rs/serde,MIT OR Apache-2.0,"Erick Tryzelaar <[email protected]>, David Tolnay <[email protected]>"
serde-value,https://github.com/arcnmx/serde-value,MIT,arcnmx
serde_core,https://github.com/serde-rs/serde,MIT OR Apache-2.0,"Erick Tryzelaar <[email protected]>, David Tolnay <[email protected]>"
Expand Down Expand Up @@ -687,13 +714,15 @@ thiserror,https://github.com/dtolnay/thiserror,MIT OR Apache-2.0,David Tolnay <d
thiserror-impl,https://github.com/dtolnay/thiserror,MIT OR Apache-2.0,David Tolnay <[email protected]>
thousands,https://github.com/tov/thousands-rs,MIT OR Apache-2.0,Jesse A. Tov <[email protected]>
thread_local,https://github.com/Amanieu/thread_local-rs,MIT OR Apache-2.0,Amanieu d'Antras <[email protected]>
thrift,https://github.com/apache/thrift/tree/master/lib/rs,Apache-2.0,Apache Thrift Developers <[email protected]>
tikv-jemalloc-ctl,https://github.com/tikv/jemallocator,MIT OR Apache-2.0,"Steven Fackler <[email protected]>, Gonzalo Brito Gadeschi <[email protected]>, The TiKV Project Developers"
tikv-jemalloc-sys,https://github.com/tikv/jemallocator,MIT OR Apache-2.0,"Alex Crichton <[email protected]>, Gonzalo Brito Gadeschi <[email protected]>, The TiKV Project Developers"
tikv-jemallocator,https://github.com/tikv/jemallocator,MIT OR Apache-2.0,"Alex Crichton <[email protected]>, Gonzalo Brito Gadeschi <[email protected]>, Simon Sapin <[email protected]>, Steven Fackler <[email protected]>, The TiKV Project Developers"
time,https://github.com/time-rs/time,MIT OR Apache-2.0,"Jacob Pratt <[email protected]>, Time contributors"
time-core,https://github.com/time-rs/time,MIT OR Apache-2.0,"Jacob Pratt <[email protected]>, Time contributors"
time-fmt,https://github.com/MiSawa/time-fmt,MIT OR Apache-2.0,mi_sawa <[email protected]>
time-macros,https://github.com/time-rs/time,MIT OR Apache-2.0,"Jacob Pratt <[email protected]>, Time contributors"
tiny-keccak,https://github.com/debris/tiny-keccak,CC0-1.0,debris <[email protected]>
tinystr,https://github.com/unicode-org/icu4x,Unicode-3.0,The ICU4X Project Developers
tinytemplate,https://github.com/bheisler/TinyTemplate,Apache-2.0 OR MIT,Brook Heisler <[email protected]>
tinyvec,https://github.com/Lokathor/tinyvec,Zlib OR Apache-2.0 OR MIT,Lokathor <[email protected]>
Expand Down
Loading
Loading