feat: add generate-chunks + invoke-function Knative handlers#1228
Closed
pyramation wants to merge 2 commits into
Closed
feat: add generate-chunks + invoke-function Knative handlers#1228pyramation wants to merge 2 commits into
pyramation wants to merge 2 commits into
Conversation
…e tables Implements the generate_chunks job handler that processes auto-chunking jobs enqueued by the embedding_chunks trigger. When content is inserted into an agent knowledge table (or any table with ProcessChunks), the trigger enqueues a job which this function picks up and processes: 1. Read content from the parent table row 2. Split into chunks (paragraph/sentence/fixed/semantic strategies) 3. Generate per-chunk embeddings via configurable provider (Ollama default) 4. Insert chunks into the child chunks table Also updates docker-compose.jobs.yml to include the generate-chunks service and adds it to the INTERNAL_GATEWAY_DEVELOPMENT_MAP routing.
Contributor
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
|
All alerts resolved. Learn more about Socket for GitHub. This PR previously contained dependency changes with security issues that have been resolved, removed, or ignored. |
Schema-aware function invocation handler that: - Resolves the correct schema (public or private) from function_module config - Reads the invocation row and validates function definitions (is_invocable) - Dispatches execution to the underlying Knative function endpoint - Updates invocation row with status, result, duration_ms, and timestamps Also adds invoke-function service to docker-compose.jobs.yml with dev routing (function:invoke -> http://invoke-function:8080). The handler queries metaschema_modules_public.function_module to resolve the PostgreSQL schema name at runtime, supporting both public and private schema configurations as well as entity-scoped invocation tables.
devin-ai-integration
Bot
changed the title
Contributor
|
Closing — the invoke-function handler was premature. The generate-chunks function will be re-submitted separately. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two new Knative function handlers:
1.
generate-chunks— Knowledge auto-chunkingProcesses
chunk:generate_chunksjobs. When a row is inserted into a knowledge table (e.g.agent_knowledge), this handler:_chunkstable withchunk_index,chunk_text, and source FKUses
include_meta: truein the job payload for schema/table resolution.2.
invoke-function— Schema-aware function invocationProcesses
function:invokejobs. When a row is INSERTed into afunction_invocationstable, this handler:metaschema_modules_public.function_moduleconfigis_invocable = true)INTERNAL_GATEWAY_DEVELOPMENT_MAPKey design: Schema resolution is dynamic — queries
function_module+schematables at runtime to find the PostgreSQL schema name. Supports both public and private schema configurations, as well as entity-scoped invocation tables.docker-compose.jobs.yml
invoke-functionservice with dev routingfunction:invoketo knative-job-serviceINTERNAL_GATEWAY_DEVELOPMENT_MAPandJOBS_SUPPORTEDSQL-side companion: constructive-io/constructive-db#1348 adds the
data_job_triggeronfunction_invocationsINSERT.Review & Testing Checklist for Human
invoke-function/src/schema-resolver.ts— verify the metaschema query correctly joinsfunction_module→schemato resolve the PostgreSQL schema nameinvoke-function/src/index.ts— verify the invocation lifecycle (pending → running → completed/failed) and that status/result/duration updates use the correct column namesdocker-compose.jobs.ymlrouting — ensurefunction:invokeappears in bothINTERNAL_GATEWAY_DEVELOPMENT_MAPandJOBS_SUPPORTEDcd functions/invoke-function && pnpm test(5 tests)Notes
The invoke-function handler pairs with constructive-db PR #1348 which adds the SQL job trigger. The generate-chunks handler pairs with constructive-db PR #1347 which documents the task identifier.
Link to Devin session: https://app.devin.ai/sessions/8820b639fbd94ac8bfae86cfb01cf827
Requested by: @pyramation