Embedding Lanes
Embedding lanes control how CoreCube shares an embedding provider between two very different kinds of work:
- Interactive query embeddings — short, latency-sensitive calls used by chat and search.
- Background document embeddings — larger batches created during ingestion, model changes, and re-embedding.
Without lane control, both kinds of work use the same embedding provider path. A heavy re-embedding job can therefore compete with query embeddings and make chat/search feel slow.
With Embedding lanes enabled, CoreCube gives query embeddings a protected lane and makes background embedding workers slow down or yield when queries need the provider.
Admin Console → Embedding → Lanes.
What embedding lanes do
Embedding lanes do not change the embedding model. They change admission control: which embedding calls are allowed to reach the provider at the same time.
CoreCube uses three practical protections:
| Protection | What it does |
|---|---|
| Interactive admission | Query embeddings use the interactive lane instead of waiting behind all background batches. |
| Background worker yielding | When a query is waiting, background workers pause before starting their next embedding batch. |
| Bounded query wait + fallback | If a query cannot get a lane permit in time, retrieval falls back to keyword search instead of hanging forever. |
Lane control does not interrupt an embedding request that is already running inside the provider. It prevents too much new background work from entering the provider and makes workers yield between batches.
Performance settings vs. lane settings
The Performance presets still decide how much background embedding work CoreCube tries to run. For example, a preset can configure:
- worker count
- batch concurrency per worker
- batch size
Lane settings decide how much of that requested work may actually call the embedding provider at once.
The effective background embedding concurrency is:
actual background calls = min(workers × batch concurrency, background lane limit)
Example:
| Setting | Value |
|---|---|
| Performance preset | Balanced |
| Worker count | 1 |
| Batch concurrency | 2 |
| Background lane limit | 1 |
| Actual background provider calls | min(1 × 2, 1) = 1 |
If you switch the performance preset to a higher-throughput mode but leave Background concurrency
at 1, the extra workers can prepare and wait, but only one background embedding call reaches the
provider at a time.
Think of the controls this way:
- Performance settings are how much background work CoreCube wants to schedule.
- Lane settings define how many provider calls that work is allowed to consume.
UI reference
The Embedding Lanes section contains live status, presets, and advanced controls.
Embedding lanes toggle
The Embedding lanes switch is the main feature flag. It is enabled by default when no explicit setting has been saved.
| State | Behavior |
|---|---|
| Off | Query and document embeddings use the prior shared path. Performance settings fully control background parallelism. |
| On | Query embeddings use the interactive lane, background workers use background permits, and query pressure can make workers yield. |
When the toggle is off, the lanes table still shows the configured lane limits, but the lanes are not active.
Provider state
Provider state summarizes the current embedding provider profile and health.
| Field | Meaning |
|---|---|
| Local profile | CoreCube inferred a local provider, usually because the active embedding provider has no API key configured. |
| Cloud/API profile | CoreCube inferred a managed API provider, usually because an API key is configured. |
| Healthy / saturated / rate-limited / circuit open | Current provider health from lane and embedding-provider telemetry. |
Local providers usually need conservative lane settings. Cloud providers can often tolerate more parallel requests.
Query protection
Query protection shows whether background workers are currently yielding to query pressure.
| State | Meaning |
|---|---|
| No active yield | No recent query pressure is causing background workers to pause. |
| Yield active | A query recently needed the provider, so background workers are pausing between batches. |
This state is most useful while a re-embedding job is running and users are actively querying the system.
Recent query degradations
This counter increases when query embedding could not complete normally and CoreCube had to degrade retrieval, such as falling back to keyword search.
0 is the expected value in a healthy system. A rising number means queries are not getting lane
permits quickly enough or the embedding provider is timing out.
Embedding lanes table
The table shows live lane state.
| Column | Meaning |
|---|---|
| Lane | Interactive, Background, or Maintenance. |
| Limit | Maximum concurrent calls allowed for that lane. |
| In flight | Calls currently holding a lane permit. |
| Waiting | Calls waiting for a lane permit. |
| Available | Remaining slots for the lane. |
| P95 wait | Recent 95th-percentile wait time for a lane permit. |
| Timeouts | Calls that could not get a lane permit in time. |
| Yields | Times background workers paused because queries needed provider access. |
| Last success | Most recent successful embedding call for that lane. |
The important row for chat/search responsiveness is Interactive. The important row for document embedding throughput is Background.
Provider presets
Provider presets are safe starting points.
| Preset | Best for | Behavior |
|---|---|---|
| Local-safe | Local CPU/GPU/MPS embedding servers | Keeps concurrency conservative and makes background workers yield for queries. |
| Cloud/API | Managed embedding APIs | Raises interactive and background limits for providers that support parallel calls. |
For local providers, start with Local-safe. For cloud providers, use Cloud/API only after checking provider rate limits and account quotas.
Lane controls
| Control | What it changes |
|---|---|
| Interactive concurrency | Number of query embedding calls that may run at once. |
| Background concurrency | Number of document/chunk embedding calls that may run at once across workers. |
| Maintenance concurrency | Number of maintenance embedding calls that may run at once. |
| Query wait budget | How long a query may wait for a lane permit before retrieval degrades. |
| Yield window | How long background workers treat recent query pressure as active. |
| Query cache TTL | How long repeated query embeddings stay cached. |
| Query cache entries | Maximum number of cached query embeddings. |
| Yield background workers when a query is waiting | Makes background workers pause between batches when query pressure is detected. |
Settings and API
Embedding lane settings live in the shared settings table. The admin UI writes them through typed retrieval-config endpoints rather than the generic settings surface.
| Setting key | Meaning |
|---|---|
embedding_lanes_enabled | Turns the lane admission path on or off. Defaults to on. |
embedding_lanes_provider_profile | Explicit local/cloud/custom profile override for lane presets. |
embedding_interactive_max_concurrency | Maximum concurrent query embedding calls. |
embedding_background_max_concurrency | Cross-worker background embedding call limit. |
embedding_maintenance_max_concurrency | Maintenance embedding call limit. |
embedding_background_yield_on_query | Whether workers pause when query pressure is present. |
embedding_background_yield_window_ms | How long query pressure stays active for workers. |
embedding_interactive_wait_budget_ms | Maximum query wait for a lane permit. |
embedding_query_cache_ttl_seconds | Query embedding cache lifetime. |
embedding_query_cache_max_entries | Query embedding cache size cap. |
embedding_provider_lane_mode | shared or split endpoint routing. |
embedding_query_endpoint_override | Optional query-only endpoint in split mode. |
embedding_background_endpoint_override | Optional background-only endpoint in split mode. |
The admin API surface is:
| Endpoint | Purpose |
|---|---|
GET /api/retrieval/config/lanes | Read lane status, settings, and query timeout. |
PATCH /api/retrieval/config/lane-settings | Update lane settings. |
POST /api/retrieval/config/lane-preset | Apply local-safe or cloud-api. |
POST /api/retrieval/config/lane-probe | Probe split query/background endpoints. |
PUT /api/retrieval/config/lane-config | Save shared/split lane endpoint routing. |
Recommended local starting point
For a local embedding provider:
| Control | Recommended value |
|---|---|
| Performance preset | Balanced |
| Interactive concurrency | 1 |
| Background concurrency | 1 |
| Maintenance concurrency | 1 |
| Yield background workers | On |
This keeps document embedding moving while making chat/search more responsive.
Recommended cloud starting point
For a managed embedding API:
| Control | Recommended value |
|---|---|
| Performance preset | Fast |
| Interactive concurrency | 2 |
| Background concurrency | 2 or higher, within provider limits |
| Maintenance concurrency | 1 |
| Yield background workers | On |
Raise concurrency gradually. If you see rate-limit cooldowns, query degradations, or provider timeouts, lower background concurrency first.
Advanced query/background split
The advanced split section is optional. It lets you send query embeddings and background embeddings to different compatible embedding endpoints.
Common examples:
- a fast endpoint reserved for query embeddings
- a higher-throughput endpoint for document re-embedding
- two local inference processes pinned to different hardware resources
Before saving a split configuration, CoreCube probes both endpoints and verifies that they are compatible:
- same embedding dimensions
- same model behavior
- same input mode
- same normalization
CoreCube rejects split lanes when the probe fails. Query and document embeddings must remain compatible because they are compared in the same vector index.
Troubleshooting
| Symptom | What to check |
|---|---|
| Chat/search slows down during re-embedding | Turn on Embedding lanes, use Local-safe, and keep Background concurrency low. |
| Document embedding is slower than expected | Check whether Background concurrency is lower than the Performance preset's requested parallelism. |
| Recent query degradations increase | Lower Background concurrency, increase Query wait budget carefully, or check provider health. |
| Yields keep increasing | Users are querying while background work is active; this is expected when query protection is doing its job. |
| Provider is rate-limited | Lower background concurrency and check provider account limits. |
| Split lanes cannot be saved | Run the probe and confirm both endpoints produce compatible embeddings. |
Related
- Embedding & Chunking — document chunking, keyword language, and active index configuration.
- Retrieval Pipeline — how query embeddings participate in retrieval.