Inference Deployment
CoreCube inference is configured in the Admin Console under Inference → Deployment. Enable local inference first, then choose the target CoreCube should use:
| Setting | What it controls |
|---|---|
| Local inference | Whether CoreCube uses local embedding and reranker workers instead of cloud or external providers |
| Platform | Hardware/runtime family: CPU, Apple MPS, NVIDIA CUDA, AMD ROCm, or External |
| Deployment mode | How CoreCube reaches inference workers: Docker sidecar, Native Python process, or External endpoint |
| Lifecycle management | Who starts and stops workers: Inference Agent (managed) or Manual commands (you run them) |
Local inference is the on/off switch for local workers. Platform selects the inference target. Deployment mode selects the transport. Lifecycle management selects whether the Inference Agent can Start, Stop, and Restart workers for you.
Saving deployment settings does not start workers. Start and restart happen only from the Deployment tab after the Inference Agent is connected, or outside CoreCube when you choose manual lifecycle.
Admin Console → Inference → Deployment tab. Connect the Inference Agent when using managed lifecycle. Use Inference → Commands & logs for generated manual commands and worker logs.
How the settings fit together
Think of deployment as five layers:
Host OS → Local inference → Platform → Deployment mode → Lifecycle management
- Host OS (Linux or macOS) is detected from the machine running the Inference Agent. CoreCube uses it to block combinations that cannot work on that host. Production inference targets Linux servers; macOS is supported for Apple Silicon development.
- Local inference must be enabled before CoreCube can connect to or manage local embedding and reranker workers.
- Platform is the inference hardware family you want (CPU, CUDA, MPS, and so on).
- Deployment mode is how workers run (containers, native processes, or a remote URL).
- Lifecycle management is whether CoreCube's Inference Agent manages start/stop, or you run commands yourself.
make dev, make deploy-dev, make local, and make deploy-local start CoreCube, PostgreSQL,
object storage, and supporting services only. They do not start inference sidecars or native
inference workers. Start embedding and reranker workers from Inference → Deployment after the
stack is running.
The matching stop commands (make down-dev, make down-dev-server, and make down-local) clean
up local inference workers and the Inference Agent if they were started during the session.
Managed vs manual lifecycle
| Lifecycle | Start / Stop / Restart in Admin | Typical use |
|---|---|---|
| Inference Agent | Yes — click Start on the Deployment tab | Day-to-day local inference and managed Docker sidecars |
| Manual commands | No — use Commands & logs or your own runbooks | External endpoints, unsupported targets, or deliberate operator control |
See When to use manual lifecycle for the full decision guide.
Important: The Inference Agent only manages lifecycle for:
- Docker sidecar on hosts where Docker sidecars are supported (Linux), and
- Native Python when Platform is Apple MPS or CPU (host processes managed by the agent).
GPU native Python on Linux (NVIDIA CUDA or AMD ROCm + Native Python) remains manual lifecycle only — use Docker sidecar with Inference Agent for managed GPU inference on Linux, or run native processes yourself and point CoreCube at the endpoints.
Inference Agent URL defaults
CoreCube recommends the Inference Agent URL automatically from the selected topology. You normally
do not edit it by hand; INFERENCE_AGENT_URL is only an advanced override for custom deployments.
| Setup | Default agent URL | Why |
|---|---|---|
| CoreCube running directly on the host | http://127.0.0.1:7444 | Server and agent share the same host namespace. |
| CoreCube in Docker + Docker sidecar | http://127.0.0.1:7444 | The managed agent runs in the CoreCube runtime namespace and starts sidecars from there. |
| CoreCube in Docker + CPU Native Python | http://127.0.0.1:7444 | CoreCube keeps the managed CPU runtime in the namespace it can control. |
| CoreCube in Docker + Apple MPS Native Python | http://host.docker.internal:7444 | Metal/MPS workers must run on the Mac host; Docker's Linux container cannot create that process. |
| External endpoint or manual lifecycle | None | CoreCube observes endpoint health but does not manage a local agent. |
When the URL is loopback, CoreCube can start the local Inference Agent automatically during
connection. When the URL points to host.docker.internal, the agent must run in the host namespace.
If it is not reachable, Admin shows a compatibility dialog with Reveal start command. CoreCube
generates the agent token for that command; you do not need to add
CORECUBE_INFERENCE_AGENT_TOKEN to .env unless you intentionally want to override the managed
token.
The Inference Agent reports host OS, Docker availability, Apple Silicon support, and Python 3.11+ availability. CoreCube uses that report to grey out or block unsupported combinations and to show a specific dialog when Python 3.11 or Docker is missing.
When to use manual lifecycle
Manual lifecycle does not mean “inference is off” or “CoreCube cannot see status.” It means CoreCube observes and routes to workers but does not start or stop them for you. Status in the Admin Console (nav indicator, overview, role health badges) still reflects whether embedding and reranker endpoints are reachable — independent of who ran the start command.
When manual is the only option
For these platform + mode combinations, Manual commands is the only valid lifecycle setting. Inference Agent is not offered or cannot manage workers on that target:
| Target | Why manual only |
|---|---|
| External → External endpoint | CoreCube does not own the process. It stores URLs, auth, and health checks only. Start, stop, and updates happen outside CoreCube. |
| Native Python on Linux (NVIDIA CUDA, AMD ROCm) | Managed native lifecycle is implemented for Apple MPS and CPU native Python only. On Linux, GPU native Python always uses manual lifecycle. |
If you run inference as a third-party service, as native Python on a Linux GPU box, or in environments where the Inference Agent cannot run on the host OS, you are in manual mode whether or not you would prefer agent-managed lifecycle.
When manual is optional — deliberate reasons to choose it
On Linux + Docker sidecar (CPU, CUDA, ROCm), Linux + CPU + Native Python, Apple MPS + Native Python, and macOS + CPU + Native Python, you can choose either Inference Agent or Manual commands. Pick manual when:
-
Security or compliance boundary — CoreCube (often running in Docker) should not drive container lifecycle on the host. Manual mode generates exact
docker pull,docker run, anddocker stopcommands but never executes them. You run commands in your own shell, CI, or configuration management instead of granting the app permission to pull images and restart containers. -
Existing orchestration — You already manage inference with docker-compose, systemd, Kubernetes, or similar. CoreCube is the consumer of embedding and reranker endpoints: configure URLs and health checks, start workers your way, and use Commands & logs as a reference for canonical run, stop, and update commands.
-
Air-gapped or restricted environments — No Inference Agent on the host, or policy forbids one. You still get command previews, endpoint monitoring, capability probes, and runtime status as long as workers are reachable at the configured URLs.
-
Maintenance windows and operator control — You want no Start button in Admin during a change window. Lifecycle is entirely operator-driven; CoreCube only observes health.
-
Bootstrap and troubleshooting — Stand up containers or native processes by hand, confirm endpoints respond, then optionally switch to Inference Agent later once everything works.
What you do in manual mode
- Configure platform, deployment mode, and role endpoint URLs under Inference → Deployment.
- Open Inference → Commands & logs and copy generated commands (
docker run,docker stop, native startup, pull, update, and rollback previews). - Run them on the host (SSH, terminal, or your automation).
- CoreCube probes those endpoints and shows worker status (Running, Stopped, Degraded) in the nav and overview. It does not start or stop workers for you.
Manual mode does not provide Start, Restart, or Stop in the Deployment tab, agent log streaming, or CoreCube-initiated image apply or rollback. Image update visibility may still show what is available; applying an update is your action via generated commands.
When to prefer Inference Agent instead
If you are on a supported managed target (Linux + Docker sidecar, Linux + CPU + Native Python, Apple Silicon Mac + Apple MPS + Native Python, or macOS + CPU + Native Python) and you can run the Inference Agent, manual mode is usually more work: extra copy-paste, no one-click start/stop, no log stream, and no coordinated restart.
Use Inference Agent for day-to-day local development and production Docker sidecars where the host agent is approved and running.
Additional constraints:
- External platform always uses External endpoint mode. Inference Agent lifecycle is never available for external deployments.
- Docker sidecar requires Docker on the Inference Agent host.
- On macOS, Docker sidecar inference is blocked because Linux containers cannot access Apple Metal/MPS, and CoreCube does not run managed inference sidecars in Docker on Mac. Use Native Python with Apple MPS (Metal) or CPU (host CPU workers) instead.
Recommended paths by host
| Host | Recommended managed setup |
|---|---|
| Linux (production) | See rows below by GPU |
| Linux (NVIDIA GPU) | NVIDIA CUDA → Docker sidecar → Inference Agent |
| Linux (AMD GPU, ROCm) | AMD ROCm → Docker sidecar → Inference Agent |
| Linux (CPU only) | CPU → Docker sidecar → Inference Agent |
| Apple Silicon Mac (dev) | Apple MPS → Native Python → Inference Agent (Metal path) |
| Apple Silicon Mac (dev) | CPU → Native Python → Inference Agent (host CPU workers) |
| Inference hosted elsewhere | External → External endpoint → Manual |
For Apple Silicon development from source, start the CoreCube stack with make deploy-local or
make dev, then configure Apple MPS → Native Python → Inference Agent and start workers from
Inference → Deployment.
Linux
| Platform | Deployment mode | Lifecycle | YES/NO |
|---|---|---|---|
| CPU | Docker sidecar | Inference Agent | YES |
| CPU | Docker sidecar | Manual | YES |
| CPU | Native Python | Inference Agent | YES |
| CPU | Native Python | Manual | YES |
| NVIDIA CUDA | Docker sidecar | Inference Agent | YES |
| NVIDIA CUDA | Docker sidecar | Manual | YES |
| NVIDIA CUDA | Native Python | Inference Agent | NO |
| NVIDIA CUDA | Native Python | Manual | YES |
| AMD ROCm | Docker sidecar | Inference Agent | YES (ROCm GPU + Docker required) |
| AMD ROCm | Docker sidecar | Manual | YES |
| AMD ROCm | Native Python | Inference Agent | NO |
| AMD ROCm | Native Python | Manual | YES |
| Apple MPS | Any | Any | NO (Apple MPS requires macOS) |
| External | External endpoint | Manual | YES |
| External | External endpoint | Inference Agent | NO |
Production Docker Compose on Linux typically uses CPU or NVIDIA CUDA with Docker sidecar. See Docker Deployment for GPU image tags and Compose examples.
For CPU-only native processes on Linux (without Docker sidecars), use CPU → Native Python →
Inference Agent. The agent runs approved host scripts (start-inference-cpu.sh /
stop-inference-cpu.sh) to launch embedding and reranker workers.
macOS
| Platform | Deployment mode | Lifecycle | YES/NO |
|---|---|---|---|
| Apple MPS | Native Python | Inference Agent | YES (Metal/MPS path) |
| Apple MPS | Native Python | Manual | YES |
| Apple MPS | Docker sidecar | Any | NO (Docker sidecar blocked on macOS) |
| CPU | Docker sidecar | Any | NO (Docker sidecar blocked on macOS) |
| CPU | Native Python | Inference Agent | YES (host CPU workers) |
| CPU | Native Python | Manual | YES |
| NVIDIA CUDA | Any | Any | NO (CUDA platform is not available on macOS) |
| AMD ROCm | Any | Any | NO (ROCm requires Linux) |
| External | External endpoint | Manual | YES |
On Apple Silicon, Docker Desktop runs Linux containers. Those containers cannot use the Mac GPU through Metal/MPS. CoreCube therefore runs inference as native macOS Python processes on the host while the rest of the stack (PostgreSQL, server, admin) stays in Docker.
Choose Apple MPS + Native Python when you want Metal acceleration. Choose CPU + Native Python when you want host CPU workers without MPS, or when you prefer the same platform family as a Linux CPU deployment.
The install script and standard Compose bundles do not enable MPS. Use a source checkout, start CoreCube with the local development stack, then enable Apple MPS inference from the Admin Console. See Install script → MPS.
When CoreCube runs in Docker on Mac, configure role endpoint URLs to reach host workers (for
example http://host.docker.internal:9440 for embedding). The Deployment tab applies these
defaults when you switch to a native Python target.
Common mistakes
Docker sidecar on macOS
Any platform with Docker sidecar on macOS is blocked before start. CoreCube shows a dialog explaining the issue and recommending Native Python with Inference Agent — either Apple MPS for Metal acceleration or CPU for host CPU workers.
Save the corrected platform and deployment mode before connecting the Inference Agent or clicking Start.
CPU + Native Python on Linux vs macOS
CPU + Native Python + Inference Agent is supported on both Linux and macOS. Do not confuse it with NVIDIA CUDA or AMD ROCm + Native Python, which remain manual-only on Linux.
On Linux servers with GPUs, prefer CUDA/ROCm → Docker sidecar → Inference Agent for managed GPU inference. Use CPU → Native Python → Inference Agent when you deliberately want native CPU workers on the host.
AMD ROCm on macOS
The AMD ROCm platform targets Linux servers with ROCm-capable AMD GPUs. On macOS, use Apple MPS + Native Python or CPU + Native Python for local development, CPU + Docker sidecar on Linux for CPU-only server paths, or an External endpoint.
External platform with Inference Agent
External inference is owned outside CoreCube. Lifecycle is always Manual — CoreCube stores connection URLs, auth references, and health probes only.
What happens when you click Start
- CoreCube checks the host OS reported by the Inference Agent (or the server when the agent is not connected yet).
- If the platform + mode + host combination is unsupported, a dialog explains why and how to fix it — no start request is sent.
- If the combination is valid and lifecycle is managed, the Inference Agent runs the approved start plan (Docker containers for sidecar mode, or native scripts for Apple MPS and CPU native Python).
- Status moves through Starting until embedding and reranker workers report ready. You can hover the Starting button and click Cancel to stop the launch.
On the Deployment tab, compatibility checks follow your current form selections (platform and deployment mode), not only the last saved configuration. Change to a supported combination, Save, then connect the Inference Agent and start workers.
If the Inference Agent URL is local loopback and the agent is not already running, CoreCube starts
it with the managed token before probing. If the URL points to host.docker.internal, Admin asks
you to reveal and run the host-agent start command first, then retry the connection.
Related pages
- Docker Deployment — production Compose, GPU images, Apple Silicon MPS
- Install script — Linux and macOS install notes
- Environment variables → Inference services —
INFERENCE_EMBEDDING_URLandINFERENCE_RERANKER_URL - Embedding & Chunking — chunking and indexing (separate from where inference runs)