# Overmind setup These are instructions for you, an AI coding agent, to set the human up on Overmind. Read them once, then act on them. The human who handed you this document wants you to do the setup and execute workflows, not to explain it back to them. There are 2 prerequisites to start this: 1. **MCP config for Overmind** ```json { ... "overmind": { "url": "https://api.overmindlab.ai/api/mcp/" } } ``` **DO NOT forget the slash in the end for the url** 2. **PyPI package overmind** - Ensure python version is alteast 3.10, if the version is lower, report the conflict to the user and stop. Use any of the following to install overmind, depending on what is available on the host machine: - pip: `pip install --upgrade "overmind>=0.1.104"` - uv: `uv pip install "overmind>=0.1.104"` - Poetry or anything else: ` -m pip install "overmind>=0.1.104"` 2.1 The overmind SDK comes with a set of loaded skills, run the command below using the coding agent you are being called within. ```bash overmind init --ide= ``` Ensure that the project's `python version is >= 3.10`, and `overmind --version is >= 0.1.104` If the project's Python is older than 3.10 or the install conflicts with the environment, report the exact conflict and stop; do not change the project's supported Python range. The command above should add a `overmind.toml` file, which will also act as a source of truth for users' resources in Overmind. Ask the user to reload their coding agent if needed to enable MCP, otherwise they will not be able to invoke the MCP. Now we can move onto getting started with overmind. # USING OVERMIND Overmind can be used in 2 ways, onboarding from a codebase or onboarding using a dataset, ASK the USER which one they want to continue with, if it hasn't already been configured. > Use Overmind MCP to issue an OVERMIND_API_KEY ## Onboarding using codebase or repository Overmind has skills to scan through a respository with agentic codebase written either in Python or Typescript/Javascript > If codebase is any other Language, avoid runnning `overmind chassis` command and just use the coding agent's output file and use it to run `overmind sync` See (overmind://prompts/onboaring-using-codebase) Prompt on the Overmind MCP - Use the `/overmind setup` skill to start scanning the codebase. ## Onboarding using a dataset > If this is a data-only project (uploaded files, no repository) You will ask the human to provide a dataset which the human wants to use for finetuning aka training a custom model. Then you will analyse the shape of the dataset and a few rows to check if the dataset has everything Overmind needs to finetune a model for the human. Then you invoke the overmind sdk to upload this dataset to overmind backend. ______________________________________________________________________ Overmind has the following features | Feature | Reference | | ----------------- | --------------------------------------------------------------------- | | Agent Scanning | [Code scanning and eval authoring](#code-scanning-and-eval-authoring) | | Observability | [Telemetry](#telemetry) | | Dataset | [Datasets](#datasets) | | Evaluations | [Evals](#evals) | | Finetuning | [Finetuning](#finetuning) | | Optimiser | [Optimiser](#optimiser) | | Model Backtesting | [Backtesting](#backtesting) | | Inference | [Inference](#inference) | ## Code scanning and eval authoring > Scans your codebase and builds a context map of how AI is used across the codebase - Run `/overmind setup` locally. MCP cannot scan or edit the repo. - Judge criteria go in decorator `expectations=[...]`; deterministic checks go in the toml `eval_matrix`. Run `overmind sync` after either. - No repo, or a rule no catalog check covers: `create_eval_matrix_row`, then `create_eval_set(activate=True)`. Rows are never edited: delete and recreate. - Never invent capability ids. Read them from `overmind://capabilities/{capability}`. ## Telemetry > Helps you add observability to your agentic stack. - Python: use the Overmind SDK. `get_instrumentation_plan`, apply each `required_decorator` verbatim. - Any other language with OpenTelemetry already enabled: no SDK and no plan. Add Overmind as one more OTLP/HTTP protobuf exporter beside the existing ones, never replacing them: - Endpoint `https://api.overmindlab.ai/api/v1/traces`, header `x-api-key: `. - Resource attribute `overmind.capability.id` (the only key that binds spans to a capability). - Span attributes: `overmind.span.type=entry_point` and `overmind.unit_kind=run` on the root span of a run, `overmind.delivery=true` on the span carrying the result, `conversation.id` on every span of a multi-turn exchange. - No OpenTelemetry in a non-Python codebase: report telemetry as not set up. Do not introduce OpenTelemetry for the user. - Verify (both paths): get the user's approval for one run (real or smoke), stamp `conversation.id` with a unique value, run, `query_traces(session=...)` (expect one root), `verify_instrumentation(trace_id)`. - Investigate: `investigate-capability` prompt, or `inspect_capability_health` and `query_failures`, then follow trace links. - Signals: `list_signals`, `set_signal_subscription`. - Use the skill `/overmind-observability` to run observability operations ## Datasets > Clients upload datasets to Overmind. Workshop turns them into versioned cells for training, evaluation or exploration - Intent is the user's choice: `train`, `eval` or `explore`. Never infer it from rows. Set it with `update_dataset`. - Upload files with the CLI: `overmind dataset upload FILE... --project-id ID --json`. Re-run the identical command to resume. Use `--json-rows-field` for JSON wrappers. - Landing is not proof of correct ingestion: compare counts and sample values with the local file. - Transform with a retained Python package: `overmind dataset pipeline-upload`, `save_dataset_pipeline`, `validate_dataset_pipeline`, preview, then full run. Use the `author-dataset-transformation` prompt. Never upload derived rows as a new source. - From production data: `create_dataset_from_traces` or `create_dataset_from_llm_calls`. - Export: `overmind dataset export DATASET --json`. - Use the skill `/overmind-datasets` to run dataset operations ## Evals > Creates a set of eval metrics relevant to your codebase or dataset and triggers evaluations - Use the `prepare-evaluation` and `evaluate-change` prompts. - Flow: `check_evaluation_readiness`, `run_evaluation`, `compare_evaluations`. - The dataset must be `eval` intent. The capability's active eval set grades by default; do not recreate it. - Report per-metric deltas and failed or missing rows, not just the average. - Use the skill `/overmind-evaluations` to run evals operations ## Finetuning > Creates a custom finetuned language model trained on a dataset in Overmind, validated using the evaluations created for that dataset or codebase - Needs a `train` dataset and, for chat models, user-confirmed eval metrics. Decision models are exempt. - Flow: `get_model_catalog`, `check_finetune_readiness`, `prepare_training_data`, `estimate_finetune`, approval, then `start_finetune` or `start_decision_job` with a stable `request_key`. - Monitor with `get_job` and `inspect_training_progress`. Cancel with `cancel_finetune`. Never relaunch a job that already exists. - Data-only projects: use the `develop-model-from-data` prompt and `save_training_plan` first. - Split by group before fitting, measure a baseline, and never select on final-role data. - Use the skill `/overmind-training` to run finetuning operations ## Optimiser > Runs an agent in your infrastructure to experiment with different prompts and models and find which works best. All results are backed by evaluations - Needs an `eval` dataset cell. - Flow: `check_optimizer_readiness`, `start_optimizer`, then run `overmind optimise start -e EXPERIMENT && overmind optimise next` in the repo and do each returned action. - Stop at `COMPLETE`. Apply the winning diff only when authorised. - Use the skill `/overmind-optimiser` to run optimiser operations ## Backtesting > Tries different models and replays your LLM spans to see which model works best - Use the `compare-models` prompt: `check_optimizer_readiness` and `start_optimizer` with `mode=model_comparison` and 1-5 model ids, then `inspect_optimizer_result`. - Or replay stored LLM spans: `overmind backtest --models a,b --since 7d`. - Use the skill `/overmind-optimiser` to run backtesting operations ## Inference > Runs inference on the finetuned model through Overmind. - Successful training deploys automatically. Read `overmind://deployments/{deployment}` for readiness and traffic. - `run_inference` only on a `ready` deployment, with a stable `request_key`. - Go live with `set_active_model` after approval and poll the `model_activation` job. Use `get_model_swap_prompt` for the code rollout; the human applies it. - `retry_deployment` only for a failed deployment. - Use the skill `/overmind-inference` to run inference operations ## Integrations - Import external traces: `connect-traces` prompt, `inspect_connectors`, `configure_connector` (propose, get approval, then `confirm_mapping=true`), `sync_connector`. - Credentials are entered by the human in their terminal, never in chat. - Use the skill `/overmind-integrations` to run integrations operations ______________________________________________________________________ Help: > If the user wants to see what overmind can do and how to invoke overmind, these are some of the commands that you need to print. 1. Overmind can help you fine-tune a model using your dataset and/or codebase. 1. Overmind can run inference on your fine-tuned models. 1. Overmind can help you transform your raw data into a structured, fine-tunable dataset. 1. Overmind can help you run evaluations and compare changes in your agents. 1. Overmind can help you add tracing to your LLM agents and monitor your agent behavior using signals. 1. Overmind can help you run experiments in your agentic codebase to see how to improve it.