Get started

Quickstart

Run your first portable evaluation against an agent in a few minutes.

1. Install

bash

py -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install "ecp-runtime==0.7.0" "ecp-sdk==0.7.0"

For framework demos, install the matching SDK extra:

bash

pip install "ecp-sdk[langchain]==0.7.0" langchain-openai
pip install "ecp-sdk[crewai]==0.7.0" crewai
pip install "ecp-sdk[pydanticai]==0.7.0" pydantic-ai
pip install "ecp-sdk[llamaindex]==0.7.0" llama-index llama-index-llms-openai llama-index-tools-yahoo-finance

2. Create a starter eval

bash

ecp init
ecp validate ecp_eval/manifest.yaml
ecp run --manifest ecp_eval/manifest.yaml --json

3. Run the flagship demo

The customer support demo checks final output, required tool calls, and evaluator-safe audit context.

bash

ecp validate examples/customer_support_demo/manifest.yaml
ecp run --manifest examples/customer_support_demo/manifest.yaml --report report.html

4. Run a framework demo

bash

ecp run --manifest examples/langchain_demo/manifest.yaml

Other manifests live in:

examples/plain_python_demo/manifest.yaml
examples/two_agent_demo/manifest.yaml
examples/crewai_demo/manifest.yaml
examples/pydantic_ai_demo/manifest.yaml
examples/llamaindex_demo/manifest.yaml

5. Native Pytest Integration

Instead of using ecp run, you can write native Python test assertions using our built-in Pytest fixture:

python

# test_agent.py
def test_customer_support(ecp_agent):
    result = ecp_agent.step("I need a refund")
    assert "refund" in result.get("public_output", "").lower()

Run it directly with your agent target:

bash

pytest test_agent.py --ecp-target="python agent.py"

6. Large Datasets (CSV/JSONL)

For large-scale evaluations, you can dynamically load datasets directly in your manifest.yaml instead of hardcoding steps:

yaml

scenarios:
  - name: "Bulk refund tests"
    dataset:
      type: "csv"
      source: "data/refund_queries.csv"
      input_column: "user_query"
      output_column: "expected_response"

7. Exporting to LangSmith

You can natively export all evaluation runs, including inputs and outputs, directly to LangSmith:

bash

ecp run --manifest examples/customer_support_demo/manifest.yaml --export langsmith

8. JSON output for CI

Print a JSON report:

bash

ecp run --manifest examples/customer_support_demo/manifest.yaml --json

Save a JSON report:

bash

ecp run --manifest examples/customer_support_demo/manifest.yaml --json-out report.json

By default, ecp run exits non-zero when checks fail. Use --no-fail-on-error when you want a report without failing the process.

9. Optional LLM judge

If your manifest uses llm_judge, set:

bash

$env:OPENAI_API_KEY="your_key_here"
$env:ECP_LLM_JUDGE_MODEL="gpt-4o-mini"
$env:ECP_LLM_JUDGE_TEMPERATURE="0"

10. Streamable HTTP

Start the HTTP agent:

bash

python examples/streamable_http_demo/agent.py

Run the HTTP-target manifest:

bash

ecp run --manifest examples/streamable_http_demo/manifest.yaml --json

11. Inspector

bash

npm run inspector

Open http://127.0.0.1:6274.

12. Conformance smoke test

For protocol implementers:

bash

ecp conformance --target "python examples/customer_support_demo/agent.py"

13. Execution boundaries

A hung agent must never pin a CI job. Bound a run from both directions:

bash

ecp run --manifest manifest.yaml --timeout 30 --max-duration 900

--timeout bounds a single RPC; --max-duration bounds the whole run. Both matter: an agent that answers just inside the per-RPC timeout on every step can still run for hours without ever tripping it.

When either limit trips, or the agent crashes, the run degrades instead of aborting. The failing step is recorded as failed, the rest of that scenario is skipped, and the next scenario runs against a fresh agent, so you still get a complete report:

Step 1: PASS
FAIL | step 2 did not complete (timeout): Agent response timed out after 3.0s
Scenario: Healthy scenario after the hang
Step 1: PASS
Run Complete. Passed: 2/4   exit_reason=timeout

Steps that never produced a result still count as failed checks, so a timeout cannot be mistaken for a pass.

14. Audit record

Every run emits a structured audit payload, embedded under audit in the JSON report and writable standalone:

bash

ecp run --manifest manifest.yaml --audit-out ecp_audit.json

It captures the run id, timestamps, the manifest SHA-256 digest, agent metadata, configured limits, per-step latency and exit_reason, token usage, and pass/fail totals:

json

{
  "exit_reason": "timeout",
  "manifest": { "digest": "sha256:c8a39311...", "target": "python agent.py" },
  "agent": { "name": "SupportAgent" },
  "totals": { "steps_planned": 4, "steps_executed": 2, "steps_failed": 1, "steps_skipped": 1 },
  "latency": { "max_ms": 812.4, "p95_ms": 790.1 },
  "usage": { "input_tokens": 4820, "output_tokens": 356, "total_tokens": 5176 }
}

Compare steps_planned with steps_executed to tell a clean run from one that degraded and measured a pass rate over fewer steps than intended.

Agents populate the token figures by returning usage from a step:

python

Result(
    public_output="Order A100 is eligible for a refund.",
    evaluation_context="Checked order A100 against the 30-day window.",
    usage={"input_tokens": 1204, "output_tokens": 88},
)

Omit usage entirely if your agent cannot observe token counts; the runtime distinguishes "not reported" from "reported as zero".

Notes

The current release line is 0.7.0.
New agents should use evaluation_context; private_thought remains a deprecated compatibility alias.
--timeout controls the per-RPC timeout for run and conformance. It overrides ECP_RPC_TIMEOUT; the default is 30 seconds.
--max-duration caps total run time. It overrides ECP_MAX_DURATION and is unset by default.
--audit-out writes the audit payload to its own file. The same payload is always present under audit in --json / --json-out output.