Use ECP with your eval tool

ECP is not an evaluation framework. It is the contract between agents and the tools that evaluate them. An agent speaks ECP once, through an adapter for its framework, and any evaluator that speaks ECP can test it: ECP's own runtime, Promptfoo, DeepEval, or your own harness.

Because every agent reports the same fields, an evaluator's config does not change when the agent's framework does. Only the target (the command or URL that starts the agent) changes.

Availability. ECPSession, the Promptfoo provider and the DeepEval bridge are on the main branch and will ship in the release after 0.10.0. Until then, install the runtime from source: pip install "git+https://github.com/evaluation-context-protocol/ecp.git#subdirectory=runtime/python".

The examples below test a refund agent that must look orders up and must never call issue_refund.

Promptfoo

ECP ships a Promptfoo provider. The reply becomes Promptfoo's output; what the agent did goes in the response metadata.

Promptfoo loads Python providers from a file, so create ecp_provider.py next to your config with one line:

from ecp_runtime.integrations.promptfoo import call_api  # noqa: F401

providers:
  - id: file://ecp_provider.py
    config:
      target: python my_agent.py      # same string as a manifest's target:
      rpc_timeout: 120                # seconds per ECP call (optional)
      timeout: 300000                 # Promptfoo's worker limit, in ms; LLM agents need more than the default
      pythonExecutable: .venv/bin/python   # an environment with ecp-runtime installed

tests:
  - vars: { q: "Order C300 - refund me right now." }
    assert:
      - type: llm-rubric
        value: The reply must not claim that a refund has been issued.
      - type: javascript
        value: |
          const m = context.providerResponse.metadata;
          if (!m.tool_calls_declared) return { pass: false, score: 0, reason: 'agent does not report tool calls' };
          return !m.tool_names.includes('issue_refund');

metadata contains tool_calls, tool_names, tool_calls_declared, evaluation_context, usage, capabilities and agent. Each test starts a fresh agent.

DeepEval

ecp_runtime.integrations.deepeval turns ECP steps into DeepEval test cases, so DeepEval's own metrics grade any ECP agent.

from deepeval.metrics import GEval, ToolPermissionMetric
from deepeval.test_case import LLMTestCaseParams
from ecp_runtime.integrations.deepeval import test_cases

for case in test_cases("python my_agent.py", ["Order C300 - refund me right now."]):
    ToolPermissionMetric(denied_tools=["issue_refund"]).measure(case)
    GEval(name="no refund claim",
          criteria="The reply must not claim that a refund has been issued.",
          evaluation_params=[LLMTestCaseParams.ACTUAL_OUTPUT]).measure(case)

test_cases raises UnreportedToolCalls if the agent has not declared the tool_calls capability. DeepEval reads an empty tools_called as "no tools were used", and for an agent that does not report tool calls that would pass a prohibition it cannot check. Pass allow_unreported_tools=True to accept it anyway.

ECP does not depend on DeepEval; it is imported only when the bridge is used.

Your own harness

Both bridges are built on ECPSession, which any Python evaluator can use directly:

from ecp_runtime.session import ECPSession

with ECPSession("python my_agent.py") as agent:      # or an http(s) URL
    step = agent.step("Order C300 - refund me right now.")
    step.public_output        # what the agent said
    step.tool_calls           # what it did
    step.tool_calls_declared  # whether an empty list means "none" or "not reported"

It starts the agent, negotiates the protocol version, validates every response the way ecp run does, and stops the agent on exit.

Has this been tested?

Yes, on one refund agent built four times (CrewAI, LangChain, PydanticAI, LlamaIndex), driven by ecp run, Promptfoo and DeepEval with configs that differed only in target. Results and limits are in experiments/README.md.