Skip to content

refactor(core): rename Evaluator to Grader across codebase#1111

Draft
christso wants to merge 1 commit intomainfrom
feat/1109-rename-evaluator-grader
Draft

refactor(core): rename Evaluator to Grader across codebase#1111
christso wants to merge 1 commit intomainfrom
feat/1109-rename-evaluator-grader

Conversation

@christso
Copy link
Copy Markdown
Collaborator

Closes #1109

Summary

This is the hard rename from Evaluator to Grader across the codebase.

  • Renames core interfaces, classes, registries, loaders, exports, tests, and directory layout from Evaluator* / evaluators/ to Grader* / graders/
  • Removes the old defineCodeJudge / CodeJudgeHandler / runCodeJudge-style aliases and related evaluator-era exports
  • Updates CLI, Studio, docs, examples, plugins, and AGENTS guidance to use grader terminology
  • Preserves YAML kind strings, scores[].type, evaluation/, and the evaluate() API

Breaking change

This PR intentionally hard-removes the old TypeScript symbol names and deep import paths. There are no backcompat aliases.

Old symbol New symbol
Evaluator (interface) Grader
EvaluatorKind GraderKind
EVALUATOR_KIND_VALUES GRADER_KIND_VALUES
EvaluatorRegistry GraderRegistry
FieldAccuracyEvaluator FieldAccuracyGrader
SkillTriggerEvaluator SkillTriggerGrader
CodeEvaluator CodeGrader
LlmGraderEvaluator LlmGrader
ToolTrajectoryEvaluator ToolTrajectoryGrader
TokenUsageEvaluator TokenUsageGrader
LatencyEvaluator LatencyGrader
CostEvaluator CostGrader
InlineAssertEvaluator InlineAssertGrader
CompositeEvaluator CompositeGrader
ExecutionMetricsEvaluator ExecutionMetricsGrader
DeterministicAssertionEvaluator DeterministicAssertionGrader
defineCodeJudge (@agentv/eval) defineCodeGrader
Import path @agentv/core/.../evaluators/... .../graders/...

Verification

  • bun run build
  • bun run typecheck
  • bun run lint
  • bun run test
  • bun run validate:examples
  • Live eval before/after comparison using examples/features/execution-metrics/evals/dataset.eval.yaml --test-id simple-thresholds-pass confirmed identical scores[].type output (["execution-metrics"]) on origin/main and this branch
  • Studio smoke-check confirmed the eval detail view now renders Grader Scores with no stale evaluator label in visible UI copy

Live eval artifacts

  • Before: /home/christso/.copilot/session-state/264152e9-2410-40ef-9068-b7b31da6998e/files/issue-1109-live-eval/main/index.jsonl
  • After: /home/christso/.copilot/session-state/264152e9-2410-40ef-9068-b7b31da6998e/files/issue-1109-live-eval/worktree/index.jsonl

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@cloudflare-workers-and-pages
Copy link
Copy Markdown

Deploying agentv with  Cloudflare Pages  Cloudflare Pages

Latest commit: e3416de
Status: ✅  Deploy successful!
Preview URL: https://95c5fa6a.agentv.pages.dev
Branch Preview URL: https://feat-1109-rename-evaluator-g.agentv.pages.dev

View logs

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

refactor(core): rename Evaluator → Grader across codebase (hard break)

1 participant