Kitaru
An open-source replay-based evaluation toolkit for AI agents that records or imports production traces as sessions and reruns them against model, prompt, or code changes.
Where it fits
Kitaru is listed in Evaluation Frameworks, Observability Frameworks. Use its original project documentation to confirm the features and setup for your intended workflow.
Find tools for evaluating agent outputs and workflows. Start with representative tasks and a written success definition so a score corresponds to a result your users actually need.
Before you choose
Use these category-level questions when reviewing this resource. They do not assert that this project supports a particular feature.
- Does the test set reflect real user tasks?
- Is task completion measured separately from answer style?
- Can the same examples be replayed after a change?
Sources & project links
About this listing
Imported directory snapshot · 2026-10-26. The description and classification come from the earlier directory, adapted from awesome-ai-agents; current project behavior has not been independently tested.
Pricing, licensing, supported models and hosting requirements may change. Check the original project before installing or paying. Inclusion is not an endorsement or a performance rating.
Historical repository popularity
The old directory displayed 300 GitHub stars. Its original observation date is unknown. This is retained as historical context, not a live count or quality score.