What metrics do you track when building agents?

When building autonomous agents, tracking the right metrics is essential to ensure performance, efficiency, and reliability. Both models emphasized Task Success Rate as a foundational metric, measuring how accurately and reliably the agent completes its assigned tasks. This includes metrics like accuracy, F1 score, recall, precision, and completion rate for tasks. Additionally, Error Rate was highlighted as a critical measure to identify and reduce failures, ensuring the agent operates robustly. Both models also stressed the importance of Latency/Response Time, particularly in real-time applications where delays can degrade user experience. Metrics like end-to-end latency, P50/P90/P99 latency, and time to first token are crucial for maintaining responsiveness. Resource Utilization was another shared focus, with both models emphasizing the need to monitor computational costs (CPU/GPU usage, memory footprint) and scalability to ensure the agent operates efficiently. In terms of Reliability, both models agreed on tracking availability (uptime), error rates, and robustness against edge cases. Lastly, User Satisfaction was highlighted as a key indicator of the agent's effectiveness, with metrics like CSAT, NPS, and repeat usage rate providing insights into user experience.

Where the models diverged, Mistral Small 4 provided a more granular breakdown, introducing specific metrics like Functional Correctness (e.g., exact match, BLEU score), Safety & Alignment (e.g., toxicity score, hallucination rate), and Operational Observability (e.g., log quality, anomaly detection). It also added long-term impact metrics such as ROI and customer retention impact, which Command R did not mention. Command R, on the other hand, emphasized Adaptability and Flexibility (e.g., transitions between tasks, speed of adaptation) and Risk Assessment (e.g., hazard avoidance, failure management), areas not explicitly covered by Mistral Small 4.

To summarize, prioritize Task Success Rate, Latency, Resource Utilization, Reliability, and User Satisfaction as core metrics. Depending on your agent's specific use case, expand into areas like Safety, Observability, Adaptability, and Long-Term Impact for a more comprehensive evaluation.