Comment by climike

Exactly! Number of turns, average tokens to achieve a task using your CLI, as well as average number of characters being returned per CLI command alongside other metrics: all important to both users and agents! I am working on allowing to accurately capture this at www.cliwatch.com! Feel free to request an example eval suite for a list of tasks you want to achieve with your CLI