daavy

We introduce two metrics, Characters per Opportunity (CPO) and Percentage Code Written (PCW). We believe CPO should be the definitive metric to benchmark different AI code assistant autocomplete capabilities and quality. This metric, unlike Acceptance Rate, is not susceptible to gameable confounding factors and is correlated with real tool value, so a higher CPO does mean a better product in terms of real value. We believe PCW should be the definitive metric to assess the value driven by a code assistant to the end developer, as it is a measure of the actual result of using the code assistant and is a relative metric that takes into account the fact that different developers code for different amounts of time to begin with. These metrics are related conceptually, but are not mathematically related in practice, as will be discussed. We make development decisions for Codeium based on CPO, and we currently have a CPO of 1.27, while we currently have a PCW of 44.6%, which means that 44.6% of new code being committed by developers using Codeium are actually from Codeium.

This paper introduces 26 guiding principles designed to streamline the process of querying and prompting large language models.