OpenAI, similar to numerous AI laboratories, believes that current AI benchmarks are flawed. They have announced their intention to address this issue through a new initiative aimed at improving these metrics.
Known as the OpenAI Pioneers Program, this initiative aims to develop assessments for AI models that "establish standards for excellence," according to how OpenAI described it. blog post .
As the speed of AI implementation quickens across various sectors, there is an increasing necessity to comprehend and enhance its influence globally," the firm went on to say in their statement. "Developing industry-focused evaluations is a method to more accurately mirror actual usage scenarios, aiding groups in gauging model effectiveness within realistic, critical settings.
As the recent controversy As both the crowdsourced benchmark of LM Arena and Meta’s Maverick model demonstrate, it has become challenging nowadays to accurately discern what sets apart various models. Numerous commonly utilized AI benchmarks evaluate capabilities through niche tasks, such as tackling doctoral-level mathematics questions. Conversely, some tests can be manipulated easily or fail to match up with general user expectations effectively.
Via the Pioneers Program, OpenAI aims to establish standards for particular sectors such as law, finance, insurance, medicine, and accounting. The organization mentioned that over the next few months, they will collaborate with "several businesses" to develop customized benchmarks which they plan to release publicly alongside sector-focused assessments.
OpenAI stated in the blog post, "The inaugural group will concentrate on startups aiming to establish the groundwork for the OpenAI Pioneers Program." They mentioned, "A select few startups have been chosen for this starting lineup, all engaged in significant, practical applications where artificial intelligence can create tangible real-world effects."
Participants in the program will get the chance to collaborate with OpenAI’s team to develop model enhancements through reinforcement fine-tuning, which is aimed at optimizing models for specific tasks, according to OpenAI.
The main issue revolves around whether the AI community will accept benchmarks developed with financial backing from OpenAI. Previously, OpenAI has provided funding for benchmark initiatives and created its own assessment tools. However, collaborating with clients to unveil AI tests could potentially be seen as crossing an ethical line.