In April 2025, Google unveiled its most advanced artificial intelligence model to date, Gemini 2.5 Pro. Shortly after, the company released a technical report detailing the outcomes of its internal safety evaluations for this model. However, the report’s lack of comprehensive information has drawn criticism from AI safety experts, who argue that the absence of detailed safety data makes it challenging to assess potential risks associated with the model.
The Role of Technical Reports in AI Development
Technical reports are essential in the AI industry, offering insights into a model’s capabilities, limitations, and safety measures. They serve as a means for companies to demonstrate transparency and allow independent researchers to evaluate and verify the safety of AI systems. By providing detailed information, these reports help build trust and facilitate collaborative efforts to enhance AI safety.
Google’s Approach to Safety Reporting
Unlike some of its competitors, Google adopts a unique approach to safety reporting. The company publishes technical reports only after a model has transitioned from the experimental phase to a more stable release. Additionally, Google separates evaluations of dangerous capabilities into distinct audits, which are not included in the standard technical reports. This methodology has led to concerns about the timeliness and comprehensiveness of the information provided.
Expert Criticisms of the Gemini 2.5 Pro Report
The technical report for Gemini 2.5 Pro has been met with disappointment from several AI safety experts. Peter Wildeford, co-founder of the Institute for AI Policy and Strategy, described the report as very sparse and lacking sufficient information. He emphasized that the delay in releasing the report, weeks after the model’s public availability, hinders the ability to verify Google’s adherence to its safety commitments. This opacity makes it difficult to assess the model’s safety and security comprehensively.
Thomas Woodside, co-founder of the Secure AI Project, echoed these sentiments. While he acknowledged the importance of releasing a report for Gemini 2.5 Pro, he expressed skepticism about Google’s dedication to providing timely and thorough safety evaluations. Woodside highlighted that the last published results of dangerous capability tests were in June 2024 for a model announced in February of the same year. He also noted the absence of a report for Gemini 2.5 Flash, a smaller and more efficient model announced recently, with Google stating that a report is coming soon.
The Frontier Safety Framework and Its Omission
A notable omission in the Gemini 2.5 Pro report is any reference to Google’s Frontier Safety Framework (FSF). Introduced in the previous year, the FSF aims to identify and mitigate potential AI capabilities that could cause significant harm. The absence of FSF-related information in the report raises questions about the framework’s implementation and effectiveness in the context of Gemini 2.5 Pro.
Industry-Wide Transparency Challenges
Google is not alone in facing scrutiny over transparency in AI safety reporting. Other major AI developers have also been criticized for providing insufficient information. For instance, Meta released a similarly brief safety evaluation for its new Llama 4 open models, and OpenAI chose not to publish any report for its GPT-4.1 series. This trend underscores a broader issue within the AI industry regarding the need for more detailed and timely safety disclosures.
Regulatory Commitments and Expectations
The lack of comprehensive safety reporting is particularly concerning given the commitments tech companies have made to regulators. Two years prior, Google assured the U.S. government that it would publish safety reports for all significant AI models. The current deficiencies in reporting raise questions about the company’s compliance with these assurances and its overall commitment to AI safety transparency.
The Importance of Transparent AI Safety Reporting
Transparent and detailed safety reporting is crucial for the responsible development and deployment of AI technologies. It enables stakeholders, including researchers, policymakers, and the public, to understand potential risks and collaborate on mitigation strategies. As AI systems become increasingly integrated into various aspects of society, the need for openness and accountability in safety evaluations becomes even more critical.
Conclusion
The release of Google’s Gemini 2.5 Pro technical report has highlighted significant gaps in AI safety transparency. The criticisms from experts underscore the necessity for more comprehensive and timely safety evaluations. As the AI industry continues to evolve, it is imperative for companies to prioritize transparency and accountability to ensure the safe and ethical development of AI technologies.
