{"version":"1.0","generated_at":"2026-09-27T14:58:12.440037","id":2637,"slug":"ai-benchmarks-have-a-trust-problem-and-google-wants-to-fix-it","title":"AI基准测试存在信任问题，谷歌想要解决它","summary":"谷歌DeepMind推出双盲测试方法，利用密码学防止AI模型提前看到测试题，以解决基准测试污染问题。","abstract":"AI基准测试存在信任问题，谷歌想要解决它 谷歌DeepMind推出双盲测试方法，利用密码学防止AI模型提前看到测试题，以解决基准测试污染问题。 谷歌DeepMind推出双盲测试方法，利用密码学防止AI模型提前看到测试题，以解决基准测试污染问题。 原贴提到：Google Deepmind wants to use a cryptographic method to stop AI models fr 来源：the-decoder.com 谷歌DeepMind希望使用一种密码学方法，阻止AI模型提前看到测试问题。与新加坡AI安全研究所及其他合作伙伴的一个试点项目，对Gemini模型进行双盲测试。如果应试者提前知道问题，即使满分也毫无价值。谷歌DeepMind用这个例子来描述评估AI模型的一个核心问题：基准测试污染。如果模型在训练时已经看过测试问题，那么结果的可信度就有限。为了防止这种情况，谷歌DeepMind表示，它正在启动首个针对专有前沿AI模型的双盲评估。外部测试被锁在密码学“盒子”中，因此模型以后无法利用这些问题专门针对测试进行优化。在试点中，谷歌正在测试来自Gemini Flash Lite系列的一个模型，使用机密基准。DeepMind表示，高度敏感的外部评估过去需要妥协。要么评估者交出他们的测试提示，这会让模型提供商提前看到问题；要么提供商交出模型权重，冒着知识产权风险。最近一个体现这种困境的例子是，Anthropic的Fable 5在ARC-AGI基准上的评估被推迟，因为这家AI公司对其最强模型执行30天数据保留政策。双盲评估旨在消除这种权衡。谷歌使用来自谷歌云机密计…","access_level":"member","access_label":"免费阅读","access_mode":"full","is_preview":false,"canonical_url":"https://opc.beizhux.com/content/2637/ai-benchmarks-have-a-trust-problem-and-google-wants-to-fix-it","html_url":"https://opc.beizhux.com/content/2637/ai-benchmarks-have-a-trust-problem-and-google-wants-to-fix-it","json_url":"https://opc.beizhux.com/content/2637/ai-benchmarks-have-a-trust-problem-and-google-wants-to-fix-it.json","published_at":"2026-08-28T21:15:10","updated_at":"2026-09-27T14:08:54","category":{"slug":"ai-business","name":"AI小生意项目库"},"source":{"site":"the-decoder.com","author":"Maximilian Schreiner","url":"https://the-decoder.com/ai-benchmarks-have-a-trust-problem-and-google-wants-to-fix-it/"},"tags":["AI基准测试","AI评估","Google DeepMind","加密","加密计算","双盲测试","双盲评估","基准污染","基准测试","密码学","数据隐私","机密计算"],"topics":[{"slug":"ai-side-business","name":"AI副业","url":"https://opc.beizhux.com/topics/ai-side-business","reason":"这篇内容命中「项目、小生意、变现」等主题信号。"},{"slug":"ai-tools","name":"AI工具","url":"https://opc.beizhux.com/topics/ai-tools","reason":"这篇内容命中「模型」等主题信号。"}],"keywords":["AI小生意项目库","AI副业","AI工具","工具","自动化","模型","Cursor","Claude","工作流","副业","创业","项目"],"questions":[{"question":"AI基准测试存在信任问题，谷歌想要解决它主要讲什么？","answer":"谷歌DeepMind推出双盲测试方法，利用密码学防止AI模型提前看到测试题，以解决基准测试污染问题。"},{"question":"这篇文章最值得关注的要点是什么？","answer":"谷歌DeepMind推出双盲测试方法，利用密码学防止AI模型提前看到测试题，以解决基准测试污染问题。；原贴提到：Google Deepmind wants to use a cryptographic method to stop AI models fr；来源：the-decoder.com"},{"question":"这篇文章和哪些AI专题相关？","answer":"它适合放在AI副业、AI工具专题里阅读。 关联原因：这篇内容命中「项目、小生意、变现」等主题信号。；这篇内容命中「模型」等主题信号。"},{"question":"阅读这篇文章建议先理解哪些关键词？","answer":"建议先理解AI工具、工具、自动化、模型、Cursor这些关键词，再结合正文判断工具、机会或风险是否值得进入自己的工作流。"}],"terms":[{"slug":"ai-tools-term","name":"AI工具","definition":"在AI觉醒星球里，「AI工具」属于「AI工具」方向。围绕AI工具、模型能力、自动化流程和真实使用场景做整理，优先关注能提升效率、降低成本和创造新交付的工具。 不只看工具热度，更看它能不能进入真实流程。","topic_slug":"ai-tools","topic_name":"AI工具","topic_title":"AI工具：模型、插件、自动化与实战场景","topic_url":"https://opc.beizhux.com/topics/ai-tools","topic_path":"/topics/ai-tools","url":"https://opc.beizhux.com/glossary/ai-tools-term","path":"/glossary/ai-tools-term","json_url":"https://opc.beizhux.com/glossary/ai-tools-term.json"},{"slug":"tools","name":"工具","definition":"在AI觉醒星球里，「工具」属于「AI工具」方向。围绕AI工具、模型能力、自动化流程和真实使用场景做整理，优先关注能提升效率、降低成本和创造新交付的工具。 不只看工具热度，更看它能不能进入真实流程。","topic_slug":"ai-tools","topic_name":"AI工具","topic_title":"AI工具：模型、插件、自动化与实战场景","topic_url":"https://opc.beizhux.com/topics/ai-tools","topic_path":"/topics/ai-tools","url":"https://opc.beizhux.com/glossary/tools","path":"/glossary/tools","json_url":"https://opc.beizhux.com/glossary/tools.json"},{"slug":"automation","name":"自动化","definition":"在AI觉醒星球里，「自动化」属于「AI工具」方向。围绕AI工具、模型能力、自动化流程和真实使用场景做整理，优先关注能提升效率、降低成本和创造新交付的工具。 不只看工具热度，更看它能不能进入真实流程。","topic_slug":"ai-tools","topic_name":"AI工具","topic_title":"AI工具：模型、插件、自动化与实战场景","topic_url":"https://opc.beizhux.com/topics/ai-tools","topic_path":"/topics/ai-tools","url":"https://opc.beizhux.com/glossary/automation","path":"/glossary/automation","json_url":"https://opc.beizhux.com/glossary/automation.json"},{"slug":"ai-model","name":"模型","definition":"在AI觉醒星球里，「模型」属于「AI工具」方向。围绕AI工具、模型能力、自动化流程和真实使用场景做整理，优先关注能提升效率、降低成本和创造新交付的工具。 不只看工具热度，更看它能不能进入真实流程。","topic_slug":"ai-tools","topic_name":"AI工具","topic_title":"AI工具：模型、插件、自动化与实战场景","topic_url":"https://opc.beizhux.com/topics/ai-tools","topic_path":"/topics/ai-tools","url":"https://opc.beizhux.com/glossary/ai-model","path":"/glossary/ai-model","json_url":"https://opc.beizhux.com/glossary/ai-model.json"},{"slug":"cursor","name":"Cursor","definition":"在AI觉醒星球里，「Cursor」属于「AI工具」方向。围绕AI工具、模型能力、自动化流程和真实使用场景做整理，优先关注能提升效率、降低成本和创造新交付的工具。 不只看工具热度，更看它能不能进入真实流程。","topic_slug":"ai-tools","topic_name":"AI工具","topic_title":"AI工具：模型、插件、自动化与实战场景","topic_url":"https://opc.beizhux.com/topics/ai-tools","topic_path":"/topics/ai-tools","url":"https://opc.beizhux.com/glossary/cursor","path":"/glossary/cursor","json_url":"https://opc.beizhux.com/glossary/cursor.json"},{"slug":"claude","name":"Claude","definition":"在AI觉醒星球里，「Claude」属于「AI工具」方向。围绕AI工具、模型能力、自动化流程和真实使用场景做整理，优先关注能提升效率、降低成本和创造新交付的工具。 不只看工具热度，更看它能不能进入真实流程。","topic_slug":"ai-tools","topic_name":"AI工具","topic_title":"AI工具：模型、插件、自动化与实战场景","topic_url":"https://opc.beizhux.com/topics/ai-tools","topic_path":"/topics/ai-tools","url":"https://opc.beizhux.com/glossary/claude","path":"/glossary/claude","json_url":"https://opc.beizhux.com/glossary/claude.json"},{"slug":"workflow","name":"工作流","definition":"在AI觉醒星球里，「工作流」属于「Agent工作流」方向。聚合Agent、AI编程、Claude Code、Cursor、工作流自动化等内容，帮助你把智能体能力落到真实任务链路。 把智能体当成任务系统，而不是聊天窗口。","topic_slug":"agent-workflow","topic_name":"Agent工作流","topic_title":"Agent工作流：智能体、AI编程与自动化协作","topic_url":"https://opc.beizhux.com/topics/agent-workflow","topic_path":"/topics/agent-workflow","url":"https://opc.beizhux.com/glossary/workflow","path":"/glossary/workflow","json_url":"https://opc.beizhux.com/glossary/workflow.json"},{"slug":"side-business","name":"副业","definition":"在AI觉醒星球里，「副业」属于「AI副业」方向。筛选AI副业、AI小生意、项目认知和可收费交付案例，关注普通人可以验证、复制和沉淀的方法。 从真实痛点出发，先做能收费的小验证。","topic_slug":"ai-side-business","topic_name":"AI副业","topic_title":"AI副业：小生意、项目机会与可收费交付","topic_url":"https://opc.beizhux.com/topics/ai-side-business","topic_path":"/topics/ai-side-business","url":"https://opc.beizhux.com/glossary/side-business","path":"/glossary/side-business","json_url":"https://opc.beizhux.com/glossary/side-business.json"}],"preview_hidden_sections":[],"related_items":[{"id":3092,"title":"三分之二的 IT 领导者报告 AI 成果，但很少有人会为此打断 CEO 的假期","url":"https://opc.beizhux.com/content/3092/two-thirds-of-it-leaders-report-ai-results-but-few-would-interrupt-the-ceo-s-vacation-over-them","summary":"AI 泡沫争论的关键问题是 AI 实验室收入能否支撑数据中心建设。Azeem Azhar 分享轶事：约 160 名 IT 副总裁中三分之二有可衡量 AI 成果，但仅约八人认为成果好到能打断 CEO 暑假。企业 AI 回报仍缓慢且难在全经济层面记录，许多公司转向开放权重模型，BCG 调查显示约 70% CEO 有动力美…","access_level":"member"},{"id":3097,"title":"Nvidia 的 SoL-Pi 系统通过优化 harness 将近乎一半的编码 agent token 用量削减","url":"https://opc.beizhux.com/content/3097/nvidia-s-sol-pi-system-cuts-coding-agent-token-usage-nearly-in-half-by-optimizing-the-harness","summary":"Nvidia 论文提出 SoL-Pi，自动优化编码 agent 的控制层 harness。在 EdgeBench 上，token 用量下降 44.7% 到 49%，性能大致持平；按当前 API 价格估算，每小时可节省 4.36 到 13.50 美元。","access_level":"member"},{"id":3084,"title":"使用指标 API 新增拉取请求审查阶段","url":"https://opc.beizhux.com/content/3084/usage-metrics-api-adds-pull-request-review-stages","summary":"GitHub 在企业与组织仓库级 Copilot 使用指标中新增 pull_request_review_times 数组，把 PR 从准备审查到合并拆成三个阶段，分别给出中位数和 P90 耗时，帮助团队定位审查等待发生在哪里。","access_level":"member"}],"usage_policy":{"summarizable":true,"preferred_url":"https://opc.beizhux.com/content/2637/ai-benchmarks-have-a-trust-problem-and-google-wants-to-fix-it","private_fields_excluded":true}}