借助 Claude 3.5 Sonnet 在 SWE-bench Verified 上抬高标杆
Raising the bar on SWE-bench Verified with Claude 3.5 Sonnet
段落导航
Our latest model, the upgraded Claude 3.5 Sonnet, achieved 49% on SWE-bench Verified, a software engineering evaluation, beating the previous state-of-the-art model's 45%.
我们最新的模型,即升级版 Claude 3.5 Sonnet,在一项软件工程评测 SWE-bench Verified 上取得了 49% 的成绩,超过了此前最先进模型的 45%。
This post explains the "agent" we built around the model, and is intended to help developers get the best possible performance out of Claude 3.5 Sonnet.
本文解释了我们围绕该模型构建的「智能体(agent)」,旨在帮助开发者从 Claude 3.5 Sonnet 中取得尽可能最佳的性能。
SWE-bench is an AI evaluation benchmark that assesses a model's ability to complete real-world software engineering tasks.
SWE-bench 是一个 AI 评测基准(benchmark),用于评估模型完成真实世界软件工程任务的能力。
Specifically, it tests how the model can resolve GitHub issues from popular open-source Python repositories.
具体来说,它测试模型如何解决来自热门开源 Python 仓库的 GitHub issue。
For each task in the benchmark, the AI model is given a set up Python environment and the checkout (a local working copy) of the repository from just before the issue was resolved.
对于基准中的每个任务,AI 模型会获得一个搭建好的 Python 环境,以及该仓库在对应 issue 被解决之前的 checkout(本地工作副本)。
The model then needs to understand, modify, and test the code before submitting its proposed solution.
随后,模型需要理解、修改并测试代码,然后提交其提出的解决方案。
Each solution is graded against the real unit tests from the pull request that closed the original GitHub issue.
每个解决方案都会依据关闭原始 GitHub issue 的那个 pull request 中的真实单元测试进行评分。
This tests whether the AI model was able to achieve the same functionality as the original human author of the PR.
这检验了 AI 模型能否实现与 PR 原作者相同的功能。
SWE-bench doesn't just evaluate the AI model in isolation, but rather an entire "agent" system.
SWE-bench 并不是孤立地评估 AI 模型,而是评估整个「智能体」系统。
In this context, an "agent" refers to the combination of an AI model and the software scaffolding around it.
在此语境下,「智能体」指的是 AI 模型与其周边软件脚手架(scaffolding)的组合。
This scaffolding is responsible for generating the prompts that go into the model, parsing the model's output to take action, and managing the interaction loop where the result of the model's previous action is incorporated into its next prompt.
这个脚手架负责生成输入模型的提示词、解析模型输出以采取行动,以及管理交互循环——在该循环中,模型上一个动作的结果会被纳入其下一个提示词。
The performance of an agent on SWE-bench can vary significantly based on this scaffolding, even when using the same underlying AI model.
即使使用相同的底层 AI 模型,智能体在 SWE-bench 上的表现也可能因这套脚手架而有显著差异。
There are many other benchmarks for the coding abilities of Large Language Models, but SWE-bench has gained in popularity for several reasons:
针对大语言模型(LLM)编程能力的基准还有很多,但 SWE-bench 之所以流行,有几个原因:
It uses real engineering tasks from actual projects, rather than competition- or interview-style questions;
它使用的是来自真实项目的真实工程任务,而不是竞赛或面试风格的题目;
It is not yet saturated—there’s plenty of room for improvement.
它尚未饱和——仍有很大的改进空间。
No model has yet crossed 50% completion on SWE-bench Verified (though the updated Claude 3.5 Sonnet is, at the time of writing, at 49%);
尚未有模型在 SWE-bench Verified 上跨过 50% 完成率(不过截至写作时,更新后的 Claude 3.5 Sonnet 为 49%);
It measures an entire "agent", rather than a model in isolation.
它衡量的是整个“智能体”,而不是孤立的一个模型。
Open-source developers and startups have had great success in optimizing scaffoldings to greatly improve the performance around the same model.
开源开发者和初创公司已经在优化脚手架方面取得了巨大成功,从而在相同模型上大幅提升了性能。
Note that the original SWE-bench dataset contains some tasks that are impossible to solve without additional context outside of the GitHub issue (for example, about specific error messages to return).
请注意,原始 SWE-bench 数据集包含一些任务,如果不借助 GitHub issue 之外的额外上下文(例如,关于要返回的具体错误消息),就无法解决。
SWE-bench-Verified is a 500 problem subset of SWE-bench that has been reviewed by humans to make sure they are solvable, and thus provides the most clear measure of coding agents' performance.
SWE-bench-Verified 是 SWE-bench 的一个包含 500 道问题的子集,已经过人工审查以确保它们可解,因此为编码智能体的性能提供了最清晰的衡量标准。
This is the benchmark to which we’ll refer in this post.
这就是我们在本文中将要引用的基准测试。
Our design philosophy when creating the agent scaffold optimized for updated Claude 3.5 Sonnet was to give as much control as possible to the language model itself, and keep the scaffolding minimal.
我们在为更新后的 Claude 3.5 Sonnet 创建优化的智能体脚手架时,设计理念是尽可能把控制权交给语言模型本身,并让脚手架保持最小化。
The agent has a prompt, a Bash Tool for executing bash commands, and an Edit Tool, for viewing and editing files and directories.
该智能体有一个提示词、一个用于执行 bash 命令的 Bash Tool,以及一个用于查看和编辑文件和目录的 Edit Tool。
We continue to sample until the model decides that it is finished, or exceeds its 200k context length.
我们持续采样,直到模型决定自己已经完成,或者超过其 200k 上下文长度。
This scaffold allows the model to use its own judgment of how to pursue the problem, rather than be hardcoded into a particular pattern or workflow.
这种脚手架允许模型自行判断如何推进问题,而不是被硬编码到特定模式或工作流中。
The prompt outlines a suggested approach for the model, but it’s not overly long or too detailed for this task.
提示词为模型勾勒了一个建议的方法,但针对这项任务,它并不过长或过于详细。
The model is free to choose how it moves from step to step, rather than having strict and discrete transitions.
模型可以自由选择如何从一个步骤进入下一个步骤,而不是有严格且离散的转换。
If you are not token-sensitive, it can help to explicitly encourage the model to produce a long response.
如果你对 token 不敏感,明确鼓励模型产生长回复可能会有所帮助。
The following code shows the prompt from our agent scaffold:
以下代码展示了我们智能体脚手架中的提示词:
<uploaded_files>
{location}
</uploaded_files>
I've uploaded a python code repository in the directory {location} (not in /tmp/inputs). Consider the following PR description:
<pr_description>
{pr_description}
</pr_description>
Can you help me implement the necessary changes to the repository so that the requirements specified in the <pr_description> are met?
I've already taken care of all changes to any of the test files described in the <pr_description>. This means you DON'T have to modify the testing logic or any of the tests in any way!
Your task is to make the minimal changes to non-tests files in the {location} directory to ensure the <pr_description> is satisfied.
Follow these steps to resolve the issue:
1. As a first step, it might be a good idea to explore the repo to familiarize yourself with its structure.
2. Create a script to reproduce the error and execute it with `python <filename.py>` using the BashTool, to confirm the error
3. Edit the sourcecode of the repo to resolve the issue
4. Rerun your reproduce script and confirm that the error is fixed!
5. Think about edgecases and make sure your fix handles them as well
Your thinking should be thorough and so it's fine if it's very long.The model's first tool executes Bash commands.
模型的第一个工具(tool)会执行 Bash 命令。
The schema is simple, taking only the command to be run in the environment.
该 schema 很简单,只接收要在环境中运行的命令。
However, the description of the tool carries more weight.
然而,工具的说明更具分量。
It includes more detailed instructions for the model, including escaping inputs, lack of internet access, and how to run commands in the background.
其中包含给模型的更详细指令,包括转义输入、无法访问互联网,以及如何后台运行命令。
Next, we show the spec for the Bash Tool:
接下来,我们展示 Bash Tool 的规范(spec):
{
"name": "bash",
"description": "Run commands in a bash shell\n
* When invoking this tool, the contents of the \"command\" parameter does NOT need to be XML-escaped.\n
* You don't have access to the internet via this tool.\n
* You do have access to a mirror of common linux and python packages via apt and pip.\n
* State is persistent across command calls and discussions with the user.\n
* To inspect a particular line range of a file, e.g. lines 10-25, try 'sed -n 10,25p /path/to/the/file'.\n
* Please avoid commands that may produce a very large amount of output.\n
* Please run long lived commands in the background, e.g. 'sleep 10 &' or start a server in the background.",
"input_schema": {
"type": "object",
"properties": {
"command": {
"type": "string",
"description": "The bash command to run."
}
},
"required": ["command"]
}
}The model's second tool (the Edit Tool) is much more complex, and contains everything the model needs for viewing, creating, and editing files.
模型的第二个工具(Edit Tool)复杂得多,包含模型查看、创建和编辑文件所需的一切。
Again, our tool description contains detailed information for the model about how to use the tool.
同样,我们的工具说明中包含给模型的详细信息,说明如何使用该工具。
We put a lot of effort into the descriptions and specs for these tools across a wide variety of agentic tasks.
我们在各种智能体任务中,为这些工具的说明和规范投入了大量精力。
We tested them to uncover any ways that the model might misunderstand the spec, or the possible pitfalls of using the tools, then edited the descriptions to preempt these problems.
我们测试它们,以发现模型可能误解规范的方式,或使用这些工具可能存在的陷阱,然后修改说明以预先避免这些问题。
We believe that much more attention should go into designing tool interfaces for models, in the same way that a large amount of attention goes into designing tool interfaces for humans.
我们认为,应该把更多注意力投入到为模型设计工具接口上,就像大量注意力被投入到为人类设计工具接口上一样。
The following code shows the description for our Edit Tool:
以下代码展示了我们的 Edit Tool 的说明:
{
"name": "str_replace_editor",
"description": "Custom editing tool for viewing, creating and editing files\n
* State is persistent across command calls and discussions with the user\n
* If `path` is a file, `view` displays the result of applying `cat -n`. If `path` is a directory, `view` lists non-hidden files and directories up to 2 levels deep\n
* The `create` command cannot be used if the specified `path` already exists as a file\n
* If a `command` generates a long output, it will be truncated and marked with `<response clipped>` \n
* The `undo_edit` command will revert the last edit made to the file at `path`\n
\n
Notes for using the `str_replace` command:\n
* The `old_str` parameter should match EXACTLY one or more consecutive lines from the original file. Be mindful of whitespaces!\n
* If the `old_str` parameter is not unique in the file, the replacement will not be performed. Make sure to include enough context in `old_str` to make it unique\n
* The `new_str` parameter should contain the edited lines that should replace the `old_str`",
...One way we improved performance was to "error-proof" our tools.
我们提升性能的一种方式,是对我们的工具进行“防错”处理。
For instance, sometimes models could mess up relative file paths after the agent had moved out of the root directory.
例如,有时在智能体移出根目录后,模型可能会弄乱相对文件路径。
To prevent this, we simply made the tool always require an absolute path.
为防止这种情况,我们干脆让该工具始终要求绝对路径。
We experimented with several different strategies for specifying edits to existing files and had the highest reliability with string replacement, where the model specifies old_str to replace with new_str in the given file.
我们试验了多种为现有文件指定编辑的不同策略,其中可靠性最高的是字符串替换,即由模型指定 old_str,在给定文件中用 new_str 将其替换。
The replacement will only occur if there is exactly one match of old_str.
只有当 old_str 恰好有一处匹配时,才会执行替换。
If there are more or fewer matches, the model is shown an appropriate error message for it to retry.
如果匹配数多于或少于一处,会向模型显示相应的错误信息,以便它重试。
The spec for our Edit Tool is shown below:
我们的 Edit Tool 的规范如下所示:
...
"input_schema": {
"type": "object",
"properties": {
"command": {
"type": "string",
"enum": ["view", "create", "str_replace", "insert", "undo_edit"],
"description": "The commands to run. Allowed options are: `view`, `create`, `str_replace`, `insert`, `undo_edit`."
},
"file_text": {
"description": "Required parameter of `create` command, with the content of the file to be created.",
"type": "string"
},
"insert_line": {
"description": "Required parameter of `insert` command. The `new_str` will be inserted AFTER the line `insert_line` of `path`.",
"type": "integer"
},
"new_str": {
"description": "Required parameter of `str_replace` command containing the new string. Required parameter of `insert` command containing the string to insert.",
"type": "string"
},
"old_str": {
"description": "Required parameter of `str_replace` command containing the string in `path` to replace.",
"type": "string"
},
"path": {
"description": "Absolute path to file or directory, e.g. `/repo/file.py` or `/repo`.",
"type": "string"
},
"view_range": {
"description": "Optional parameter of `view` command when `path` points to a file. If none is given, the full file is shown. If provided, the file will be shown in the indicated line number range, e.g. [11, 12] will show lines 11 and 12. Indexing at 1 to start. Setting `[start_line, -1]` shows all lines from `start_line` to the end of the file.",
"items": {
"type": "integer"
},
"type": "array"
}
},
"required": ["command", "path"]
}
}In general, the upgraded Claude 3.5 Sonnet demonstrates higher reasoning, coding, and mathematical abilities than our prior models, and the previous state-of-the-art model.
总体而言,升级后的 Claude 3.5 Sonnet 在推理、编码和数学能力上均优于我们此前的模型以及之前的最先进模型。
It also demonstrates improved agentic capabilities: the tools and scaffolding help put those improved abilities to their best use.
它还展现出更强的智能体能力:工具与脚手架有助于把这些提升后的能力发挥到最佳。
Model
模型
Claude 3.5 Sonnet (new)
Claude 3.5 Sonnet(新)
Previous SOTA
此前的 SOTA
Claude 3.5 Sonnet (old)
Claude 3.5 Sonnet(旧)
Claude 3 Opus
Claude 3 Opus
SWE-bench Verified score
SWE-bench Verified 分数
49%
49%
45%
45%
33%
33%
22%
22%
For running the benchmark, we used the SWE-Agent framework as a foundation for our agent code.
为运行该基准测试,我们以 SWE-Agent 框架作为智能体代码的基础。
In our logs below, we render the agent's text output, tool calls, and tool responses as THOUGHT, ACTION, and OBSERVATION, even though we don’t constrain the model to a fixed ordering.
在下面的日志中,我们把智能体的文本输出、工具调用和工具响应分别渲染为 THOUGHT、ACTION 和 OBSERVATION,尽管我们并没有把模型限制为固定的顺序。
The code blocks below will walk through a typical case of the Sonnet 3.5 solving a SWE-bench problem.
下面的代码块将逐步展示 Sonnet 3.5 解决一个 SWE-bench 问题的典型过程。
In this first block, you can see part of the initial prompt given to the model, with {pr_description} filled in with the real value from a SWE-bench task.
在第一个代码块中,你可以看到提供给模型的初始提示词的一部分,其中 {pr_description} 已填入来自某个 SWE-bench 任务的真实值。
Importantly, this task contains steps to reproduce the issue, which will give the model a valuable starting point to investigate.
重要的是,这个任务包含了复现该问题的步骤,这将为模型提供一个有价值的调查起点。
<uploaded_files>
/repo
</uploaded_files>
I've uploaded a python code repository in the directory /repo (not in /tmp/inputs). Consider the following PR description:
<pr_description>
linear_model.RidgeClassifierCV's Parameter store_cv_values issue
#### Description
Parameter store_cv_values error on sklearn.linear_model.RidgeClassifierCV
#### Steps/Code to Reproduce
import numpy as np
from sklearn import linear_model as lm
#test database
n = 100
x = np.random.randn(n, 30)
y = np.random.normal(size = n)
rr = lm.RidgeClassifierCV(alphas = np.arange(0.1, 1000, 0.1), normalize = True,
store_cv_values = True).fit(x, y)
#### Expected Results
Expected to get the usual ridge regression model output, keeping the cross validation predictions as attribute.
#### Actual Results
TypeError: __init__() got an unexpected keyword argument 'store_cv_values'
lm.RidgeClassifierCV actually has no parameter store_cv_values, even though some attributes depends on it.The model responds and first uses the Edit Tool to view the repository structure.
模型作出响应,首先使用 Edit Tool 查看仓库结构。
You can see the model's text output and tool call arguments under THOUGHT and ACTION, and part of the tool's output under OBSERVATION:
你可以在 THOUGHT 和 ACTION 下看到模型的文本输出和工具调用参数,在 OBSERVATION 下看到工具输出的一部分:
THOUGHT:
I'll help you implement the changes to satisfy the PR description. Let's follow the steps you outlined.
1. First, let's explore the repo structure:
ACTION (str_replace_editor):
command:
view
path:
/repo
OBSERVATION:
Here's the files and directories up to 2 levels deep in /repo, excluding hidden items:
/repo
/repo/PULL_REQUEST_TEMPLATE.md
/repo/ISSUE_TEMPLATE.md
/repo/sklearn
/repo/sklearn/decomposition
/repo/sklearn/kernel_ridge.py
/repo/sklearn/model_selection
/repo/sklearn/covariance
/repo/sklearn/cluster
/repo/sklearn/base.py
/repo/sklearn/dummy.py
/repo/sklearn/naive_bayes.py
...Now that the model has a better understanding of the repository structure, it uses the Edit Tool to create a new script that it will use to reproduce the issue and test its fix:
现在模型对仓库结构有了更好的理解,它使用 Edit Tool 创建一个新脚本,用来复现该问题并测试其修复:
THOUGHT:
Based on the repository structure, we'll need to modify the Ridge Classifier in the `sklearn/linear_model` directory.
Let's create a script to reproduce the error:
ACTION (str_replace_editor):
command:
create
path:
/repo/reproduce_error.py
file_text:
import numpy as np
from sklearn import linear_model as lm
#test database
n = 100
x = np.random.randn(n, 30)
y = np.random.normal(size = n)
rr = lm.RidgeClassifierCV(alphas=np.arange(0.1, 1000, 0.1), normalize=True,
store_cv_values=True).fit(x, y)
OBSERVATION:
File created successfully at: /repo/reproduce_error.pyThe model then uses the Bash Tool to execute the script it wrote, and successfully reproduces the issue from the task:
随后模型使用 Bash Tool 执行它编写的脚本,并成功复现了任务中的问题:
THOUGHT:
Let's verify that we can reproduce the error:
ACTION (bash):
command:
python3 /repo/reproduce_error.py
OBSERVATION:
Traceback (most recent call last):
File "/repo/reproduce_error.py", line 10, in <module>
store_cv_values=True).fit(x, y)
TypeError: __init__() got an unexpected keyword argument 'store_cv_values'From here on, the model uses the Edit Tool to change the source code in the repository and reruns its script to verify whether the change has resolved the issue:
从这里开始,模型使用 Edit Tool 修改仓库中的源代码,并重新运行其脚本,以验证该修改是否解决了问题:
THOUGHT:
The error is reproducible. Looking at the code, I can see that the issue is that while the `RidgeClassifierCV` class inherits from `_BaseRidgeCV`, it fails to pass the `store_cv_values` parameter from its constructor through to the base class constructor.
Let's fix this:
ACTION (str_replace_editor):
command:
str_replace
path:
/repo/sklearn/linear_model/ridge.py
old_str:
def __init__(self, alphas=(0.1, 1.0, 10.0), fit_intercept=True,
normalize=False, scoring=None, cv=None, class_weight=None):
super(RidgeClassifierCV, self).__init__(
alphas=alphas, fit_intercept=fit_intercept, normalize=normalize,
scoring=scoring, cv=cv)
self.class_weight = class_weight
new_str:
def __init__(self, alphas=(0.1, 1.0, 10.0), fit_intercept=True,
normalize=False, scoring=None, cv=None, class_weight=None,
store_cv_values=False):
super(RidgeClassifierCV, self).__init__(
alphas=alphas, fit_intercept=fit_intercept, normalize=normalize,
scoring=scoring, cv=cv, store_cv_values=store_cv_values)
self.class_weight = class_weightIn this particular example, the model worked for 12 steps before deciding that it was ready to submit.
在这个特定示例中,模型工作了 12 步,才决定可以提交。
The task's tests then ran successfully, verifying that the model's solution addressed the problem.
随后该任务的测试成功运行,验证了模型的解决方案解决了问题。
Some tasks took more than 100 turns before the model submitted its solution; in others, the model kept trying until it ran out of context.
有些任务在模型提交解决方案前耗时超过 100 轮;在另一些任务中,模型不断尝试,直到耗尽上下文窗口。
From reviewing attempts from the updated Claude 3.5 Sonnet compared to older models, updated 3.5 Sonnet self-corrects more often.
通过审查更新后的 Claude 3.5 Sonnet 与更早模型相比的尝试,更新后的 3.5 Sonnet 更常自我纠正。
It also shows an ability to try several different solutions, rather than getting stuck making the same mistake over and over.
它还展现出尝试多种不同解决方案的能力,而不是陷入一遍又一遍犯同样错误的境地。
SWE-bench Verified is a powerful evaluation, but it’s also more complex to run than simple, single-turn evals.
SWE-bench Verified 是一个强大的评测(eval),但运行起来也比简单的单轮评测更复杂。
These are some of the challenges that we faced in using it—challenges that other AI developers might also encounter.
这些是我们在使用它时面临的一些挑战——其他 AI 开发者也可能遇到这些挑战。
Duration and high token costs.
耗时和高 token 成本。
The examples above are from a case that was successfully completed in 12 steps.
上面的示例来自一个在 12 步内成功完成的案例。
However, many successful runs took hundreds of turns for the model to resolve, and >100k tokens.
然而,许多成功运行需要模型花费数百轮才能解决,且消耗超过 100k token。
The updated Claude 3.5 Sonnet is tenacious: it can often find its way around a problem given enough time, but that can be expensive;
更新后的 Claude 3.5 Sonnet 非常坚韧:只要有足够时间,它往往能找到绕过问题的方法,但这可能代价高昂;
Grading.
评分。
While inspecting failed tasks, we found cases where the model behaved correctly, but there were environment setup issues, or problems with install patches being applied twice.
在检查失败任务时,我们发现有些案例中模型行为正确,但存在环境设置问题,或者安装补丁被应用两次的问题。
Resolving these systems issues is crucial for getting an accurate picture of an AI agent's performance.
解决这些系统问题对于准确了解 AI 智能体的性能至关重要。
Hidden tests.
隐藏测试。
Because the model cannot see the tests it's being graded against, it often “thinks” that it has succeeded when the task actually is a failure.
由于模型看不到用于评分对照的测试,它常常“认为”自己已经成功,而任务实际上是失败的。
Some of these failures are because the model solved the problem at the wrong level of abstraction (applying a bandaid instead of a deeper refactor).
其中一些失败是因为模型在错误的抽象层级上解决了问题(打补丁而不是进行更深入的重构)。
Other failures feel a little less fair: they solve the problem, but do not match the unit tests from the original task.
其他失败则感觉有点不太公平:它们解决了问题,但与原始任务的单元测试不匹配。
Multimodal.
多模态(multimodal)。
Despite the updated Claude 3.5 Sonnet having excellent vision and multimodal capabilities, we did not implement a way for it to view files saved to the filesystem or referenced as URLs.
尽管更新后的 Claude 3.5 Sonnet 拥有出色的视觉与多模态能力,我们没有实现让它查看保存到文件系统或通过 URL 引用的文件的方式。
This made debugging certain tasks (especially those from Matplotlib) especially difficult, and also prone to model hallucinations.
这使调试某些任务(尤其是来自 Matplotlib 的任务)变得尤为困难,也容易导致模型幻觉。
There is definitely low-hanging fruit here for developers to improve upon—and SWE-bench has launched a new evaluation focused on multi-modal tasks.
对开发者而言,这里肯定有唾手可得的改进空间——而且 SWE-bench 已经推出了一项专注于多模态任务的新评测。
We look forward to seeing developers achieve higher scores on this eval with Claude in the near future.
我们期待在不久的将来看到开发者用 Claude 在这项评测上取得更高分数。
The upgraded Claude 3.5 Sonnet achieved 49% on SWE-bench Verified, beating the previous state-of-the-art (45%), with a simple prompt and two general purpose tools.
升级后的 Claude 3.5 Sonnet 在 SWE-bench Verified 上达到了 49%,击败了此前的最优水平(45%),凭借一个简单的提示词(prompt)和两个通用工具。
We feel confident that developers building with the new Claude 3.5 Sonnet will quickly find new, better ways to improve SWE-bench scores over what we've initially demonstrated here.
我们确信,使用新版 Claude 3.5 Sonnet 进行构建的开发者将很快找到新的、更好的方法,把 SWE-bench 分数提升到超过我们这里初步展示的水平。
Erik Schluntz optimized the SWE-bench agent and wrote this blog post.
Erik Schluntz 优化了 SWE-bench 智能体并撰写了这篇博客文章。
Simon Biggs, Dawn Drain, and Eric Christiansen helped implement the benchmark.
Simon Biggs、Dawn Drain 和 Eric Christiansen 帮助实现了该基准。
Shauna Kravec, Dawn Drain, Felipe Rosso, Nova DasSarma, Ven Chandrasekaran, and many others contributed to training Claude 3.5 Sonnet to be excellent at agentic coding.
Shauna Kravec、Dawn Drain、Felipe Rosso、Nova DasSarma、Ven Chandrasekaran 以及许多其他人,为训练 Claude 3.5 Sonnet 使其在智能体编程方面表现卓越做出了贡献。