从 RL 视频教程到生产级 RL 工程技能
构建 Terminal-Bench 和 Harbor 的团队分享了大量来之不易的 RL 工程经验——他们如何打包任务、如何发现智能体作弊、如何把 rollout 变成训练数据。其中大部分都藏在冗长的会议讲座里。
其中三个讲座加起来大约一个半小时。每一分钟都值得看,但很难复用。下次构建环境时,你又得在时间轴上来回拖动,寻找记忆中的那一张幻灯片。
于是我们把“看视频”这件事交给了一支智能体团队。在 Eigent 中使用 Gemini 4,一份任务说明就把三个 Terminal-Bench 讲座变成了四个面向生产级 RL 环境工程的文件式技能,外加一份带来源链接、能说明每个观点出处的报告。
完整工作流一览
讲座由你挑选,之后一份任务说明就能一路做到智能体可复用的技能。每个阶段都会留下文件,方便你逐步检查。
| 阶段 | 做什么 | 得到什么 |
|---|---|---|
| 挑选 | 选择你想学习的 YouTube 讲座 | 任务说明中的一份简短视频链接列表 |
| 观看 | 下载字幕和画面帧,完整观看每个讲座 | 每个讲座带时间戳的大纲 |
| 分析 | 提取方法、洞见、报告的结果和警示 | 标注为 OBSERVED、INFERRED 或 UPDATED 的学习要点 |
| 对比 | 把讲座放在一起相互比较 | 它们的共识、分歧,以及如何互相补足空白 |
| 提炼 | 按实际能力对方法和洞见进行归类 | 每个技能一个 SKILL.md 和一个 evidence.md |
| 验证 | 检查结构、运行脚本、在新任务上试用技能 | 每个技能标记为 STRUCTURALLY CHECKED、EXECUTED 或 UNTESTED |
| 报告 | 构建 HTML 报告并在 127.0.0.1 上提供服务 | 嵌入的讲座、可点击的时间戳和标出空白的覆盖矩阵 |
| 复用 | 让协作智能体读取某个 SKILL.md | 在你自己的项目中开展生产级 RL 环境工程 |
下面一步步教你自己跑一遍,其中还包括我们在撰写任务说明时学到的一课。
你将得到什么
- 可复用的技能——每个都是一个
SKILL.md加一个evidence.md,其他智能体无需观看讲座即可直接应用。 - 一份带来源链接的 HTML 报告——在本地提供服务,嵌入每个讲座,附带时间戳大纲,学习要点可点击直接跳到对应时刻。
- 一个覆盖矩阵——哪个技能覆盖 RL 工作流的哪一部分,以及哪些领域没有任何讲座涉及。
- 一份诚实的记录——每条结论都标注为 OBSERVED、INFERRED 或 UPDATED,每个技能都标记为已执行、已做结构检查或未测试。
开始之前
- 在你的电脑上安装好 Eigent。
- 一个 Gemini API 密钥。
- 你希望智能体学习的 YouTube 链接。我们用的是这三个:
| 讲座 | 频道 | 时长 |
|---|---|---|
| Introducing Terminal-Bench: Evaluating LLM Agents in Realistic Terminal Settings | Anyscale (Ray Summit 2025) | 31:23 |
| Everything Is a Rollout - Alex Shaw + Ryan Marten | AI Engineer | 21:11 |
| Terminal-Bench 2.0: Launch + Q&A with the founders | Latent Space | 35:00 |
添加你的 Gemini API 密钥
- 打开 Settings → Models。
- 找到 Gemini 并打开其设置。
- 粘贴你的 API key。
- 将 model type 设为
gemini-4。 - 点击 Save。
选择 Gemini 4
- 在主屏幕上,打开输入框下方的 模型选择器。
- 选择 Gemini → gemini-4。
- 如有需要可设置思考强度,默认值即可。
粘贴任务说明
把下面的任务说明复制到 Cowork with 输入框中。换成你自己的视频链接,并把 Docker 一行设为 allowed 或 not allowed。
Watch the YouTube videos listed below and turn their methods and insights into reusable,
file-based agent skills. Then publish the results as a well-designed HTML report served from a
local web server.
Videos:
- https://www.youtube.com/watch?v=zt-U4lDdenY&t=1559s
- https://www.youtube.com/watch?v=jRCpXUjz4CI
- https://www.youtube.com/watch?v=5wo0mLlG0fk&t=1815s
Use only these videos as primary sources. Do not search for, substitute, or add other videos.
You may consult official documentation and the code repositories the videos link to or name,
but only to check or update details shown in the videos.
## Focus
Extract practical, repeatable methods. Where the videos cover them, include:
- Defining tasks, observations, actions, and termination conditions.
- Creating reproducible environments and reset behaviour.
- Building verifiers and reward functions.
- Detecting reward hacking and invalid successes.
- Collecting, scoring, and reviewing agent trajectories.
- Preparing data for agent training and evaluation.
Do not invent coverage for areas the videos don't address. List those areas as gaps instead.
Capture insights, not only procedures. For every method, record why the speakers use it, what
went wrong before they did, any results or numbers they reported, and the warnings they gave.
Treat named tools and frameworks (for example Harbor or Terminal-Bench) as one implementation
of a method. Record the underlying principle first and the tool-specific steps second.
## Permissions and limits
- Allowed: reading the video pages; downloading subtitles, transcripts, and the video files
for analysis; extracting frames; reading official docs and repositories named in the
videos; creating a local virtual environment and installing open-source packages into it;
running a local static web server bound to 127.0.0.1.
- Docker: [allowed / not allowed].
- Not allowed: model training, GPU jobs, paid hosted services, calls to paid LLM APIs, or
publishing anything to a public host. For trajectory collection tests, use a scripted,
rule-based, or random policy.
- Keep all files inside ./rl-env-skills/ in the current working directory. Delete downloaded
video files when finished.
- Do not ask for confirmation for the actions allowed above. Stop and report only if a required
action falls outside them.
- If a video is private, removed, region-locked, or has no usable audio or transcript, say so in
the report. Continue with the remaining videos, and do not replace it.
## Capability honesty
Before starting, record which relevant capabilities you have: video viewing, frame
extraction, transcript access, code execution, browser preview. If you cannot inspect video
visuals, work from transcripts and mark every visual claim as "not observed".
Never describe on-screen content you did not actually see.
## Analysis
For each video:
- Record title, channel, published date, duration, and view count with the date checked.
- Watch it in full. Inspect visual demonstrations, code, diagrams, and narration where your
capabilities allow.
- Build a timestamped outline of the main sections.
- Preserve timestamps for every important procedure, decision, and warning.
- List the key insights and takeaways: design decisions and the reasons behind them, lessons
learned, failure stories, reported results, and warnings. Give each one a timestamp.
- Label each claim OBSERVED (seen or heard in the video), INFERRED (your interpretation), or
UPDATED (adapted to current official docs; cite the doc URL and the date checked).
- Re-inspect unclear moments. Record unresolved details instead of guessing.
- Paraphrase. Quotes must be under 15 words and carry a timestamp. Do not reproduce transcripts.
Then note where the videos agree, where they differ, and where one fills a gap in another.
## Skills
Create 2–4 skills grouped by practical capability. A skill may draw from several videos.
Layout:
rl-env-skills/
<skill-name>/
SKILL.md # required
evidence.md # timestamps, OBSERVED/INFERRED/UPDATED labels, source per point
scripts/ templates/ # only if they materially help; scripts must be tested
report/
index.html # the HTML report (see below)
assets/ # extracted frames or screenshots used in the report, if any
REPORT.md
manifest.json
SKILL.md must begin with YAML frontmatter:
---
name: <kebab-case, matches folder name>
description: <1–2 sentences: what it does and when to use it, including trigger phrases>
---
The body must cover: prerequisites, required inputs, key insights (the principles and lessons
behind the procedure, each with its source and timestamp), procedure (numbered steps), expected
outputs, decision rules, failure recovery, observable success checks, and a tool-agnostic path
for projects that don't use the tools shown in the videos. Another agent should be able to
apply it to a new project without watching the videos.
A skill must not be only a wrapper around one tool's features. If you removed every
tool-specific command, the insights and decision rules should still stand on their own.
Keep these four things distinct throughout: task datasets, recorded trajectories, supervised
training examples, and RL policy optimisation. Never imply that exporting trajectories is RL
training.
## Validation
- Structural: frontmatter is present and valid, the name matches the folder, every referenced
file exists, every evidence timestamp falls within the source video's duration, and every
skill has a key insights section with at least three sourced insights.
- Execution: run every script you created. Where feasible, apply at least one skill end to end on
a small example the videos don't use (e.g. a text grid-world, or a "make this failing unit
test pass" task).
- Report each item as STRUCTURALLY CHECKED, EXECUTED (with command and result), or UNTESTED
(with reason).
## HTML report (report/index.html)
### Content, in this order
1. Header: report title, date checked, the agent's capability summary (had / lacked).
2. Summary: 3–5 sentences on what was produced and the main limitations. No marketing language.
3. Sources: one row per video (title, channel, published date, duration, views with date
checked, linked code or repository, access status).
4. One section per video:
- The original video embedded with a privacy-enhanced YouTube embed
(https://www.youtube-nocookie.com/embed/<VIDEO_ID>), with a 16:9 responsive frame.
- Link to the original.
- Timestamped outline of the video.
- Key insights and takeaways, each with a clickable timestamp and a label.
- Learning points: numbered, each with a clickable timestamp that seeks the embedded player
(YouTube IFrame Player API) and an OBSERVED / INFERRED / UPDATED label.
- Unresolved details.
- If embedding is blocked, show the video thumbnail linked to youtube.com/watch?v=<ID>
and state that embedding was unavailable.
5. How the videos compare: agreements, differences, and how each fills the others' gaps.
6. One section per skill:
- Name, absolute path to SKILL.md, one-sentence purpose, and when to use it.
- Key insights the skill carries, with their sources.
- Procedure summary (the key steps, not the whole file).
- Evidence: every supporting source point as a row — video, timestamp (clickable), label,
and what the skill took from it. Include doc URLs for UPDATED items.
- Validation status with the command and result if executed.
7. Coverage matrix: skills × the videos, showing which video supports which skill, plus
the focus areas no video covered.
8. Data distinctions: a short table defining task dataset, trajectory, supervised example, and
RL policy optimisation, and which of them the skills actually produce.
9. Limitations and untested items.
10. Three copy-paste reuse requests, each in a code block with a copy button.
### Visual design: light mode, in Google's design language
- Style only: do not use Google logos or wordmarks, or suggest the report is a Google product.
- Colours: page #FFFFFF; secondary surfaces #F8F9FA; borders #DADCE0; primary text #202124;
secondary text #5F6368; links and primary actions #1A73E8. Use the Google accent set
(blue #4285F4, red #EA4335, yellow #FBBC04, green #34A853) only for status and labels,
e.g. OBSERVED blue, INFERRED yellow tint with dark text, UPDATED green, UNTESTED red.
Text must meet WCAG AA contrast.
- Type: Google Sans / Google Sans Text if available from Google Fonts, otherwise Roboto; Roboto
Mono or Google Sans Code for code and timestamps. Clear scale: one H1, H2 per section, 16px
body, 1.5 line height, comfortable line length (about 70–80 characters).
- Layout: 8px spacing grid, max content width about 1100px, sticky left table of contents on
desktop that collapses on mobile, flat cards with 1px borders or a subtle Material elevation
shadow, 8–12px corner radius, plain tables with hairline row dividers.
- Avoid the generic "AI-generated" look: no gradients, purple/indigo themes, glassmorphism,
glowing effects, emoji used as icons, sparkle icons, oversized hero banners, decorative
blobs, or hype copy ("unlock", "supercharge", "seamless"). Prefer real content, tables, and
data over decoration.
- Technical: one self-contained HTML file with inline CSS and JS. Only external requests allowed:
Google Fonts and YouTube embed/API. Responsive down to 360px width with no horizontal
page scroll. Semantic HTML, keyboard-accessible controls, alt text on images.
### Serve and verify
1. Start a static server from rl-env-skills/report/ bound to 127.0.0.1, e.g.
`python3 -m http.server 8000 --bind 127.0.0.1`. If port 8000 is busy, use the next free port.
Keep the server running when you finish.
2. Do not rely on opening the file from disk: YouTube embeds often fail without an http origin.
3. Verify: the page returns HTTP 200; every embed loads or shows the fallback; timestamp links
seek the player; all anchors and file links resolve; no console errors. If you have a browser
tool, take a desktop and a 375px-wide screenshot and save them to report/assets/.
4. Report how you verified each item, and which checks you could not run.
## Deliverables
- REPORT.md: a plain-text version of the report's findings.
- manifest.json:
{ "date_checked": "...",
"videos": [{"url","title","channel","published","duration","views","access_status"}],
"skills": [{"name","path","sources":["<video id>", "..."],
"insights": <number of sourced insights>,
"validation":"executed|structural|untested"}],
"capabilities": {"video": bool, "frames": bool, "transcripts": bool, "code_exec": bool,
"browser_preview": bool},
"report": {"path": "<absolute path to index.html>", "preview_url": "http://127.0.0.1:<port>/",
"server_command": "...", "verified": [...], "not_verified": [...]} }
- Final reply: the preview link on its own line first
(http://127.0.0.1:<port>/), then the command to restart the server if it stops, then a short
summary of the videos, skills with absolute paths, validation results, and limitations.
这份任务说明中,有几部分起着关键作用:
- Use only these videos 让智能体只使用你选定的来源——不替换,也不额外添加。
- Capture insights, not only procedures 确保“为什么”与“怎么做”一起保留下来。下文会详细说明。
- Capability honesty 防止任何人描述一个根本没人看到过的视频画面。
- OBSERVED / INFERRED / UPDATED 标签 把事实、解读和经文档核对的更新区分开来。
- 数据区分 让任务数据集、轨迹、监督样本和 RL 策略优化保持独立,这样导出轨迹就永远不会被误认为是训练。
我们对任务说明做了哪些修改,以及原因
我们视频中的那次运行使用的是这份任务说明的早期版本。它做对了很多事。Gemini 4 看完了全部三个讲座,对照 Harbor 文档和代码仓库核对了细节,并构建了四个技能。每个技能的脚本都在一个讲座中没有用到的任务上完整跑通——修复一个名为 fix-ini-interpolator 的 Python 配置解析器。
但在阅读这些技能时,我们注意到一个问题:它们过度依赖 Harbor 的功能——任务文件夹结构、轨迹格式和 CLI 命令。而它们背后的洞见大多留在了报告里。
这些讲座里充满了洞见。报告本身就记录了这些:
- 脚手架和模型同样重要。 尽早取消卡住的命令带来约 5–10% 的提升,在一次运行中混用多个模型带来约 5–6% 的提升(Ray Summit 讲座,18:05–19:23)。
- 尽早为规模做规划。 一台笔记本最多同时运行约六个容器,这正是各团队转向云沙箱的原因(全部三个讲座)。
- 奖励作弊就是 RL 中的过拟合。 对待一个高得可疑的分数,要像对待低得可疑的验证损失一样(AI Engineer 讲座,06:35)。
一个只知道如何运行某个工具的技能,无法告诉你这种方法何时会失效,也无法告诉你在没有该工具的技术栈上该怎么做。所以我们做了四处修改:
- 洞见是一等产出。 现在每个视频章节和每个技能都会记录讲者的推理、经验教训、结果和警示——每条都带时间戳。
- 先原则,后工具。 Harbor 和 Terminal-Bench 只是应用某种方法的一种方式,而不是方法本身。
- 不做工具包装。 每个技能都需要一个关键洞见章节和一条工具无关的路径。去掉所有命令后,技能仍应能教会你一些东西。
- 验证会检查这一点。 每个技能必须包含至少三条有来源的洞见,否则无法通过结构检查。
我们还修正了一个小的不一致。早期的任务说明写的是“两个视频”,却列出了三个链接。Gemini 4 发现了这一点,在报告中做了说明,并分析了全部三个视频——而新的措辞适用于任意数量的视频。
让智能体开始工作
发送任务说明。Gemini 4 会规划工作并启动智能体。之后,整个运行会按照任务说明进行:
- 观看 每个讲座——字幕、画面帧、幻灯片和现场演示——并保留时间戳。
- 提炼 方法和洞见成为技能,每个技能都有自己的证据文件。
- 验证 技能,记录哪些经过检查、哪些已执行、哪些未测试。
- 生成报告 并在
127.0.0.1上提供服务。
在我们的运行中,智能体下载了全部三个讲座,提取了 25 个以上的关键帧,获取了完整的带时间戳字幕,并对照官方文档核对了 Harbor 的细节——然后按照任务说明的要求清理了下载的视频文件。
打开报告
报告在本地服务器上打开——在我们的运行中是 http://127.0.0.1:8000。你会看到:
- 能力——智能体具备什么、缺少什么,从视频观看到浏览器预览。
- 摘要与来源——每个讲座的频道、发布日期、时长、观看次数和关联的代码仓库。
- 每个讲座一个章节——以隐私增强型播放器嵌入视频,附带时间戳大纲和带标签的学习要点。
- 讲座对比——它们的共识、分歧,以及如何互相补足空白。
- 每个技能一个章节——用途、关键步骤、证据以及确切的验证命令。
- 局限性——哪些内容未经测试及其原因,比如云沙箱和付费模型 API。
解读覆盖矩阵
矩阵显示了哪个技能覆盖 RL 工作流的哪一部分——以及讲座留下了哪些空白。
| 重点领域 | Task Scaffolder | Verifier & Anti-Cheat | Rollout Collector | SFT Dataset Preparer |
|---|---|---|---|---|
| 定义任务、观测、动作和终止条件 | 主要 | 次要 | 主要 | - |
| 可复现的环境与重置 | 主要 | 次要 | 次要 | - |
| 构建验证器与奖励函数 | 次要 | 主要 | - | - |
| 检测奖励作弊与无效成功 | - | 主要 | 次要 | - |
| 收集、评分和审阅轨迹 | - | - | 主要 | 次要 |
| 为 SFT 训练和评估准备数据 | - | - | - | 主要 |
| 空白: RL 策略梯度数学与训练器内部机制 | 未覆盖 | 未覆盖 | 未覆盖 | 单独处理 |
| 空白: 步骤级稠密过程奖励塑形 | 未覆盖 | 仅结果奖励 | 未覆盖 | 未覆盖 |
这两行空白很重要。讲座提到了 SkyRL 和 Tinker 等训练器,但从未展示训练循环——所以这些技能也不会假装覆盖它。
让技能投入使用
每个技能都位于 rl-env-skills/ 下各自的文件夹中:
sandboxed-task-environment-scaffolder——搭建并验证可复现的容器化任务。verifier-and-anti-cheat-auditor——构建结果验证器,并审计其中的奖励作弊。rollout-trajectory-collector-and-debugger——运行沙箱 rollout,记录轨迹,并对失败进行分类。trajectory-to-sft-dataset-preparer——把经过验证的轨迹过滤成干净的 SFT 数据,并与 RL 训练严格分开。
让协作智能体读取其中一个,并交给它一项真实的任务,例如:
Use rl-env-skills/verifier-and-anti-cheat-auditor/SKILL.md to design a verifier
for my "make this failing unit test pass" task. Flag any way an agent could
pass without fixing the bug.
技能包含流程、决策规则和成功检查,证据文件则记录了出处——而有了新的任务说明,洞见也会一并保留。
打造你自己的版本
这份任务说明适用于任何一组技术讲座。替换视频链接和重点列表,其余部分保持不变:
- 保留洞见规则。 正是它们把一本工具手册变成来之不易的判断力。
- 保留诚实规则。 能力检查和结论标签让输出值得信赖。
- 保留验证步骤。 从未运行过的技能只是笔记,不是技能。
一个半小时的讲座,不必永远只是一个半小时的讲座。把它们交给你的智能体,并留住它们学到的东西。
鸣谢:使用 Gemini 4 和 Eigent 制作。讲座来自 Mike Merrill(Ray Summit 2025)、Alex Shaw 和 Ryan Marten(AI Engineer),以及 Latent Space 上的 Terminal-Bench 创始人,原视频请在 YouTube 观看。音乐:"Wallpaper" Kevin MacLeod (incompetech.com),采用 Creative Commons: By Attribution 4.0 许可。



