AI Evaluations Clearly Explained in 50 Minutes (Real Example) | Hamel Husain

Name: AI Evaluations Clearly Explained in 50 Minutes (Real Example) | Hamel Husain
Uploaded: 2025-09-28
Duration: 3150 s
Channel: Peter Yang
Description: Read a transcript preview of “AI Evaluations Clearly Explained in 50 Minutes (Real Example) | Hamel Husain” from Peter Yang, uploaded on 2025-09-28.

Channel: Peter Yang

Uploaded: 2025-09-28

Duration: PT3150S

Read a transcript preview of “AI Evaluations Clearly Explained in 50 Minutes (Real Example) | Hamel Husain” from Peter Yang, uploaded on 2025-09-28.

Transcript preview

Language: English (original, YouTube automatic captions)

[0:00] this looking at data accounting. You can

[0:02] get insane value by just doing that. And

[0:04] that's the one part that everyone skips.

[0:07] I'm going to use a spreadsheet to drive

[0:08] home the fact that this process can be

[0:11] dead simple and you can get immense

[0:13] value out of it. When you see an average

[0:15] score of 3.2 versus 3.7, no one really

[0:18] knows what the hell that means. It's not

[0:20] really actionable, honestly. They're

[0:22] like, "Oh, it's like getting better."

[0:23] Honestly, like nobody really knows

[0:25] whether it's getting better or not. So,

[0:26] as a product manager, if you ever see

[0:28] the word agreement, you need to pause

[0:30] and be like, hm, let me dig into this,

[0:33] please. If people don't trust your

[0:34] evals, they won't even trust you. You're

[0:37] done.

[0:39] Okay, welcome everyone. My guest today

[0:41] is uh Hamill. Haml has trained over

[0:43] 2,000 PMs and engineers from companies

[0:45] like OpenAI, Anthropic, and Google on

[0:48] how to run AI evaluations. He teaches

[0:50] the most popular course on this topic at

[0:52] Mayheaven. So really excited to dig into

[0:55] these best practices and I feel like I

[0:57] made a lot of mistakes that you know and

[0:59] have a lot of assumptions that Haml can

[1:01] help. So welcome.

[1:03] >> Really happy to be here. Excited to talk

[1:06] about evals.

[1:07] >> All right. So you know there's been a

[1:09] lot of like online debate about like you

[1:10] know are evaluable or not like blah blah

[1:12] blah on Twitter and and um let's just

[1:15] make this really practical. Um like let

[1:18] us talk about evals for like a real

[1:20] product like like do you have a do you

[1:21] have a product example that you want to

[1:23] talk about