Complete Beginner's Course on AI Evaluations in 50 Minutes (2025) | Aman Khan
Channel: Peter Yang
Uploaded:
Duration: PT3108S
Read a transcript preview of “Complete Beginner's Course on AI Evaluations in 50 Minutes (2025) | Aman Khan” from Peter Yang, uploaded on 2025-08-24.
Transcript preview
Language: English (original, YouTube automatic captions)
[0:00] The CPOS of these companies are telling
[0:02] you eval are really important. You
[0:04] should probably think what are AI eval
[0:07] exactly. I think there's really just
[0:09] four types of of evalu.
[0:19] And we can define what's good or bad for
[0:22] any of these. So all you're doing here
[0:24] is basically building out a golden data
[0:26] set in the first place. So, we're going
[0:28] to hit generate and let's see what the
[0:29] agent comes up with here. So, okay. So,
[0:31] it starts giving me a prompt. You want
[0:33] LLM as a judge to align with your human
[0:36] label and the the judgment that you're
[0:37] giving it. Let's get super practical.
[0:39] Now, there's like a lot of eval videos
[0:41] out there about frameworks and blah blah
[0:43] blah. Like, I just want to like actually
[0:44] walk through a real example. All right,
[0:46] let's look at one. So,
[0:50] okay, welcome everyone. My guest today
[0:52] is Aman, head of product at Arise. And
[0:55] today, Aman and I are going to show you
[0:57] exactly how experienced PMs like
[0:59] ourselves run AI evaluations using a
[1:03] real example. And we're going to walk
[1:05] through all the steps including uh
[1:07] defining the evaluation rubric, creating
[1:09] our human label golden data set, running
[1:11] Elm as a judge, eval, and more. So,
[1:14] welcome Aman.
[1:15] >> Yeah, thanks for having me back, man.
[1:17] It's awesome to be back, and it feels
[1:19] like a lot has changed since we last
[1:20] talked about this, so I'm pretty excited
[1:22] to dive in. All right, so let's do a
[1:24] quick recap and then we'll dive into our
[1:26] example, right? So like um what are AI
[1:29] evals exactly and why are they uh you
[1:31] know the most important skill for PMs to