The AI Skill That Will Define Your PM Career in 2025 | Aman Khan (Arize)
Channel: Peter Yang
Uploaded:
Duration: PT2787S
Read a transcript preview of “The AI Skill That Will Define Your PM Career in 2025 | Aman Khan (Arize)” from Peter Yang, uploaded on 2025-01-12.
Transcript preview
Language: English (original, YouTube automatic captions)
[0:00] evals actually force you to get into the
[0:02] shoes of your user like you can no
[0:05] longer just hypothesize I wonder if this
[0:07] they'll do this or I wonder if they'll
[0:09] do that like it kind of forces the PM to
[0:12] be way more empathetic of the customer
[0:15] and what the customer is going to try to
[0:16] do and articulate that in writing which
[0:19] is a whole another skill so you want to
[0:21] translate your understanding of the user
[0:23] into text and then you want to kind of
[0:25] use that to iterate on the system I
[0:27] think that's hard I also think that you
[0:29] know there's a high level of
[0:31] subjectivity and non-determinism which
[0:33] makes it challenging like it's not as
[0:35] easy to predict as writing code like
[0:37] what the outcomes are going to be in
[0:38] this case and you're trying to model
[0:39] human behavior as much as possible let's
[0:42] say the llm writes a sentence or the AI
[0:44] writes a sentence like was that a good
[0:45] sentence it depends a lot on the context
[0:47] it can depend a lot on the different
[0:49] scenarios that you're putting the system
[0:51] into and so understanding and looking at
[0:53] the data and basically seeing okay
[0:55] here's the clusters of types of
[0:58] scenarios that the system is going to
[1:00] put into and being able to break the
[1:02] system up into more components and more
[1:04] fine grain scenarios makes the writing
[1:06] of the evales easier because now you're
[1:09] you're iterating in a tighter Loop but
[1:11] it is really hard to think about this
[1:12] when you're first getting started all
[1:13] right welcome everyone my guest today is
[1:15] Aman KH director of product at arise AI
[1:19] so Aman and I attended a talk rec