WEBVTT

1
00:00:00.300 --> 00:00:02.668
An LLM answers by writing words.

2
00:00:03.800 --> 00:00:04.801
Jev never writes.

3
00:00:04.801 --> 00:00:06.509
It picks one of your options.

4
00:00:07.800 --> 00:00:09.293
And it tells you how sure it is.

5
00:00:11.300 --> 00:00:13.796
Every request is one of three shapes.

6
00:00:13.796 --> 00:00:16.697
A choice, a score, or a single
probability.

7
00:00:17.600 --> 00:00:19.893
That number decides what you automate.

8
00:00:19.893 --> 00:00:21.702
Above your threshold it ships.

9
00:00:21.702 --> 00:00:23.211
Below it, a person looks.

10
00:00:24.600 --> 00:00:25.955
Calibrated means this.

11
00:00:25.955 --> 00:00:30.083
Say seventy percent, and be right about
seventy times in a hundred.

12
00:00:31.000 --> 00:00:33.820
The same answers, under a dishonest
number.

13
00:00:33.820 --> 00:00:35.395
That gap is the problem.

14
00:00:36.300 --> 00:00:38.945
So why report a number that makes it look
unsure?

15
00:00:40.200 --> 00:00:44.680
Score it on a straight line, and the
cheapest answer is always total certainty.

16
00:00:46.200 --> 00:00:51.341
Score it on this curve instead, and the
cheapest answer is exactly what it really believes.

17
00:00:53.200 --> 00:00:55.749
That is what RLCD is named after.

18
00:00:55.749 --> 00:00:57.679
Not constrained decoding.

19
00:00:57.679 --> 00:00:59.301
Calibrated decisions.

20
00:00:60.000 --> 00:01:02.158
TypeSafe publishes the contract.

21
00:01:02.158 --> 00:01:05.867
The loss, the data, and any calibration
number are not.

22
00:01:06.600 --> 00:01:08.712
It decides, and it says how sure.
