4.4 MLE

Up to now, inference has felt like this: we observe a sample, we compute a number from it, and we use that number as our best guess about the population.

If we want the average session length in the population, we compute the sample mean. If we want the overall click rate in the population, we compute the sample proportion. If we want variability, we compute a sample variance.

All of these examples have one thing in common. The object you are trying to learn is a single population summary: one number (or a small handful of numbers). The entire dataset is being compressed into a small summary because the question itself asks for a summary.

Now we take one step that quietly changes everything.

4.4.1 What are we inferring?

Instead of asking for a summary of the population, we start asking for how the outcome changes with inputs. In other words, we stop asking ``what is true on average?'' and start asking ``what happens under conditions?''

That single shift changes the object we are trying to infer.

Two kinds of inference:

The clean way to frame this is not inference vs modelling. Both are inferences. The difference is what you are inferring.

In population-summary inference, the unknown is a population quantity like a mean, a variance, or an overall proportion.

For example, if  is click (0 or 1), then the population click rate is:

If your question is What is the average click rate of customers?, you are asking for this one number . A sample proportion is built for this job.

In model-based conditional inference, the unknown is not a single summary number. The unknown is the parameter set of a probability mechanism that describes the outcome under different inputs.

Here the object is:

If you ask How does click probability depend on age and device? you are no longer asking for . You are asking for a conditional probability: how the probability changes as  changes. That requires a different kind of inferential target.

This is the core hinge: the moment the question becomes conditional, the thing you are trying to infer is no longer a single population summary. It is a parameterised conditional rule.

Conditioning requires a model, even when x is present

At this point, we might ask: Why can we not just use sample statistics conditionally? For example, why not compute click rate for age 25?

That is a fair question and it is exactly where we earn the need for models.

Consider this question: Among users aged 25 in the sample, what is the click rate?

If we have enough users aged exactly 25, we can compute a sample proportion in that subset. But immediately we hit a practical and mathematical problem: real inputs like age are not repeated many times at exactly the same value. Even if we do have some repeats, the estimate becomes unstable as soon as the subset is small.

So what do we do? We bin. We change age 25 into age between 24 and 26 or age between 20 and 30.

Now we have an answer, but we paid for it with two costs.

First, the answer depends on arbitrary bin choices. Change the bins and we change the estimate. Second, as soon as we add more inputs (age and device and time), the number of bins explodes and most bins become sparse or empty.

That means the binning approach is not really solving the conditional question in a principled way. It is creating a rough discretised approximation whose behaviour depends on your bin design.

Now compare that to a slightly different question:

What is the click probability for age 25?

Notice what changed. This is no longer asking “in my sample subset, what fraction clicked?” It is asking for the probability of click at a specific input value as a property of the underlying process:

Even if age 25 appears in the dataset, the moment we want a stable conditional probability that is not at the mercy of binning decisions, we are asking for a structured conditional object, not a subset summary.

And if we ask:

What is the click probability for age 78?

Now we may have no users aged 78 in the dataset at all. Yet the business question is still meaningful. A sample statistic cannot be defined on an empty subset. The only way this question makes sense is if we assume some structure that links neighbouring ages and allows the conditional probability to be defined beyond the exact values we observed.

A sample statistic is a function of the observed dataset only. It produces a finite set of numbers.  

A conditional probability is a function over the input space. It must assign a probability for every relevant input value, not just those that happen to appear frequently in the sample. We cannot get such a function from raw sample summaries unless we assume a structure.

That structure is what we call a probability model. The reason we need probability models is not because sample statistics are wrong, but because the object we are trying to infer has changed.

4.4.2 A Probability Model

Until now, we have earned one necessity: if the question is conditional, we must impose structure. Otherwise we are stuck with arbitrary binning and unstable subsets.

A probability model is the formal way of imposing that structure. It is a parameterised family of distributions^[a].

This sentence has two parts:

• “distribution” means a rule that assigns probabilities to outcomes (a PMF for discrete outcomes, a PDF for continuous outcomes)
• “parameterised family” means the rule contains unknown quantities, called parameters, that determine its shape.

When we choose a model, we are not claiming we know the parameters. We are claiming we know the form of the probability mechanism up to unknown parameters.

A model is not only the visible probability function. It also includes assumptions that define how the data is generated.

In particular, a probability model typically encodes assumptions about:

• Support
• Independence structure
• Noise structure
• How inputs enter

The i.i.d. assumption:

In most introductory MLE settings, we assume observations are independent and identically distributed (i.i.d.), unless stated otherwise.

• independent: one row of the dataset does not affect another row.
• identically distributed: the same underlying mechanism^[b] generates each row

A few examples of probability models are:

If you are doing population-summary inference about a single variable , you might choose a model like a “Normal model”:

Here the parameters  and  describe the centre and spread of the population distribution of .

If your outcome is binary and you are not conditioning on inputs, you might choose a “Bernoulli model”:

Here  is the population probability of success.

If you are doing model-based conditional inference, the model must be conditional:

For logistic regression,  is Bernoulli, but its success probability depends on  through parameters . That is the mathematical object that turns conditional probability as a function of inputs into something estimable.

At this stage we still have not estimated anything. We have only made a choice: we have selected a model family that can assign probabilities to outcomes.

This is necessary because the next concept, likelihood, is built from a simple idea:

Given a model and parameters, what probability does the model assign to the outcomes that actually occurred?

We cannot even ask that question without a probability model in hand and that takes us to the subsequent section.