Probability & Statistic in Machine Learning(II)

There are many different measures of central tendency

Mean
Median
Mode

Expected Value#

Discrete random variables

Weighted using the PMF:

E[X] = \sum_{x} x \cdot P_X(x)

Continuous random variables

Weighted using the PDF:

E[X] = \int_{-\infty}^{\infty} x \cdot f_X(x) \, dx

Variance#

Variance is the expected value of the square minus the square of the expected value:

\text{Var}(X) = E[X^2] - (E[X])^2

If $X$ and $Y$ are independent:

\text{Var}(X + Y) = \text{Var}(X) + \text{Var}(Y)

Standard Deviation#

\text{SD}(X) = \sqrt{\text{Var}(X)}

Sum of Gaussians#

The sum of two Gaussian random variables is still Gaussian.

Skewness and Kurtosis#

Consider two examples:

In case 1, you buy a $1 lottery ticket: 99% of the time you lose$ 1, and 1% of the time you win $99.

In case 2, you are an insurance company selling a $1 car insurance policy: 99% of the time you gain$ 1, and 1% of the time you lose $99.

Checking the first two moments:

E[X]_{\text{Case 1}} = 0.99(-1) + 0.01(99) = 0

E[X]_{\text{Case 2}} = 0.99(1) + 0.01(-99) = 0

\text{Var}_{\text{Case 1}}(X) = 0.99(-1)^2 + 0.01(99)^2 = 99

\text{Var}_{\text{Case 2}}(X) = 0.99(1)^2 + 0.01(-99)^2 = 99

The expected return and variance are identical — so how do we tell them apart?

We can use $E[X^3]$ , the third moment, to distinguish them:

E[X^3]_{\text{Case 1}} = 0.99(-1)^3 + 0.01(99)^3 \approx 9703

E[X^3]_{\text{Case 2}} = 0.99(1)^3 + 0.01(-99)^3 \approx -9703

The formal definition of skewness is:

\text{Skew}(X) = \frac{E[(X - \mu)^3]}{\sigma^3}

Case 1 (the lottery ticket) has positive skew: a long right tail, since most outcomes are small losses punctuated by a rare large gain. Case 2 (the insurance policy) has negative skew: a long left tail, since most outcomes are small gains punctuated by a rare large loss. Unlike variance, which only measures spread, skewness captures asymmetry — which direction the rare extreme outcome lies in.

This is the same pattern we'll see below with kurtosis: skewness (the 3rd moment) reveals the direction of tail risk, while kurtosis (the 4th moment) reveals its magnitude, and both can be invisible if you only look at mean and variance.

Here's another example to illustrate kurtosis:

Case 1

0.5 probability of winning $1
0.5 probability of losing $1

Case 2

100/202 probability of winning 10 cents
100/202 probability of losing 10 cents
1/202 probability of winning $10
1/202 probability of losing $10

Both cases have $E[X] = 0$ by symmetry. Checking the variance:

\text{Var}_{\text{Case 1}}(X) = 0.5(1)^2 + 0.5(-1)^2 = 1

\text{Var}_{\text{Case 2}}(X) = \frac{200}{202}(0.1)^2 + \frac{2}{202}(10)^2 \approx 1

So a first- and second-moment view (mean and variance) says these two bets are essentially identical. But they feel very different: Case 1 is a steady coin flip, while Case 2 is "almost always a tiny, boring outcome, but with a rare catastrophic tail" — much like an insurance payout or a market crash. This is exactly what kurtosis is designed to capture.

The formal definition of kurtosis is the fourth standardized moment:

\text{Kurt}(X) = \frac{E[(X - \mu)^4]}{\sigma^4}

Because the deviations are raised to the 4th power, kurtosis is very sensitive to rare, large values (outliers) and much less sensitive to the frequent, small ones near the mean. In Case 2, the rare $\pm 10$ outcome dominates $E[(X-\mu)^4]$ even though it barely affects the variance, so Case 2 has much higher kurtosis than Case 1.

This is why kurtosis matters: two distributions can share the same mean and variance, yet the fourth moment reveals a difference in tail risk — how likely extreme, low-probability events are. A distribution with higher kurtosis than the Gaussian is called leptokurtic ("fat-tailed"), and one with lower kurtosis is platykurtic ("thin-tailed"). In finance and ML, relying on variance alone can seriously understate the risk of rare but extreme events.

Quantiles and Box-Plots#

A quantile is a value below which a given proportion of the data falls — for example, the median is the 50th percentile (the 0.5-quantile).

The box plot below visualizes the quartiles, median, and outliers of a distribution:

Screenshot 2026-07-22 at 12.04.38 AM.png

Kernel Density Estimation: Approximate the PDF based on the data#

A histogram approximates a distribution's shape, but it's sensitive to bin width and bin placement, and it produces a blocky, discontinuous estimate. Kernel Density Estimation (KDE) is a non-parametric way to estimate a smooth PDF directly from samples, without assuming a parametric family (like Gaussian) up front.

The idea: place a small "bump" (the kernel) centered at each data point, then sum all the bumps and normalize:

\hat{f}_h(x) = \frac{1}{n h} \sum_{i=1}^{n} K\left(\frac{x - x_i}{h}\right)

where $x_1, \dots, x_n$ are the observed samples, $K$ is the kernel function (often a standard Gaussian), and $h > 0$ is the bandwidth.

The bandwidth $h$ controls the smoothness of the estimate, and is the key bias-variance tradeoff in KDE:

Too small $h$ : each bump is narrow, so the estimate is spiky and overfits the noise in the sample (high variance, low bias).
Too large $h$ : the bumps are wide and overlap heavily, oversmoothing the estimate and washing out real structure like multiple modes (high bias, low variance).

In practice, $h$ is tuned via cross-validation or a plug-in rule (e.g. Silverman's rule of thumb), analogous to choosing bin width for a histogram — except KDE's output is a smooth, continuous curve rather than discrete bars.

Violin Plots#

Violin plots combine a KDE with a box plot to show both the distribution's shape and its summary statistics.

QQ Plots#

A QQ (quantile-quantile) plot compares the quantiles of a sample against the quantiles of a theoretical distribution (e.g. Gaussian) — points falling along the diagonal line indicate the sample follows that distribution.

Marginal and Conditional Distribution#

Marginal Distribution

Discrete case:

P_X(x) = \sum_{y} P(X=x, Y=y)

Continuous case:

f_X(x) = \int_{-\infty}^{\infty} f(x,y) \, dy

Conditional Distribution

Discrete case:

P(X=x \mid Y=y) = \frac{P(X=x, Y=y)}{P(Y=y)}

Continuous case:

f(x \mid y) = \frac{f(x,y)}{f_Y(y)}

Covariance & Covariance Matrix#

\text{Cov}(X, Y) = E[XY] - E[X]E[Y]

If independent:

\text{Cov}(X_1, X_2) = 0.0

For a random vector, the covariance matrix collects the pairwise covariances between components:

\Sigma_{ij} = \text{Cov}(X_i, X_j)

Correlation Coefficient#

Looking at covariance alone can be misleading about the strength of a relationship, since it isn't scale-invariant. The correlation coefficient normalizes by the standard deviations:

\rho_{X,Y} = \frac{\text{Cov}(X,Y)}{\sigma_X \sigma_Y}

Multivariate Gaussian Distribution#

Real-world data is often generated by multiple correlated variables jointly, which is where the multivariate Gaussian distribution comes in:

f_X(\mathbf{x}) = \frac{1}{\sqrt{(2\pi)^k |\Sigma|}} \exp\left(-\frac{1}{2}(\mathbf{x}-\mu)^T \Sigma^{-1} (\mathbf{x}-\mu)\right)