Joint Distributions and Independence

Attempt these from memory before reading.

• Define the joint CDF of $(X, Y)$.
• Does knowing the marginal distributions of $X$ and $Y$ determine the joint distribution of $(X, Y)$?
• Define covariance. What does $\operatorname{Cov}(X, Y) = 0$ tell you?
• Define independence of two random variables.
• Is the sum of two independent normal random variables always normal?

The single most important fact on this page is that the marginal distributions of $X$ and $Y$ do not determine the joint distribution. Knowing everything about $X$ and everything about $Y$ tells you nothing about how they move together.

That is not a technicality; it is the entire subject of dependence. The joint distribution is extra information, and covariance and correlation are attempts to summarise one aspect of it.

Worked example (the canonical counterexample). Let $X$ be standard normal and let $Y = X$ with probability $\frac12$ and $Y = -X$ with probability $\frac12$, independently of $X$‘s value. Then $Y$ is also standard normal, so $X$ and $Y$ have identical marginals. But $(X, Y)$ and $(X, -X)$ are completely different joint distributions: one is concentrated on the diagonal, the other on the anti-diagonal.

The marginals are the same. The dependence is opposite. Anything you want to know about how $X$ and $Y$ relate requires the joint distribution, and no marginal summary can substitute.

That is the reason correlation is not causation, and it is the reason two portfolios with identical marginal return distributions can have completely different risk.

Definition. The joint CDF of $(X, Y)$ is $F_{X,Y}(x, y) = P(X \leq x, Y \leq y)$.

Definition. The joint PDF (for continuous random variables) is $f_{X,Y}(x, y) \geq 0$ such that

Definition. The marginal PDF of $X$ is $f_X(x) = \int_{-\infty}^{\infty} f_{X,Y}(x, y)\, dy$.

Definition. The covariance of $X$ and $Y$ is

Proposition 2.6. $\mathrm{Cov}(X, Y) = \mathrm{Cov}(Y, X)$ and $\mathrm{Cov}(aX + b, cY + d) = ac\,\mathrm{Cov}(X, Y)$.

Definition. The correlation coefficient is

Theorem 2.7 (Cauchy—Schwarz for Random Variables). $|\rho(X, Y)| \leq 1$, with equality if and only if $Y = aX + b$ almost surely for some $a, b$.

Definition. $X$ and $Y$ are independent if $F_{X,Y}(x, y) = F_X(x)\, F_Y(y)$ for all $x, y$.

For continuous random variables, this is equivalent to $f_{X,Y}(x, y) = f_X(x)\, f_Y(y)$.

Proposition 2.8. If $X$ and $Y$ are independent, then $\mathrm{Cov}(X, Y) = 0$. The converse is false.

Definition. The conditional PDF of $Y$ given $X = x$ is:

provided $f_X(x) > 0$.

Definition. The conditional expectation of $Y$ given $X = x$ is:

The random variable $E[Y | X]$ has the property that $E[E[Y|X]] = E[Y]$ (law of total expectation).

The bivariate normal distribution has joint PDF:

where $\rho = \mathrm{Corr}(X, Y)$.

Properties of the bivariate normal:

• Marginal distributions are normal: $X \sim N(\mu_X, \sigma_X^2)$, $Y \sim N(\mu_Y, \sigma_Y^2)$.
• Conditional distribution is normal: $Y | X = x \sim N\left(\mu_Y + \rho \frac{\sigma_Y}{\sigma_X}(x - \mu_X),\ \sigma_Y^2(1 - \rho^2)\right)$.
• $X$ and $Y$ are independent if and only if $\rho = 0$.

For a transformation $(U, V) = g(X, Y)$ where $g$ is a diffeomorphism, the joint PDF is:

where $J$ is the Jacobian matrix of the inverse transformation.

Example. Let $X, Y$ be independent exponential($\lambda$) random variables. The joint PDF of $U = X + Y$ and $V = X/(X + Y)$ factors as $f_{U,V}(u, v) = \lambda^2 u e^{-\lambda u} \cdot \chi_{(0,1)}(v)$, showing $U$ and $V$ are independent with $U \sim \mathrm{Gamma}(2, \lambda)$ and $V \sim \mathrm{Uniform}(0, 1)$.

If $X$ and $Y$ are independent, the PDF of $Z = X + Y$ is the convolution:

For moment generating functions: $M_Z(t) = M_X(t)\, M_Y(t)$.

Uncorrelated does not mean independent. The $Y = X^2$ example is the one to remember: $Y$ is completely determined by $X$, yet the covariance is zero by symmetry. Correlation measures linear dependence only, and it is blind to every other kind.

Anscombe’s quartet makes the same point visually. Four datasets with the same means, the same variances, the same correlation and the same fitted line — one linear, one curved, one with an outlier driving everything, one with a vertical line. Summary statistics are not a substitute for looking at the data.

The normal is the one family where the counterexamples stop. For a bivariate normal:

• uncorrelated implies independent
• the marginals are normal
• the sum of the two is normal
• the conditional expectation $E[Y \mid X]$ is linear in $X$

Every one of those is false in general, and every one is true for the bivariate normal. That is why the normal is the distribution statistical methods are built around — not because data is normal, but because theorems are provable.

Mix these with material from Random Variables, Limit Theorems and Measure Theory.

1. (Probability.) Show that if $X$ and $Y$ are independent then $\operatorname{Cov}(X, Y) = 0$, and give the counterexample showing the converse fails.
2. (Probability.) Show $\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y) + 2\operatorname{Cov}(X, Y)$, and deduce the variance of a sum of independent variables.
3. (Probability.) Show that the bivariate normal with correlation $ho$ has conditional mean \mu_Y + ho rac{\sigma_Y}{\sigma_X}(x - \mu_X), and explain why this is the regression line.
4. (Limit theorems.) Use the CLT and the delta method to find the asymptotic distribution of the sample correlation coefficient.
5. (Measure theory.) Show that if $X$ and $Y$ are independent then the joint measure is the product measure, and that Fubini’s theorem is what lets you compute $E[X + Y]$ by iterated integrals.

Problem 1. Let $X$ and $Y$ have joint PDF $f_{X,Y}(x, y) = c(1 - x^2 - y^2)$ for $x^2 + y^2 \leq 1$. Find $c$, the marginal PDFs, and $P(X > 0, Y > 0)$.

Problem 2. Show that if $\mathrm{Cov}(X, Y) = 0$ for a bivariate normal pair, then $X$ and $Y$ are independent.

Problem 3. Let $X$ and $Y$ be independent standard normal variables. Find the distribution of $R = \sqrt{X^2 + Y^2}$ and $\Theta = \arctan(Y/X)$.

Solution. $R$ has Rayleigh distribution with PDF $f_R(r) = r e^{-r^2/2}$ for $r \geq 0$, and $\Theta \sim \mathrm{Uniform}(0, 2\pi)$, and $R$ and $\Theta$ are independent. $\blacksquare$

Problem 4. Suppose $X \sim N(0, 1)$ and $Y = X + Z$ where $Z \sim N(0, 1)$ is independent of $X$. Find the joint distribution of $(X, Y)$ and compute $\mathrm{Cov}(X, Y)$.

Law of total probability: $f_Y(y) = \int_{-\infty}^{\infty} f_{Y|X}(y|x)\, f_X(x)\, dx$.

Bayes’ theorem: $f_{X|Y}(x|y) = \frac{f_{Y|X}(y|x)\, f_X(x)}{f_Y(y)} = \frac{f_{Y|X}(y|x)\, f_X(x)}{\int f_{Y|X}(y|t)\, f_X(t)\, dt}$.

The joint moment generating function of $(X, Y)$ is:

If $X$ and $Y$ are independent, then $M_{X,Y}(s, t) = M_X(s)\, M_Y(t)$. Joint moments can be recovered by differentiation:

Problem 5. Let $(X, Y)$ have joint PDF $f_{X,Y}(x, y) = 6x$ for $0 < x < 1$, $0 < y < 1 - x$. Find the marginal PDFs and determine if $X$ and $Y$ are independent.

Problem 6. Suppose $X$ and $Y$ are independent exponential random variables with rate $\lambda$. Find the distribution of $X + Y$ and $X/(X + Y)$. Are they independent?

Problem 7. Show that $\mathrm{Var}(X) = E[\mathrm{Var}(X|Y)] + \mathrm{Var}(E[X|Y])$ (the variance decomposition formula).

Mistake 1: Confusing independence with pairwise independence. Two random variables $X$ and $Y$ are independent if their joint distribution factors as $F_{X,Y}(x, y) = F_X(x)F_Y(y)$. For three or more variables, pairwise independence (each pair is independent) does not imply mutual independence. There exist random variables $X, Y, Z$ that are pairwise independent but not mutually independent.

Mistake 2: Assuming that zero covariance implies independence. If $X$ and $Y$ are independent, then $\mathrm{Cov}(X, Y) = 0$, but the converse is false. Uncorrelated variables can be dependent. For example, let $X$ be uniform on $\{-1, 0, 1\}$ and $Y = X^2$. Then $\mathrm{Cov}(X, Y) = 0$, but $X$ and $Y$ are not independent.

Mistake 3: Forgetting that the marginal distribution does not determine the joint distribution. Knowing the marginal distributions $f_X(x)$ and $f_Y(y)$ does not determine the joint distribution $f_{X,Y}(x, y)$. The joint distribution also depends on the dependence structure between $X$ and $Y$. Two different joint distributions can have the same marginals.

Mistake 4: Confusing the correlation coefficient with the slope of a regression line. The correlation coefficient $\rho(X, Y)$ measures the strength of the linear relationship between $X$ and $Y$, but it is not the same as the slope of the regression line of $Y$ on $X$. The slope is $\rho(X, Y) \cdot \sigma_Y / \sigma_X$, which depends on the standard deviations.

Mistake 5: Assuming that linear regression is appropriate for nonlinear relationships. Linear regression fits a straight line to the data, but if the relationship between $X$ and $Y$ is nonlinear, the regression line may be misleading. Always check the scatterplot and consider nonlinear models if the relationship appears curved.

Joint distributions describe how two or more random variables vary together. Independence means knowing one variable tells you nothing about the other, which mathematically factorizes the joint distribution into a product. Dependence creates structure: positive correlation means variables tend to move together, negative correlation means they move oppositely. The marginal distribution is what you see when you ignore one variable, like looking at a 3D landscape from above. Conditional distributions slice the joint distribution along one variable, revealing how the other behaves given specific information. These concepts form the backbone of statistical modeling.

• Random Variables: Random variables provide the building blocks for constructing joint distributions and analyzing dependence.

• Probability Spaces: Probability spaces provide the measure-theoretic foundation for defining joint distributions.

• Limit Theorems: The central limit theorem extends to sums of independent random variables with different distributions.

• Quantum Mechanics

• Graph Theory

• Classical Mechanics

• Electromagnetism

• Statistical Learning

• Statistical Mechanics

• Estimation and Hypothesis Testing
• Limit Theorems
• Probability Spaces
• Probability and Statistics