PDF of (X, X)

Problem statement

Let $X$ be a continuous real-valued random variable. What is the pdf of $Z = (X, X)$? This is of course a problème mal posé because $Z$ is not a continuous random variable in $\R^2$. However, the pdf of $Z$ makes sense in a different sense. Let's have a look...

Let us start by definining the set

$$E = \{(x, y)\in\R^2: x = y\}.$$

Note that $Z:(\Omega, \mathcal{F}, \mathsf{P}) \to E$ takes values in $E$.

We also define the "lifting" map $\pi:\R\ni x \mapsto (x, x)\in E$, which is continuous and invertible.

There is no such PDF

We have a continuous real-valued random variable $X$. The random variable $Z=(X, X)$, seen as a random variable in $\R^2$ doesn't have a pdf; let's see why: take a subset of $E$, say $N=\{(x, x); x\in [a, b]\}$, with $a < b$, which is Lebesgue-measurable and a null set (measure zero). However, $\mathsf{P}[Z\in N] = \mathsf{P}[X\in [a, b]] \neq 0$. Therefore, there is no function $p_Z:\R^2\to\R_+$ such that

$$\int_{N}p_Z \mathrm{d}(x, y) \neq 0,$$

so $Z$ does not have a pdf.

The Radon-Nikodym derivative

The Radon-Nikodym derivative states that if $\mu, \nu$ are measures and $\mu \ll \nu$, then there is a nonnegative-valued function $f\in\mathcal{L}^1(\mu)$ such that $\nu(A) = \int_A f\mathrm{d}\mu$. The function $f$ is unique $\mu$-a.e. We call $f$ the Radon-Nikodym derivative of $\nu$ with respect to $\mu$ and we denote $f = \frac{\mathrm{d}\nu}{\mathrm{d}\mu}$. This result can be extended to cases where $\nu$ is not absolutely continuous with respect to $\mu$.

PDFs as Radon-Nikodym derivatives

We can generalise the notion of a pdf: a pdf can be seen as Radon-Nikodym derivative with respect to a measure (the default being the Lebesgue measure). Here we need to use a different measure.

Firstly, recall the definition of the pushforward measure, $Z_*\mathsf{P}$, of the probability measure $\mathsf{P}$ via $Z$, which is defined as the measure

$$(Z_*\mathsf{P})(A) = \mathsf{P}[Z\in A],$$

If $Z_*\mathsf{P}$ is absolutely continuous with respect to a measure $\nu$, i.e., $Z_*\mathsf{P} \ll \nu$, then there is a function $p_Z$, called the pdf of $Z$ with respect to $\nu$, such that

$$(Z_*\mathsf{P})(A) = \int_A p_Z \mathrm{d}\nu.$$

Next, we will construct such a measure, $\nu$.

Constructing new measures

We start with the Lebesgue measure, $\mu$, on $\R$. We will define a measure on $E$. Firstly, we make $E$ into a measurable space by endowing it with the σ-algebra $\mathcal{E} = \{\pi(B), B\in \mathcal{B}_\R\}$, where $\mathcal{B}_\R$ is the Borel σ-algebra on $\R$. Now we define a measure $\nu:\mathcal{E}\to\R_+$ as

$$\nu(\pi(B)) =\mu(B)$$

It can be seen that $Z_*\mathsf{P} \ll \nu$, so there is a pdf of $Z$ wrt $\nu$, i.e., $\mathsf{P}[Z\in \pi(B)] = \int_{\pi(B)}p_Z \mathrm{d}\nu$, where $\pi_Z:E\to\R_+$. Note that $\nu$ is the pushforward measure of $\mu$ via $\pi$, i.e., $\nu = \pi_*\mu$. Let us recall that for a measurable function $g:E\to\R$,

$$\int_E g\mathrm{d}\nu = \int_E g\mathrm{d}(\pi_*\mu) = \int_\R (g\circ \pi)\mathrm{d}\mu.$$

This is a change of measure formula in integration.

We can now extend the measure $\nu$ to the entire $\mathcal{B}_{\R^2}$. The σ-algebra $\mathcal{E}$ is a sub-σ-algebra of $\mathcal{B}_{\R^2}$; in fact, $\mathcal{E}$ is the restriction of $\mathcal{B}_{\R^2}$ on $E$. We can construct a measure $\tilde{\nu}$ on $\mathcal{B}_{\R^2}$ such that

$$\tilde{\nu}(B) = \nu(B\cap E),$$

for $B\in\mathcal{B}_{\R^2}$.

If we treat $Z$ as a random variable that takes values in $\R^2$, we see that $Z_*\mathsf{P}$ is absolutely continuous wrt $\tilde{\nu}$.

The pdf of $Z$ wrt $\nu$

Here we treat $Z$ as a random variable that takes values in $E$. Let us denote the pdf of $Z$ wrt the measure $\nu$ by $p_Z^\nu$. We claim that $p^{\nu}_Z(x, x) = p_X(x)$ for $\mu$-almost all $x$.

By definition, for the pdf of $Z$ wrt $\nu$, which is a function $p_Z^\nu:E\to\R_+$, we have that for $S\subseteq E$, measurable,

$$\mathsf{P}[Z\in S] = \int_S p_Z^\nu\mathrm{d}\nu = \int_S p_Z^\nu \mathrm{d}(\pi_*\mu) = \int_{\pi^{-1}(S)}p_Z^\nu \circ \pi \mathrm{d} \mu.$$

At the same time,

$$\mathsf{P}[Z\in S] = \mathsf{P}[X\in \pi^{-1}(S)] = \int_{\pi^{-1}(S)}p_X\mathrm{d}\mu,$$

where $p_X$ is the pdf of $X$ wrt $\mu$. We conclude that...

The pdf of $Z$ wrt $\tilde{\nu}$

Here we are looking for a function $p_Z^{\tilde{\nu}}:\R^2\to\R_+$ such that for all measurable sets $A\subseteq \R^2$,

$$\mathsf{P}[Z\in A] = \int_A p_Z^{\tilde{\nu}} \mathrm{d}\tilde{\nu}.$$

By following the same procedure as above we'll find that $(p_Z^{\tilde{\nu}}1_E)\circ \pi = p_X$, $\mu$-a.s., but the Radon-Nikodym derivative is $\tilde{\nu}$-a.e. unique, so $p_Z^{\tilde{\nu}} | E^c$ is free. Note that $E^c$ is a $\tilde{\nu}$-null set. This leads to the following result...