3 Integration & expectation

3.1 Why??

We have now built a firm mathematical foundation for our adventure: we’ve captured the idea of running a probabilistic experiment (cf. Definition 2.1) as a probability space $(\Omega,\mathcal{F},\P)$ where $\Omega$ is a set/universe containing the possible outcomes of the experiment, $\mathcal{F}$ a $\sigma$-algebra on $\Omega$ whose elements are subsets of $\Omega$ that are our events, and a probability measure $\P: \mathcal{F} \to [0,1]$ that assigns to each event a probability/likelihood of that event happening (cf. Definition 1.50). Further we can have on that space a (or more) random variable $X$, which is a mapping $X:\Omega \to \R$ with a measurability property (cf. Definition 2.13), where the interpretation is that each time we run the experiment, we observe an outcome $\omega \in \Omega$ and then observe that $X$ takes the value $X(\omega)$ in $\R$ (its co-domain). So the “randomness” around which outcome is observed naturally causes “randomness” around which value $X$ takes on the real line.

That measurability property of $X$ was a key theme in Chapter 2, as it is the technical condition needed to allow us to study how $X$ “behaves on the real line” (the co-domain of $X$) i.e. what that “randomness” around which value $X$ takes on the real line looks like. Indeed we can make use of inverse images to translate behaviour on the real line back into events for which probabilities are actually defined: the probability that $X$ ends up in a Borel set $B$ on the real line is naturally given by $\P(X^{-1}(B))$ (cf. Definition 2.13). Mathematically we can describe/capture the whole “random behaviour” of $X$ on the real line in one object, namely its distribution (cf. Definition 2.20), or, equivalently, its cdf (cf. Proposition 2.22).

In this Part I of our lovely course, we have two key topics left to address:

Studying distributions of random variables, which is increasingly what we’ll essentially be doing, involve two key operations: computing probabilities and computing expectations. The former we have addressed already (as recalled above), for the latter we have so far only reiterated the formulas we knew already from our earlier Probability courses for the specific cases of discrete and continuous random variables (cf. Section 2.4). Two issues here: firstly we know nothing about expectations for random variables that are neither discrete nor continuous, and secondly (maybe more a cosmetic issue, but still!) it somehow feels weird that one type requires a summation while the other requires an integral. This chapter is devoted to resolving these issues: as we will see, there actually does exist a beautiful theory that provides a single expression for the expectation of any random variable (which is also backwards compatible with the specific formulas for discrete and continuous random variables we knew already)!
When we have the concept of expectation for any random variable available, we should also revisit the idea of conditional expectations: firstly to consider stuff we know already in this new concept of expectation, and secondly because it turns out that besides conditioning on events and random variables, as you knew already, it turns out that we can also make sense of conditioning on $\sigma$-algebras. We’ll discuss this in Chapter 4.

To tackle key topic 1 above in this chapter, we’ll develop a whole new type of integral called a Lebesgue integral (indeed, our hero Lebesgue again who also gave us the measure from Proposition 1.56!). Exciting stuff! :). Unfortunately we don’t have much time, only one week/chapter, so our focus throughout this chapter will in particular be on trying to give you a strong intuitive understanding rather than desperately trying to squeeze in/extensively discuss each and every mathematical detail. If you’re interested in a full account, then see e.g. Chapter III in Stroock (1994). The plan for this chapter is as follows:

In Section 3.3 & Section 3.4 we’ll introduce the Lebesgue integral and discuss key properties. Not much mentioning of Probability Theory in these sections!
Then in Section 3.5 we’ll make the link back to Probability Theory and see why it is natural to use Lebesgue integrals to define the expectation of a random variable. We’ll also discuss properties as well as backwards compatibility with the formulas for expectation we knew already for discrete and continuous random variables.
Finally in Section 3.6 we’ll finish the story we started in Section 2.7 about “modes of convergence”, by discussing the Monotone and Dominated Convergence Theorems.

3.2 A quick word about integration of real valued functions in general [⛔]

When you read the word “integral” or “integration”, most of you will automatically think back to the very first (and maybe only!) type of integral you have encountered in your earlier studies: the Riemann integral to call it by its full name. Given some (suitable) non-negative¹ function $f:[a,b] \to [0,\infty)$ we denote it by \[\int_a^b f(x) \d x \tag{3.1}\] and it has some value in $[0,\infty]:=[0,\infty) \cup \{\infty\}$ that we can naturally interpret as “the area under the graph of $f$” (i.e. the area of the region enclosed by the lines $x=a$ and $x=b$, the horizontal axis, and the graph of $f$). We typically compute its value as $F(b)-F(a)$, where $F$ is any antiderivative of $f$. Obligatory figure for illustration:

¹ Of course, as we know well we don’t need to limit ourselves to non-negative functions only but it is a natural place to start this discussion

Figure 3.1: The graph (dark blue) of $f(x)=x^2+1/2$ on the interval $[-1,2]$; and the area of the light blue region equals $\int_{-1}^2 x^2+1/2 \d x=4.5$

Obviously Riemann integrals are very important and regularly all you need. However, and here is also the mathematician’s desire for generalisation again, it turns out that you can take the core principles behind Riemann integrals and apply these more generally to construct a large zoo of many kinds of more exotic types of integrals:

you can integrate real valued functions with other domains than (part of) the real line,
you can integrate more complicated objects as well, not just functions!

In this course we will not touch on point 2. (within the Probability Theory context, you may very well encounter for instance stochastic integration in a different course) but we will spend this chapter exploring point 1.

Briefly about the global idea behind point 1. Let’s fix some $a$ and $b$. What does the Riemann integral actually do? Well, you choose a (suitable) function $f: [a,b] \to \R$ and compute its integral i.e. obtain a number. So essentially it is nothing but a mapping, say $I$, that has as domain some large set of suitable functions $[a,b] \to \R$ and that maps each of these to a real number (typically $\pm \infty$ are allowed as well, see further below), respecting some core principles. For example, if we let $I_R$ be the Riemann integral for functions $[0,1] \to \R$, then for $f(x)=x$ we get $I_R(f)=1/2$. Here are these core principles:

Definition 3.1 (What is an integral anyway? [⛔]) In general an integral (acting on real valued functions) is a mapping $I: \mathbf{D} \to \R$, where $\mathbf{D}$ is some collection of real valued functions (that forms a vector lattice), with the following defining properties:

linearity: $I(\alpha f+ \beta g)=\alpha I(f)+\beta I(g)$ for $\alpha,\beta \in \R$,
non-negativity: if $f$ is non-negative then $I(f) \geq 0$,
continuity: if $f_1 \geq f_2 \geq \ldots$ is a non-increasing sequence of non-negative functions so that $f_n \downarrow 0$ as $n \to \infty$ (pointwise) then also $I(f_n) \downarrow 0$ as $n \to \infty$.

It is not hard to see that these defining properties imply the following two properties (cf. Exercise 3.1):

if $f \leq g$ then $I(f) \leq I(g)$,
if $f_1 \leq f_2 \leq \ldots$ is a non-decreasing sequence of functions so that $f_n \uparrow f$ as $n \to \infty$ (pointwise), then also $I(f_n) \uparrow I(f)$ as $n \to \infty$.

Some notes:

Property 2 is in particular inspirational, it gives rise to the following strategy for constructing a new integral: identify within your $\mathbf{D}$ an easy to understand family of non-negative functions for which you can easily write down what value your integral should have, and that allows you to pointwise approximate any other non-negative function in $\mathbf{D}$, so that via property 2 the integral value for any non-negative function follows. The extension to any function is then a matter of considering its so-called positive and negative parts separately (using linearity i.e. property i). Indeed this is the strategy we’ll also be following to construct the Lebesgue integral!
Generally we’ll want to slightly extend this concept by allowing for integral values of $\pm \infty$ as well, in particular in property 2 it is natural to set $I(f)=\infty$ for cases where $\lim_{n \to \infty} I(f_n)=\infty$.

Fine — but how many different types of integrals i.e. operators as defined in Definition 3.1 do there actually exist? Maybe surprisingly at first sight, but a seminal result called the Riesz representation theorem guarantees that (a bit loosely speaking/under some additional technicalities) there is only one type: the Lebesgue integral! So arguably, the Lebesgue integral is the true king rather than the Riemann integral! But where does the Riemann integral fit in this picture then? Well, as we will see later on, it turns out that the Riemann integral is a special case of a Lebesgue integral: if a function $f: [a,b] \to \R$ is Riemann integrable, then its Riemann integral and Lebesgue integral are identical. If you’re interested in more, see e.g. Chapter VII in Stroock (1994).

Exercises

You can now do Exercise 3.1.

3.3 A little bit of preparation

Before we carry on with constructing Lebesgue integrals for real valued functions in Section 3.4, we need to do a little bit of prep. The context in this section is a real valued function i.e. a function $f: \Omega \to \R$, where $\Omega$ is some set — may of course be (part of) the real line but not necessarily. We use the symbol “$x$” for the argument of $f$ here, just because that’s what we’re used to in the context of Riemann integrals, but obviously it could equally well have been an “$\omega$” as we typically to use for the elements of $\Omega$ in the Probability context.

3.3.1 The positive and negative part of a function

In the context of constructing/working with integrals, it is convenient to be able to focus on functions with only one sign. To faciliate this, any function $f:\Omega \to \R$ can be naturally split up in two parts with that property:

The positive part of $f$ is the function $f^+:\Omega \to [0,\infty)$ obtained by keeping the positive values that $f$ has, while replacing the negative values by $0$ i.e. mathematically for any $x \in \Omega$ \[f^+(x) := \begin{cases} f(x) & \text{if $f(x) \geq 0$} \\ 0 & \text{if $f(x) < 0$} \end{cases} = \max\{f(x),0\}. \tag{3.2}\]
The negative part of $f$ is the function $f^-:\Omega \to [0,\infty)$ obtained by keeping the negative values that $f$ has but multiplied by $-1$ i.e. flipped in the horizontal axis, while replacing the positive values by $0$. So mathematically, for any $x \in \Omega$ \[f^-(x) := \begin{cases} -f(x) & \text{if $f(x)<0$} \\ 0 & \text{if $f(x) \geq 0$} \end{cases} = \max\{-f(x),0\}. \tag{3.3}\]

Note that both the positive and negative part are non-negative functions! Visually it is immediately clear what is going on, cf. Figure 3.2 (a)–Figure 3.2 (c).

You can easily gaze that we have the following very handy relationships (see also Figure 3.2 (d)):

Proposition 3.2 (Useful properties of positive and negative parts of a function) Given a function $f:\Omega \to \R$ with its positive and negative parts $f^+, f^-: \Omega \to [0,\infty)$ as defined in Equation 3.2 and Equation 3.3, we have the following:

$f(x)=f^+(x)-f^-(x)$ for all $x \in \Omega$,
$\lvert f(x) \rvert=f^+(x)+f^-(x)$ for all $x \in \Omega$.

Example 3.3 (Illustrating the role of positive and negative parts in integration) Of course, we’re introducing the positive and negative parts of a real valued function to prepare for the construction of Lebesgue integrals. But in fact, they play exactly the same role in the construction of the good old Riemann integral, so let’s quickly remind ourselves of that context to get some feel.

Indeed, you may or may not have this very sharp anymore, but for a function $f: [a,b] \to \R$ its Riemann integral is defined as \[\int_a^b f(x) \d x := \int_a^b f^+(x) \d x - \int_a^b f^-(x) \d x, \tag{3.4}\] where $f^+, f^-$ are the positive and negative parts given by Equation 3.2 & Equation 3.3. Doesn’t mean that for any given Riemann integral you’ve been asked to compute at some point in the past you actually used this definition, but that’s just because the rules we know for computing Riemann integrals don’t (always) require that. Here’s the rationale for this definition:

Defining/constructing/computing the integral of a non-negative function is easier for technical reasons (it avoids having to make decisions about how positive and negative values of a general real valued function $f$ should both be incorporated/balanced early on in the construction/computation).
If we take the lesson from the previous point and first make sure that we can do non-negative functions, then we can also do both $\int_a^b f^+(x) \d x$ and $\int_a^b f^-(x) \d x$. Obviously we do want to do the integral for $f$ as well if reasonably possible. For this, we know from Proposition 3.2 i that $f=f^+-f^-$. And since one of the core principles of integration is linearity (cf. Definition 3.1 i), Equation 3.4 is the only possible definition we could make at this point not violating that principle. Now Equation 3.4 is the natural place where positive and negative contributions can battle things out.

Here’s an explicit example. Take the function $f(x)=1/x$ for $x \not=0$ and say $f(0)=0$ (the value in $0$ is not relevant for the below, we just choose something). Cf. Figure 3.3. Let’s consider the good old Riemann integral for this function on some different intervals.

As you can readily check from Equation 3.2 & Equation 3.3, its positive and negative parts look as follows: \[f^+(x) = \begin{cases} 1/x & \text{if $x>0$} \\ 0 & \text{if $x \leq 0$} \end{cases} \quad \text{and} \quad f^-(x) = \begin{cases} -1/x & \text{if $x<0$} \\ 0 & \text{if $x \geq 0$} \end{cases}\]

First let’s look at integration over the interval $[a,b]=[0,1]$. There we get \[\int_0^1 f^+(x) \d x = \int_0^1 1/x \d x=\infty \quad \text{and} \quad \int_0^1 f^-(x) \d x = \int_0^1 0 \d x=0\] and hence Equation 3.4 gives us \[\int_0^1 f(x) \d x=\infty-0=\infty.\]

Next let’s consider $[a,b]=[-1,0]$: \[\int_{-1}^0 f^+(x) \d x = \int_{-1}^0 0 \d x=0 \quad \text{and} \quad \int_{-1}^0 f^-(x) \d x = \int_{-1}^0 -1/x \d x=\infty\] and hence in this case Equation 3.4 gives us \[\int_{-1}^0 f(x) \d x=0-\infty=-\infty.\]

Now let’s make it a bit more interesting by considering the interval $[a,b]=[-1,1]$. Then \[\int_{-1}^1 f^+(x) \d x = \int_{0}^1 1/x \d x = \infty \quad \text{and} \quad \int_{-1}^1 f^-(x) \d x = \int_{-1}^0 -1/x \d x=\infty.\] If we now again try to go back to Equation 3.4 to get the integral for $f$ we end up with $\infty-\infty$ which, as you’ll remember, cannot be given a (consistent) meaning i.e. is undefined. So we are forced to conclude that the integral \[\int_{-1}^1 f(x) \d x \tag{3.5}\] does not exist/is not well defined! This is one of these cases that motivates the rationale mentioned above, namely to carefully avoid balancing positive and negative values early on the construction and leave that for the final step Equation 3.4.

Arguing via the “integral is area of the region under the graph of $f$” principle: on the interval $[0,1]$ we have an area of $\infty$, on $[-1,0]$ we have an area of $-\infty$ (that’s how it works for negative functions!), and hence the area on $[-1,1]$ is undefined as it forces us to add both these areas i.e. to evaluate $\infty-\infty$.

3.3.2 The Riemann integral as the “average value” of a function

Each type of integral started its life somewhere small, as a desire in the mind of its creator to be able to measure something about functions. In the case of a Riemann integral, you’re probably most familiar with the interpretation that it measures the area under the graph of a (non-negative) function (cf. Figure 3.1). The goal of this section is simple: to try and convince you (if necessary!) that an alternative interpretation of the Riemann integral exists, namely that it measures the “average value” that a function takes (with a normalisation constant for the size of the domain) — cf. Proposition 3.7. The motivation for this bit of prep is that this interpretation naturally extends to and will be used as the key philosophy for constructing Lebesgue integrals in Section 3.4!

Yes that concept of “average value” of a function is absolutely a bit vague at the moment! You can approach it a bit more rigorously (cf. Remark 3.8) but my goal here is rather to convince you intuitively. Here’s the idea. Take a non-negative function $f: [a,b] \to [0,\infty)$. Then it has a range (cf. Definition 2.3) i.e. a subset in its co-domain filled with the real numbers that the function can actually visit/take. What we’re looking for is a proper sense of the “average of the real numbers in that range”.

Examples $>$ 1000 words!

Example 3.4 (Example of average value: constant functions) Let $f: [a,b] \to [0,\infty)$ be a constant function i.e. for some $c \in [0,\infty)$, $f(x)=c$ for all $x \in [a,b]$. Then the range of $f$ is simply the singleton $\{c\}$, and I’m sure you’ll agree that the only sensible interpretation of the “average value” of this function is, indeed, the value $c$!

Example 3.5 (Example of average value: piecewise constant functions) Next let’s look at piecewise constant functions $f: [a,b] \to [0,\infty)$.

First consider two explicit examples: $f_{1,2}: [0,5] \to [0,\infty)$ given by \[f_1(x)=\begin{cases} 1 & \text{if } x \in [0,2.5) \\ 2 & \text{if } x \in [2.5,5] \end{cases} \quad \text{and} \quad f_2(x)=\begin{cases} 1 & \text{if } x \in [0,1) \\ 2 & \text{if } x \in [1,5], \end{cases} \] see Figure 3.4 for plots.

Clearly the range of $f_1$ is the set $\{1,2\}$. So its “average value” is naturally equal to $1.5$ no? Agreed! But $f_2$ also has range $\{1,2\}$, should the “average value” be $1.5$ in that case as well? Hmm. That feels a bit unfair no? The difference with $f_1$ is that $f_2$ sends a lot more values from the domain $[0,5]$ to the value $2$ than to the value $1$, and you’d want to see that reflected in the “average value” no, in the same way that the average of the five numbers $1,1,2,2,2$ is not $1.5$ but a bit larger. Looking at the formula (or the plot) for $f_2$, it’s clear that the natural thing to do is to weigh each value in the range $\{1,2\}$ with how large the part of the domain is that is sent to that value. That yields \[\text{"average value" of } f_2=\frac{1}{5} \cdot 1+\frac{4}{5} \cdot 2=\frac{9}{5}\] (and of course, the “average value” of $f_1$ remains $1.5$ due to equal weights).

We can naturally write this idea down more generally: if $f:[a,b] \to [0,\infty)$ is a piecewise constant function of the form \[f(x)=\begin{cases} y_1 & \text{if } x \in I_1 \\ y_2 & \text{if } x \in I_2 \\ \, \vdots & \, \vdots \\ y_n & \text{if } x \in I_n \end{cases} \] for intervals $I_1, \ldots, I_n$ that form a partition (cf. Definition 1.23) of $[a,b]$, then \[\begin{align*} \text{"average value" of } f &= \frac{\text{length of } I_1}{b-a} y_1 + \frac{\text{length of } I_2}{b-a} y_2 \\ & \quad \quad + \ldots + \frac{\text{length of } I_n}{b-a} y_n. \end{align*} \tag{3.6}\]

Example 3.6 (Example of average value: linear functions) One more then — a linear function $f: [a,b] \to [0,\infty)$ i.e. of the form \[f(x)=cx+d\] for some constants $c,d \in \R$. For a plot, see Figure 3.5. In this case, as we see the range is an interval, namely $[f(a),f(b)]$. Because this is an uncountable set (cf. Definition 1.16), we drift a bit further away from our classic understanding of average as we know it (only) for a (finite/countable) sequence of values. Nevertheless, intuitively, every value in that range gets visited exactly once and it makes sense to think that the average value of this $f$ is exactly in the middle of the range i.e. the value $(f(a)+f(b))/2$ no?

Figure 3.5: Plot of the function $f$ used in Example 3.6

Fine, I hope that these examples give you a decent sense of what we have in mind when we talk about the “average value” of a function — at least for the simpler cases! Now let’s make the jump to the case of a general non-negative function as well as the link with the Riemann integral. The whole argument relies on a single visual observation.

Have a look at Figure 3.6 below. In the left hand column we have the plots we saw before in Example 3.5 & Example 3.6, with the area under the graph illustrated as well. To each of these plots we apply the following operation to obtain the corresponding plot on the right: we take a pair of scissors and some glue, and we reshape the area under the graph into a rectangle. And here is the key observation: in each case, the height $h$ of the graph on the right turns out to be exactly equal to what in Example 3.5 & Example 3.6 we decided to be the “average value” of the function in question!

Ok. If you agree with that message, then my claim is that we can do this in general! That is, take any (half decent) non-negative function $f:[a,b] \to [0,\infty)$ and imagine plotting it plus illustrating the area under its graph. By reshaping that area under the graph into a rectangle, this rectangle starts at $x=a$ and ends at $x=b$ and its height is very naturally understood/defined as the “average value” of $f$. If we now use that the area under the graph of $f$ equals the Riemann integral $\int_a^b f(x) \d x$ and equate that to the area of the rectangle, we get \[\int_a^b f(x) \d x=(b-a) \cdot \text{"average value" of } f\] i.e. here is our promised expression for the “average value” of $f$ in terms of its Riemann integral: \[\text{"average value" of } f = \frac{1}{b-a} \int_a^b f(x) \d x.\] Isn’t that very nice! :). If you’d like, I’d invite you to verify that this identity indeed holds in all the examples we have seen in Example 3.4–Example 3.6.

Proposition 3.7 (The Riemann integral as the “average value” of a function) Let $f:[a,b] \to [0,\infty)$ be a non-negative Riemann integrable function. Then we have/can naturally understand \[\text{the "average value" of } f = \frac{1}{b-a} \int_a^b f(x) \d x. \tag{3.7}\]

Remark 3.8. [⛔] A short play.

You: “Yeah, ehm, you know Kees, it’s a cute story with lovely visuals and everything, but I must honestly say that I do feel a bit cheated here, as if I walked into a wrong establishment and some shady smooth talking guy in a second hand suit and with too much gel in his hair is trying to sell me a second hand car for way too much money! I still haven’t seen an actual proper mathematical definition of that”average value” of a function you’re going on about, and hence I can’t actually prove that Proposition 3.7 holds! Do you know how much tuition fee I’m paying for this??”

Kevin the dog: “This is Kevin speaking, Kees just received your tuition fee and is busy buying even more Diet Coke. Thank you on his behalf. I will try and help you. Let me answer with a question: are you convinced that a Riemann integral gives the area under the graph?”

You: “Yes, of course!”

Kevin the dog: “Why? Because you have been told it so very often, by very confident teachers/lecturers that your naive teenage self trusted with your life? Or because you have seen an actual proof like you’re demanding here?”

You: “…”

Kevin the dog: “Well?”

You: “But surely it can’t be that difficult to actually prove that a Riemann integral gives the area under the graph!”

Kevin the dog: “Are you sure? I mean, don’t you for starters need a proper mathematical definition for the area under a graph? Exactly like you’re demanding one for the”average value” here?”

You: “Listen dog, I KNOW how to properly compute the area of shapes! All kinds of shapes! You can’t even hold a protractor in your paw!”

Kevin the dog: “Well no reason to get personal. It’s not as if you can properly sniff a dog bum with your nose all in the wrong place! Ok Mr Fancy Area Computer, then tell me, how do you properly compute the area of the blue shape in Figure 3.1?”

You: “…”

Kevin the dog: “Well? You can’t actually do it, can you?”

You walk off the stage frustrated, shaking your head in bewilderment wondering whether you’re actually dreaming or whether you really just have lost a mathematical argument with a dog.

Moral of the play: both statements “a Riemann integral gives the area under the graph” and Equation 3.7 have the same level of truth to them, as follows. Let $f:[a,b] \to [0,\infty)$ be a non-negative Riemann integrable function. Then, as you’ll remember (or see Wiki e.g.), modulo some technical details, we approximate $f$ (pointwise) by a sequence of piecewise constant functions, say $f_1, f_2, \ldots$ and we define its Riemann integral as \[\int_a^b f(x) \d x := \lim_{n \to \infty} \int_a^b f_n(x) \d x. \tag{3.8}\] Now, the best justification we have for the statement “a Riemann integral gives the area under the graph” is the following. For a piecewise constant function we can actually prove the statement, because the shape under its graph consists of a collection of rectangles and hence we can properly compute its area and verify that it’s equal to $\int_a^b f_n(x) \d x$. For a general function $f$, we look at this approximating sequence $f_1, f_2, \ldots$ and we postulate that the area under the graph of $f_n$ converges to the area under the graph of $f$ (or: define the area under the graph of $f$ as such). Together with Equation 3.8, the statement then also follows for $f$. Similarly, the “average value” of a piecewise constant function can be properly defined/understood, consistent with a weighted sequence of values (cf. Equation 3.6), and we can properly verify that Proposition 3.7 indeed holds for such functions. And then for general $f$ we need to again postulate that the “average value” of $f_n$ converges to “average value” of $f$ (or: define the “average value” of $f$ as such) and use Equation 3.8 to arrive at Proposition 3.7 for such $f$.

Remark 3.9. [⛔] Remember the Mean Value theorem? It states (in the integral form) that if $f:[a,b] \to [0,\infty)$ is a continuous function (which implies Riemann integrability), then there exists a $\xi \in (a,b)$ so that \[f(\xi)=\frac{1}{b-a} \int_a^b f(x) \d x. \tag{3.9}\] Comparing this one with Proposition 3.7, it is telling us: there exists a point $\xi$ in the interior of the domain where the value that $f$ takes coincides with the “average value” of $f$ — and hence of course the name Mean(=average) Value theorem!

The one explicit example of a continuous function we looked at in this section (well besides the constant one) is the linear function in Example 3.6. Recall that we convinced ourselves that the “average value” of $f$ is the value exactly in the middle of the range i.e. $(f(a)+f(b))/2$. For what argument does $f$ take that value? Well clearly for $\xi=(a+b)/2$. Cf. Figure 3.7. And indeed that choice also solves Equation 3.9!

Figure 3.7: Looking again at the linear function $f$ used in Example 3.6

Exercises

You can now do Exercise 3.2.

3.4 Lebesgue integrals

After these bits of prep, time to start delivering on that promise that we are going to be developing a shiny new type of integral! In this section we will construct it and discuss some properties. It will be a bit of work, here’s our starting point:

Context: let $(\Omega,\mathcal{F},\P)$ be a probability space (cf. Definition 1.50) and consider random variables i.e. measurable functions $X: \Omega \to \R$ on this space. (Note that we use the symbol “$X$” for such a function rather than “$f$” because we’re Probabilists at heart, but as argued before, that’s only cosmetics.)
Goal: construct the Lebesgue integral, which is a new type of integral i.e. an operator denoted by $I_L$ for now, that takes as input one of these measurable functions $X$ and gives as output $I_L(X)$ a number that we can interpret as the “average value” of the function $X$. trying to capture the same idea as in Subsection 3.3.2 i.e. in a proper sense a weighted average of the values that $X$ takes on the real line.

But doesn’t Proposition 3.7 with the good old Riemann integral already largely fulfill that goal? It does, but Riemann integrals only exist for functions that have (part of) the real line as domain and its construction fundamentally relies on this. We very deliberately want our new integral to be able to handle functions with any universe/set $\Omega$ as domain! We’ll develop this new Lebesgue integral independently from scratch, and then in Subsection 3.4.5 we’ll look back at the connection with the Riemann integral.

We follow a standard construction technique for the Lebesgue integral/$I_L$: we start by defining it for simple enough measurable non-negative functions $X: \Omega \to [0,\infty)$, where our philosophy that it should return the “average value” of $X$ is enough to unambiguously define it (cf. Subsection 3.4.1). Then we extend it to (almost) general measurable functions $X: \Omega \to \R$ on the basis of the fact that we want $I_L$ to have certain continuity and linearity properties (as formulated in Definition 3.1), and we’ll argue that the interpretation as “average value” of $X$ naturally comes along as well.

Take a deep breath and grab a Diet Coke, here we go!

3.4.1 Step 1: non-negative step functions

A step function (in this context) is one that has a finite range (cf. Definition 2.3), and also throwing non-negative in the mix then means that we’re talking about a measurable function $X: \Omega \to \R$ whose range, say $\{y_1,\ldots,y_n\}$ consists of non-negative numbers only. See Figure 3.8 for a visual, and recall that we also discussed such functions in Exercise 2.3!

Now, what value should we give our new integral, i.e. $I_L(X)$, for such an $X$ that reflects the “average value” of $X$? Maybe simply the average of the numbers $y_1,\ldots,y_n$? That’s a good first idea, but as in Example 3.5 we should also consider that potentially not all $y_i$’s are “equally important”. Look at Figure 3.8 again for instance. In Example 3.5 we were working with functions that had an interval $[a,b]$ as domain, and the natural way there to express that “importance” was to weigh each $y_i$ with the length of the subinterval that gets mapped to it. If we want to apply that same principle here, then we should weigh each $y_i$ with somehow the “size” of the corresponding inverse image in the domain $\Omega$ \[A_i := \{ X=y_i \}=X^{-1}(\{y_i\})\] (recall these guys from Chapter 2, in particular also that the fact that $X$ is measurable ensures that they are elements of $\mathcal{F}$, cf. e.g. Exercise 2.3). But we have, in this context of a probability space $(\Omega,\mathcal{F},\P)$, a very natural sense of “size” of these $A_i$’s, namely their probabilities $\P(A_i)$!

Lovely! So if we write down this weighted average of the $y_i$’s with weights $\P(A_i)$, we offically arrive at our first Lebesgue integral expression (horns sounding in the distance and things): \[I_L(X):=\sum_{i=1}^n y_i \P(A_i)=\sum_{i=1}^n y_i \P(X=y_i).\] Note that this epression is entirely the natural analogue of Equation 3.6, with maybe one point left to address: we haven’t included a normalisation factor analogue to the $1/(b-a)$ in Equation 3.6. But that is just because in this case, the analogue normalising factor is \[\frac{1}{\P(A_1) + \ldots + \P(A_n)},\] and since $A_1, \ldots, A_n$ forms a partition of $\Omega$ (as is the case in Figure 3.8, see also Exercise 2.3), we see that \[\P(A_1) + \ldots + \P(A_n)=\P(A_1 \cup \ldots \cup A_n)=\P(\Omega)=1\] (by countable additivity and that by def $\P(\Omega)=1$, cf. Definition 1.50 for both) i.e. that the normalising factor is simply $1$.

Definition 3.10 (The Lebesgue integral of a non-negative step function) On some probability space $(\Omega,\mathcal{F},\P)$ (cf. Definition 1.50), let $X: \Omega \to \R$ be a measurable function/random variable (cf. Definition 2.13) which is a non-negative step function i.e. has range $\{y_1,\ldots,y_n\} \subseteq [0,\infty)$ (cf. Definition 2.3). Then we define the Lebesgue integral of $X$, denoted by $I_L(X)$, as \[I_L(X):=\sum_{i=1}^n y_i \P(X=y_i) \in [0,\infty). \tag{3.10}\]

We can naturally understand/interpret $I_L(X)$ as the “average value” that $X$ takes on the real line.

Figure 3.8: Visualising a non-negative step function $X: \Omega \to \R$

We want to write down one convenient property — intuively obvious but the proof a bit annoying:

Proposition 3.11 (Monotonicity of the Lebesgue integral of non-negative step functions) Let $X,Y$ be measurable non-negative step functions as in Definition 3.10. If $X \leq Y$, then $I_L(X) \leq I_L(Y)$.

Note: recall that a statement like “$X \leq Y$” without further qualifiers is understood as pointwise, cf. Subsection 2.3.1.

Proof. It’s maybe not immediately obvious how to massage the assumption that $X(\omega) \leq Y(\omega)$ for all $\omega \in \Omega$ exactly to be able to compare $I_L(X)$ and $I_L(Y)$ as defined in Definition 3.10 — indeed that’s the annoying bit. Here’s one way forward.

For $X$ let its range be $\{y_1,\ldots,y_n\}$ with associated inverse images $A_i:=\{ X=y_i\}$ for $i=1,\ldots,n$ (which form a partition of $\Omega$, as mentioned above Definition 3.10), and similarly for $Y$ let its range be $\{z_1,\ldots,z_m\}$ with associated inverse images $B_j:=\{ Y=z_j\}$ for $j=1,\ldots,m$ (which then also form a partition of $\Omega$ of course).

Here’s the key idea. Fix some $B_j$ to focus on. We know $Y$ has the value $z_j$ there. The value that $X$ takes on $B_j$ is not necessarily constant, it can be any of the $y_i$’s in principle (if and only if $A_i \cap B_j \not=\emptyset$) but the assumption guarantees that we cannot see any $y_i$’s that are larger than $z_j$. If we now sum over all $B_j$’s to obtain the expression for $I_L(Y)$ then we cover the whole of $\Omega$. We must then also be able to find back all $A_i$’s during that journey, and hopefully using the observation of what happens on each $B_j$ be able to establish that $I_L(X) \leq I_L(Y)$.

Here’s how that looks in maths. Observe that since $A_1, \ldots, A_n$ is a partition of $\Omega$, we can cut any $\Omega_0 \subseteq \Omega$ up in disjoint parts that are contained in each of the $A_i$: \[\Omega_0 = \big( A_1 \cap \Omega_0 \big) \cup \big( A_2 \cap \Omega_0 \big) \cup \ldots \cup \big( A_n \cap \Omega_0 \big)\] and by countable additivity (cf. Definition 1.50) this yields \[\P(\Omega_0)=\sum_{i=1}^n \P(A_i \cap \Omega_0). \tag{3.11}\]

Now fix some $B_j$. Using Equation 3.11 we can write \[z_j \P(B_j) = z_j \sum_{i=1}^n \P(A_i \cap B_j)=\sum_{i=1}^n z_j \P(A_i \cap B_j). \tag{3.12}\] We give that right hand side a good solid stare, and we claim that for any $i=1,\ldots,n$ it holds that \[y_i \P(A_i \cap B_j) \leq z_j \P(A_i \cap B_j). \tag{3.13}\] Here’s why. If $A_i \cap B_j \not= \emptyset$, then for any $\omega \in A_i \cap B_j$ we have by def that $X(\omega)=y_i$ and $Y(\omega)=z_j$ and by assumption that $y_i = X(\omega) \leq Y(\omega)=z_j$, yielding Equation 3.13. Otherwise, if $A_i \cap B_j = \emptyset$, then Equation 3.13 trivially also holds since then both sides equal $0$.

Plugging Equation 3.13 into Equation 3.12 gives \[\sum_{i=1}^n y_i \P(A_i \cap B_j) \leq z_j \P(B_j)\] and summing over $j$ \[\sum_{j=1}^m \sum_{i=1}^n y_i \P(A_i \cap B_j) \leq \sum_{j=1}^m z_j \P(B_j). \tag{3.14}\] On the left hand side, changing the order of summation and again using Equation 3.11 (but now with $B_1,\ldots,B_m$ as partition) \[\begin{align*} \sum_{j=1}^m \sum_{i=1}^n y_i \P(A_i \cap B_j) &= \sum_{i=1}^n \sum_{j=1}^m y_i \P(A_i \cap B_j) \\ &= \sum_{i=1}^n y_i \sum_{j=1}^m \P(A_i \cap B_j) \\ &= \sum_{i=1}^n y_i \P(A_i) \end{align*} \] so that Equation 3.14 translates to \[\sum_{i=1}^n y_i \P(A_i) \leq \sum_{j=1}^m z_j \P(B_j).\] And indeed, plugging in Definition 3.10 this also reads as $I_L(X) \leq I_L(Y)$.

3.4.2 Step 2: any non-negative function

Next stop: any non-negative measurable function $X: \Omega \to \R$. Here we will skip the technical details, but the short version is this: it turns out that you can construct a sequence of non-negative step functions, say $X^{(n)}$ for $n=1,2,\ldots$ which is pointwise non-decreasing (i.e. $X^{(1)}(\omega) \leq X^{(2)}(\omega) \leq \ldots$ for all $\omega \in \Omega$) and has $X$ as pointwise limit i.e. \[X(\omega)=\lim_{n \to \infty} X^{(n)}(\omega) \quad \text{for all } \omega \in \Omega \tag{3.15}\] (note that we used this terminology before in Definition 2.33).

We know from Proposition 3.11 that the Lebesgue integrals of these step functions, $I_L(X^{(n)})$ for $n=1,2,\ldots$, forms a non-decreasing sequence of real numbers. And as you’ll remember (or see Wiki), such a sequence is guaranteed to have a non-negative limit, which may be $\infty$ though.

Given these facts, and that we want our new Lebesgue integral to have properties you’d expect every sensible concept of integration to have, which includes a sense of continuity (cf. property 2 in Definition 3.1), the only logical thing to do is to define the Lebesgue integral of $X$ as \[I_L(X) := \lim_{n \to \infty} I_L \left( X^{(n)} \right) \in [0,\infty] \tag{3.16}\] (note that this is along the same lines as how the Riemann integral is constructed, cf. Remark 3.8).

Just to highlight one particular aspect of the technical details we’re ignoring here: it needs not only to be shown that such a monotone approximating sequence of step functions can be constructed (cf. Remark 3.15 if you’re interested), but also that the limit Equation 3.16 is independent of the choice of sequence — otherwise Equation 3.16 would be a largely unworkable definition!

Final thing to say here: how about that “average value” interpretation of the Lebesgue integral?? For step functions we had a nice and firm justification in Definition 3.10. Here our argument is, inevitably (cf. Remark 3.8), slightly more loose, as follows. Are you happy to accept that “average value” should naturally have a notion of continuity in the sense that if the difference between two functions is marginally small that then the difference between their average values should also be marginally small? Note that this is the case for well establish notions of average, for instance if you take a sequence of numbers and change one of the numbers slightly, then the average also changes only slightly. If that’s ok with you, then we’re all good, because then you’ll agree that Equation 3.15 implies that the sequence of average values of the $X^{(n)}$’s converges to the average value of $X$. And since the former is the sequence of $I_L(X^{(n)})$’s, it follows from Equation 3.16 that $I_L(X)$ is then also the average value of $X$!

Definition 3.12 (The Lebesgue integral of a non-negative function) On some probability space $(\Omega,\mathcal{F},\P)$ (cf. Definition 1.50), let $X: \Omega \to \R$ be a non-negative measurable function/random variable (cf. Definition 2.13). Then we define the Lebesgue integral of $X$, denoted by $I_L(X)$, as \[I_L(X) := \lim_{n \to \infty} I_L\left( X^{(n)} \right) \in [0,\infty]:=[0,\infty) \cup \{\infty\}, \tag{3.17}\] where the $X^{(n)}$’s are non-negative, measurable step functions with the properties that

$X^{(1)} \leq X^{(2)} \leq \ldots$ (pointwise, cf. Subsection 2.3.1),
$X^{(n)} \to X$ as $n \to \infty$ pointwise (cf. Definition 2.33),

with their Lebesgue integrals $I_L(X^{(1)})$, $I_L(X^{(2)})$, $\ldots$ are defined in Definition 3.10.

Some notes:

There is a standard procedure for constructing such a sequence of step functions (cf. Remark 3.15), and furthermore this definition i.e. the limit value in Equation 3.17 does not depend on what sequence is chosen.
We can still naturally interpret $I_L(X)$ as the “average value” that $X$ takes on the real line.
This definition is “backwards compatible” with Definition 3.10: if $X$ here is a step function, then we can simply set $X^{(n)}:=X$ for all $n=1,2,\ldots$, so that Equation 3.17 yields the same result as Definition 3.10 does.
Obviously this definition is a lot less explicit than one in Definition 3.10. For the good old Riemann integral the situation is no different in principle (cf. Remark 3.8), but in that case we’ve seen over the years a whole array of tools to compute them, so that (normally) we never actually need to explicitly use the limit definition.

Example 3.13 (An example of a non-negative function with infinite Lebesgue integral) Note that Definition 3.12, and in particular Equation 3.17, explicitly allows for the possibility that a non-negative measurable function $X: \Omega \to \R$ on a probability space $(\Omega,\mathcal{F},\P)$ has an infinite Lebesgue integral i.e. $I_L(X)=\infty$. I thought it might be insightful to see an example of one of those!

First recall the cute identities \[\sum_{n=1}^m n = \frac{m(m+1)}{2} \tag{3.18}\] and \[\sum_{n=1}^\infty \frac{1}{n^2}=\frac{\pi^2}{6}. \tag{3.19}\]

Now, suppose that we can find events $A_1, A_2, \ldots$ that form a(n infinite) partition of $\Omega$ (cf. Definition 1.23) and with \[\P(A_n)=\frac{6}{\pi^2 n^2}\] (so that the sum of their probs equals $1$ due to Equation 3.19). If you need some convincing that such a case exists, then make things fully explicit by e.g. taking the probability space $([0,1],\mathcal{B}([0,1]),\P)$, with $\P$ the Lebesgue measure (recall from Example 1.58 e.g.) and cut the interval $[0,1]$ up in pieces of length $6/(\pi^2 n^2)$ for $n=1,2,\ldots$.

Further let $X: \Omega \to \R$ be the non-negative function that takes the value $n^3$ on $A_n$, for $n=1,2,\ldots$. That is: \[X(\omega)=\begin{cases} 1 & \text{if } \omega \in A_1 \\ 2^3 & \text{if } \omega \in A_2 \\ 3^3 & \text{if } \omega \in A_3 \\ \vdots & \quad \vdots \end{cases} \] Note that $X$ looks quite like a step function as defined in Definition 3.10, but isn’t really one since its range $\{1,2^3,3^3,\ldots\}$ is not finite. You can readily check that it’s indeed measurable, exactly because all the $A_1, A_2, \ldots$ are events.

Ok, let’s try and compute the Lebesgue integral of this guy. Since it’s non-negative but not a step function, we’ll need to go via Definition 3.12. For this, we’ll need to find a sequence of step functions as described in Definition 3.12 — any choice will do as noted in that def. Cunningly ;), that’s not so difficult here! Indeed, for any $n=1,2,\ldots$, define $X^{(n)}: \Omega \to \R$ by \[X^{(n)}(\omega)=\begin{cases} 1 & \text{if } \omega \in A_1 \\ 2^3 & \text{if } \omega \in A_2 \\ 3^3 & \text{if } \omega \in A_3 \\ \vdots & \quad \vdots \\ n^3 & \text{if } \omega \in A_n \\ 0 & \text{otherwise}. \end{cases} \] Clearly, $X^{(n)}$ has range $\{1,2^3,\ldots,n^3\}$, which is finite and consists of non-negative values, so it is indeed a non-negative step function as in Definition 3.10. Further we need to prove the pointwise monotonicity, but that’s also quite obvious, just look at $X^{(n)}-X^{(n-1)}$ and observe that it equals $n^3$ on $A_n$ while it equals $0$ on $A_n^c$.

Lovely, ok, then we get from Definition 3.12 that \[I_L(X) = \lim_{n \to \infty} I_L\left( X^{(n)} \right). \tag{3.20}\] Further, from Definition 3.10 we get for any $n=1,2,\ldots$ that \[\begin{align*} I_L\left( X^{(n)} \right) &= 0 \cdot \P \left( X^{(n)}=0 \right) + \sum_{i=1}^n i^3 \P \left( X^{(n)}=i^3 \right) \\ &= \sum_{i=1}^n i^3 \P(A_i) \\ &= \sum_{i=1}^n i^3 \frac{6}{\pi^2 i^2} \\ &= \frac{6}{\pi^2} \sum_{i=1}^n i \\ &= \frac{6}{\pi^2} \frac{n(n+1)}{2}, \end{align*} \] where the second equation uses that the inverse image $\{ X^{(n)}=i^3 \}$ equals $A_i$, and the final one uses Equation 3.18.

So, indeed Equation 3.20 yields \[I_L(X) = \lim_{n \to \infty} \frac{6}{\pi^2} \frac{n(n+1)}{2}=\infty.\]

Slightly annoying point to conclude this subsection with, sorry. There is one final, rather special case we want to mention. Normally we always limit ourselves to functions on $\Omega$ with co-domain $\R$. However, occasionally we can’t escape having to deal with functions that can take the value $-\infty$ or $\infty$ as well. Mostly this happens in the context of dealing with limits or supremums/infimums, as also briefly mentioned in the notes for Proposition 2.11. We also mentioned there that there is a mathematically proper way to deal with this, by including that possibility from the very beginning in the discussion. Because in this course we have to draw the line somewhere, we haven’t done that. Here we just want to very briefly highlight what this means for the Lebesgue integral, as follows.

Suppose that $X: \Omega \to \R \cup \{\infty\}$ is a non-negative measurable function which is also allowed to take the value $\infty$. Let $A=\{X=\infty\}$ be the event in $\Omega$ where $X$ takes that value $\infty$, and assume that $\P(A)>0$ (otherwise we don’t care). Then the only possible value for the Lebesgue integral of $X$ consistent with Definition 3.12 is to set $I_L(X)=\infty$. Here’s the argument. Consider an approximating sequence of non-negative step functions $X^{(1)}, X^{(2)}, \ldots$. Slightly adjust these to a new approximating sequence by setting $X^{(n)}(\omega)=n$ for $\omega \in A$ (if that wasn’t yet the case). Then we get from Definition 3.10 that \[I_L\left( X^{(n)} \right) \geq n \P(A)\] (as all terms in the right hand side of Equation 3.10 are non-negative) and hence from Equation 3.17 that \[I_L(X) \geq \lim_{n \to \infty} n \P(A)=\infty.\]

Definition 3.14 (The Lebesgue integral of a non-negative function with infinity as possible value) In the setting of Definition 3.12, if we consider a non-negative measurable function with extended co-domain $X: \Omega \to \R \cup \{\infty\}$ and $\P(X=\infty)>0$, then we set $I_L(X):=\infty$.

Remark 3.15. [⛔]. If you’re interested, there is a standard way to construct a sequence of step functions needed in Definition 3.12 to approximate a non-negative function $X: \Omega \to \R$, as follows.

Fix some $n=1,2,\ldots$. Partition the (effective) codomain $[0,\infty)$ of $X$ into $4^n+1$ parts/intervals, as follows: $B^{(n)}_k=[k/2^n,(k+1)/2^n)$ for $k=0,\ldots,4^n-1$ and $B^{(n)}_{4^n}=[2^n,\infty)$. Their inverse images $A^{(n)}_k:=X^{-1}(B^{(n)}_k)$ then form a partition of $\Omega$ (cf. Lemma 2.5). Now let $X^{(n)}$ be the step function that is constant on each $A^{(n)}_k$, taking the value $k/2^n$ there: \[X^{(n)}(\omega) = \sum_{k=0}^{4^n} \frac{k}{2^n} \mathbf{1}_{A^{(n)}_k}(\omega).\]

Let’s briefly convince ourselves that this construction works in the way intended. First note on each $A^{(n)}_k$, $X^{(n)}$ takes the value $k/2^n$ while $X$ has there as smallest value $k/2^n$ (by def of $B^{(n)}_k$), so indeed $X^{(n)} \leq X$ (pointwise and for all $n$). Fix any $\omega \in \Omega$ and set $y=X(\omega) \in [0,\infty)$. Then for all $n$ large enough (so that $2^n>y$), $y \in B^{(n)}_k$ for the $k$ that satisfies $k/2^n \leq y < (k+1)/2^n$. That means that $X(\omega)-X^{(n)}(\omega)=y-k/2^n<1/2^n$ which vanishes as $n \to \infty$, so indeed $X^{(n)} \uparrow X$ pointwise as $n \to \infty$. Finally (and here is where the reason for choosing these maybe awkward looking powers of $2$ becomes clear), if we make the step from $X^{(n)}$ to $X^{(n+1)}$, then each interval $[k/2^n,(k+1)/2^n)$ in the partition at step $n$ splits into two new intervals in the partition at step $n+1$, namely $[l/2^{n+1},(l+1)/2^{n+1})$ and $[(l+1)/2^{n+1},(l+2)/2^{n+1})$, where $l=2k$. This means that on the set in $\Omega$ where $X^{(n)}$ was constantly equal $k/2^n$, $X^{(n+1)}$ has two possible values: $2k/2^{n+1}$ and $(2k+1)/2^{n+1}$ i.e. $k/2^n$ and a value larger than $k/2^n$ but smaller than $(k+1)/2^n$. This shows that this particular way of “refining” the partition as $n$ grows ensures that $X^{(1)} \leq X^{(2)} \leq \ldots$ (pointwise).

3.4.3 Step 3: any function

Final stop then: any measurable function $X: \Omega \to \R$! In the previous two steps we have been focussing on non-negative functions so that we didn’t need to worry about balancing positive vs negative values and potential nasty subtleties in there. We’re going to take care of that now.

This is exactly why we did the bit of prep in Subsection 3.3.1, we’ll follow the same ideas as there (and hence, also the same ideas as used for the Riemann integral at the same construction stage, cf. Example 3.3). Given our measurable $X: \Omega \to \R$, applying Equation 3.2 and Equation 3.3 gives expressions for the positive part and negative part of $X$: the non-negative measurable (because cf. Proposition 2.11 iii) functions $X^+,X^-:\Omega \to \R$ given by \[X^+(\omega)=\max\{X(\omega),0\} \quad \text{and} \quad X^-(\omega)=\max\{-X(\omega),0\} \quad \text{for all $\omega \in \Omega$}. \tag{3.21}\] Note that the Lebesgue integrals of these two guys are already defined via Definition 3.12 (and in the very occasional case Definition 3.14), with some values \[I_L \left( X^+ \right) \in [0,\infty] \quad \text{and} \quad I_L \left( X^- \right) \in [0,\infty].\]

But of course we want the Lebesgue integral of $X$ itself! Recall from Proposition 3.2 that we have that $X=X^+-X^-$. Then, similar to what we did in Subsection 3.4.2, we appeal to a property we would like our new integral to have, again one that any sensible integral should have, namely linearity (cf. Definition 3.1). Since $X=X^+-X^-$, the only way to not immediately violate the desired linearity property is to define the Lebesgue integral of $X$ as \[I_L(X):=I_L \left( X^+ \right)-I_L \left( X^- \right). \tag{3.22}\] Wow, there we are! :). Given any measurable $X: \Omega \to \R$, work out its positive and negative parts, use Definition 3.12 to compute the Lebesgue integrals of each of these parts, plug these into Equation 3.22 to compute $I_L(X)$ and Bob’s your uncle!

Just one little possibly quite poisonous little snake wiggling through the lovely green grass: indeed that balancing of positive and negative values! To list the cases we can encounter, also consider the non-negative function $\lvert X \rvert: \Omega \to \R$ (of course, this is the function $\omega \mapsto \lvert X(\omega) \rvert$, and note that Proposition 2.11 iii again guarantees measurability). Its Lebesgue integral $I_L(\lvert X \rvert)$ is also defined via Definition 3.12 (and in the very occasional case Definition 3.14). Further we know from Proposition 3.2 that $\lvert X \rvert=X^++X^-$ which yields (it’s easily seen from Definition 3.12) \[I_L(\lvert X \rvert)=I_L \left( X^+ \right)+I_L \left( X^- \right). \tag{3.23}\]

Definition 3.16 (The Lebesgue integral of any function) On some probability space $(\Omega,\mathcal{F},\P)$ (cf. Definition 1.50), let $X: \Omega \to \R$ be any measurable function/random variable (cf. Definition 2.13).

Recall the non-negative measurable functions $X^+,X^-:\Omega \to \R$ (the positive and negative part of $X$) given by Equation 3.2 and Equation 3.3. Further also consider the absolute value $\lvert X \rvert:\Omega \to \R$, also non-negative and measurable. Observe that for all three, the Lebesgue integral is defined in Definition 3.12 (or Definition 3.14).

Then we have the following cases.

If $I_L(\lvert X \rvert)<\infty$ or equivalently $I_L(X^+)<\infty$ and $I_L(X^-)<\infty$ (cf. Equation 3.23), then we say that $X$ is Lebesgue integrable and we define the Lebesgue integral of $X$, denoted by $I_L(X)$, as \[I_L(X):=I_L \left( X^+ \right)-I_L \left( X^- \right). \tag{3.24}\] Observe that in this case, $I_L(X) \in \R$.
If $I_L(X^+)=\infty$ or $I_L(X^-)=\infty$ (but not both!), then $I_L(\lvert X \rvert)=\infty$ (cf. Equation 3.23). In this case we can still define the Lebesgue integral of $X$ using Equation 3.24 although we now have $I_L(X)=-\infty$ or $I_L(X)=\infty$.
If $I_L(X^+)=I_L(X^-)=\infty$, then also $I_L(\lvert X \rvert)=\infty$ (cf. Equation 3.23). In this case there is no way to make sense of Equation 3.24, in this case the Lebesgue integral of $X$ does not exist.

Some notes:

This definition is “backwards compatible” with Definition 3.12: if $X$ here is non-negative, then $X^+=X$ and $X^-=0$ (cf. Equation 3.21), and to evaluate the right hand side of Equation 3.24 we end up applying Definition 3.12 to $X^+=X$.
How about that “average value” of $X$? Recall from Definition 3.12 that we had convinced ourselves that we can naturally interpret $I_L(X^+)$ and $I_L(X^-)$ as the “average value” of $X^+$ and $X^-$ resp. If you are happy to accept that “average value” should naturally be linear (it is for sequences of numbers, as you can easily verify!) then since $X=X^+-X^-$ (cf. Proposition 3.2) and Equation 3.24, $I_L(X)$ can also naturally be interpreted as the “average value” of $X$!

Example 3.17 (The Lebesgue integral crash) So, on a probability space $(\Omega,\mathcal{F},\P)$ almost any measurable function $X: \Omega \to \R$ has a well defined Lebesgue integral, per Definition 3.16 the only exception are cases where $I_L(X^+)=I_L(X^-)=\infty$.

This can be caused merely as a consequence of the “edge case” Definition 3.14. For instance, if we are in a situation where there exist events $A,B$ with $\P(A)>0$ and $\P(B)>0$ so that $X(\omega)=\infty$ for all $\omega \in A$ and $X(\omega)=-\infty$ for all $\omega \in B$. It is intuitively already quite clear that there is no good way to balance $\infty$ and $-\infty$ against each other here. And indeed, $X^+$ will satisfy $X^+(\omega)=\infty$ for all $\omega \in A$ while $X^-$ will satisfy $X^-(\omega)=\infty$ for all $\omega \in B$. So applying Definition 3.14 to each of them indeed yields $I_L(X^+)=I_L(X^-)=\infty$ and Definition 3.16 slams the door in our face.

But there are more “normal” cases that are problematic as well — the analogue of Equation 3.5 in the Riemann case from Example 3.3! For example, consider again the setting from Example 3.13. Slightly adjust the $X$ we used there so that it takes the value $(-1)^n n^3$ on $A_n$, for $n=1,2,\ldots$. I.e.: \[X(\omega)=\begin{cases} -1 & \text{if } \omega \in A_1 \\ 2^3 & \text{if } \omega \in A_2 \\ -3^3 & \text{if } \omega \in A_3 \\ 4^3 & \text{if } \omega \in A_4 \\ \vdots & \quad \vdots \end{cases} \] If we try to find the Lebesgue integral of this hairy friend, then we first recognise that it is not a non-negative function, so we’ll have to use Definition 3.16. Working out the positive and negative parts, we see that \[X^+(\omega)=\begin{cases} 0 & \text{if } \omega \in A_1 \\ 2^3 & \text{if } \omega \in A_2 \\ 0 & \text{if } \omega \in A_3 \\ 4^3 & \text{if } \omega \in A_4 \\ \vdots & \quad \vdots \end{cases} \] and \[X^-(\omega)=\begin{cases} 1 & \text{if } \omega \in A_1 \\ 0 & \text{if } \omega \in A_2 \\ 3^3 & \text{if } \omega \in A_3 \\ 0 & \text{if } \omega \in A_4 \\ \vdots & \quad \vdots \end{cases}. \] Now we need to apply Definition 3.12 twice: once to compute $I_L(X^+)$ and once to compute $I_L(X^-)$. Both of these can be attacked in the same way as we dealt with the $X$ we saw in Example 3.13: you can use the same type of approximating sequence and compute the limit in the same way. Leaving the details up to you, but you’ll ultimately find that $I_L(X^+)=\infty$ and $I_L(X^-)=\infty$. So again Definition 3.16 refuses to serve us and the Lebesgue integral of $X$ does not exist! The reason is simply that the mix of positive and negative values that $X$ can take, and in particular how large these get as we get further into the sequence of events $A_1, A_2, \ldots$ (more precisely, that these grow faster than the probabilities of these events declines), beat the Lebesgue integrals ability to balance things out…

3.4.4 Step 4: proper notation

The last bit to wrap up this crash intro to Lebesgue integrals: obviously, given the amount of hard work we’ve done, we now also want a bit of properly majestic notation for a Lebesgue integral to replace that rather boring “$I_L(X)$” guy! And equally obviously, since it is an integral, we would very much like it to contain the “$\int$” symbol. So here we go then: if $X: \Omega \to \R$ is a measurable function/random variable on some probability space $(\Omega,\mathcal{F},\P)$ (cf. Definition 1.50), then we denote the Lebesgue integral of $X$ (provided it exists, which is not always the case as we saw in Definition 3.16) more fancily by \[\int_\Omega X \dP \tag{3.25}\] or sometimes also \[\int_\Omega X(\omega) \, \P(\mathrm{d}\omega). \tag{3.26}\] Pretty! :).

Observe that Equation 3.25 and Equation 3.26 carry three bits of information with them that are essentially parameters for the integral: the function $X$ that is being integrated, the domain $\Omega$ over which we integrate, and the probability measure $\P$ involved. They all play a role in the value of the integral. Indeed, we see $\P$ explicitly appearing in Definition 3.10 and the later defs build on Definition 3.10. The $\sigma$-algebra $\mathcal{F}$ does not appear as parameter/part of notation of the integral, and that makes sense: it’s part of the context since $X$ needs to be measurable (crucial for Definition 3.10 and hence later defs as well) but otherwise doesn’t influence the value of the integral: you could swap it for another $\sigma$-algebra and as long as $X$ remains measurable the integral (value) doesn’t change.

Note that the “$\omega$” in Equation 3.26 is a dummy variable (like we’re used in the Riemann context as well, recall from Remark 1.5) rather than a parameter that has meaning outside the integral as well. The notation Equation 3.26 is for instance handy if you want to integrate an “anonymous” function or if you don’t want to separately define the function the function you’re integrating. For instance if you want to integrate the function $\omega \mapsto \omega^2$ then you could either first define $X(\omega)=\omega^2$ and then use the notation $\int_\Omega X \d\P$, or you could just write $\int_\Omega \omega^2 \, \P(\mathrm{d}\omega)$.

Now, be a bit careful not to read too much into this — the temptation is there because of how very used you are to manipulating Riemann (and possibly other) integrals. Keep in mind that Equation 3.25 and Equation 3.26 are “just” symbols/notation, representing the Lebesgue integral of $X$ (with parameters $\Omega$ and $\P$). That’s all that it is. In particular, any techniques/tricks that you know for manipulating Riemann integrals should be left firmly to the side as we have no reason to think (unless explicitly stated and proven otherwise) that they apply to Lebesgue integrals as well! At this point, all we know about these Lebesgue integrals are the results we have established in Subsection 3.4.1–Subsection 3.4.3: all of Definition 3.10, Proposition 3.11, Definition 3.12, Definition 3.14 and Definition 3.16, in which all appearances of “$I_L(X)$” can now be replaced by one/both of Equation 3.25 & Equation 3.26. Of course, where we wrote “$I_L(X^{(n)})$” in Definition 3.12 we can now write “$\int_\Omega X^{(n)} \dP$” or “$\int_\Omega X^{(n)}(\omega) \, \P(\mathrm{d}\omega)$”, similar for the $X^+$, $X^-$ and $\lvert X \rvert$ appearing in Definition 3.16 etc.

Besides notational issues, there is one little bit of extra maths to introduce here. As we’re used to with Riemann integrals, even if $f$ is real valued function with say $\R$ as domain, sometimes you want to integrate $f$ over part of the domain, say an interval $[a,b]$ only. We express this by using $a$ and $b$ as bounds in the Riemann integral notation. Observe that this is equivalent to integrating a function over the whole of $\R$ that equals $f$ on $[a,b]$ but is $0$ otherwise.

We can do the analogue thing with Lebesgue integrals. If we take some event $A \subseteq \Omega$, and we want to integrate $X:\Omega \to \R$ over $A$ only then we write this by replacing the “$\int_\Omega$” in Equation 3.25 and/or Equation 3.26 by “$\int_A$”. Of course, in principle this is a separate integral from the one over the whole of $\Omega$, and all our defintions from the previous subsections apply from scratch². However it turns out that that works exactly as you would expect and in line with the Riemann experience: it is the same as integrating over the whole of $\Omega$ as long as you set $X=0$ on $A^c$. To formulate this mathematically, our reliable workhorse the indicator function (cf. Subsection 2.4.2) comes in very handy, the product $\mathbf{1}_A X$ does exactly the trick: \[\mathbf{1}_A(\omega) X(\omega)= \begin{cases} X(\omega) & \text{if } \omega \in A \\ 0 & \text{if } \omega \not\in A \end{cases} \] and so we can naturally define \[\int_A X \dP := \int_\Omega \mathbf{1}_A X \dP.\]

² I’m slightly cheating here, did you catch me?? To apply these definitions from scratch, you’d need to work with the “shrinked” space that has a $A$ as universe. That’s fine in principle, see e.g. Exercise 1.6 for how to “shrink” the $\sigma$-algebra $\mathcal{F}$ and it’s also straightforward how “shrink” $\P$. However there is a small technical problem: if we choose $A$ with $\P(A)<1$, then the “shrinked” measure is no longer a probability measure, and all our defs have been formulated assuming it is one! That’s easily resolved though — the Lebesgue integral theory that we’ve discussed can trivially be extended from probability spaces to more general measure spaces (cf. Definition 1.50)!

Definition 3.18 (Proper notation for the Lebesgue integral) If $X: \Omega \to \R$ is a measurable function/random variable on some probability space $(\Omega,\mathcal{F},\P)$, then we like to use one of the following two pieces of cute notation for the Lebesgue integral of $X$, previously known as $I_L(X)$: \[\int_\Omega X \dP \quad \text{or} \quad \int_\Omega X(\omega) \, \P(\mathrm{d}\omega).\]

Observe that we have all of Definition 3.10, Proposition 3.11, Definition 3.12, Definition 3.14 and Definition 3.16 available, with this new notation replacing the old $I_L(X)$ ones.

Further, for $A \in \mathcal{F}$, we can integrate $X$ over $A$ only rather than the whole of $\Omega$: \[\int_A X \dP := \int_\Omega \mathbf{1}_A X \dP. \tag{3.27}\]

3.4.5 Step 5: Lebesgue vs Riemann

We started this whole “new integral” discussion with our trusted friend the Riemann integral in Subsection 3.3.2, so it’s only appropriate that we come back to it now that we have established the Lebesgue integral! Here’s why. Recall that by very def, Riemann integrals are only equipped to handle functions that have (part of) the real line as domain. Our new Lebesgue integral can handle functions any any universe/set $\Omega$, provided we build a probability space $(\Omega,\mathcal{F},\P)$ around it (so that the function is measurable).

As we have seen several times before (see e.g. Example 1.58), we can also take an interval $[0,1]$ as our universe $\Omega$ to build a probability space around: $([0,1],\mathcal{B}([0,1]),\P)$, where $\mathcal{B}([0,1])$ is the collection of Borel sets in $[0,1]$ (cf. Definition 1.41) and for $\P$ we take the Lebesgue measure $\lambda$ (cf. Proposition 1.56).

This means that if you take any (half decent) function $f:[0,1] \to \R$, then we now have two concepts of integrating it: the Riemann integral \[\int_0^1 f(x) \d x\] and the Lebesgue integral (cf. Definition 3.18) \[\int_{[0,1]} f \d\lambda.\] Obviously, even though they’re both integrals, it’s a priori not at all clear how they relate to each other, they could very well have very different values. But, if we think back to the interpretation of these integrals we have discussed, namely as average value of the function, then we actually do have good reason to expect that they should be equal! Indeed, taking Proposition 3.7 and Definition 3.16, we would expect that \[\int_0^1 f(x) \d x = \text{the "average value" of } f = \int_{[0,1]} f \d\lambda.\]

And, quite beautifully, this turns out to be the case indeed! :). Cf. Proposition 3.19 below (see e.g. Theorem 5.1.2 in Stroock (1994) for a full proof if you’re interested). There are some technicalities going on there but don’t get too distracted by these, we just need to ensure that both the Riemann integral and Lebesgue integrals do indeed exist:

Proposition 3.19 (Lebesgue vs Riemann) Let $f:[0,1] \to \R$ be a bounded function and $\lambda$ the Lebesgue measure on the Borel sets $\mathcal{B}([0,1])$ (cf. Definition 1.41 & Proposition 1.56) Then:

The Riemann integral $\int_0^1 f(x) \d x$ exists if and only if $f$ is continuous on $[0,1]$ outside a set of Lebesgue measure $0$.
If the Riemann integral exists then $f$ is a measurable function on $([0,1],\mathcal{B}([0,1]),\lambda)$ and the Lebesgue integral (cf. Definition 3.18 & Definition 3.16) coincides with the Riemann integral i.e. \[\int_{[0,1]} f \d\lambda=\int_0^1 f(x) \d x. \tag{3.28}\]

Some notes:

This result remains true if we replace $[0,1]$ by any other interval $[a,b]$ (though the space $([a,b],\mathcal{B}([a,b]),\lambda)$ is not actually a probability space if $b-a \not= 1$, which we have assumed throughout to construct our Lebesgue integral, this is not an actual problem, cf. Subsection 3.4.6 below).
A nice side effect of this result: if we are in the situation of Equation 3.28, then we can employ our massive arsenal of Riemann tools to work out the integral!

Example 3.20 (Lebesgue vs Riemann: Lebesgue wins!) Well ok, maybe a slightly provocative title ;), but here’s what it’s trying to say: also just within the context of (bounded) functions on the domain $[0,1]$ (or any other interval on the real line), the Lebesgue integral is truly an extension of the Riemann integral, in the following sense. The above Proposition 3.19 tells us that if the Riemann integral exists, then Lebesgue=Riemann. But it turns out that Lebesgue can reach further: there exist bounded functions $f:[0,1] \to \R$ that are Lebesgue integrable but not Riemann integrable! And because we’re talking bounded functions, failure to be Riemann integrable cannot simply be due to difficulties with balancing positive and negative values (as in Example 3.3 for instance), it is something more fundamental.

Here is a famous example to illustrate this. Working on $([0,1],\mathcal{B}([0,1]),\lambda)$, let $f:[0,1] \to \R$ be the indicator function (cf. Subsection 2.4.2) of the rational numbers in the unit interval $[0,1]$ i.e. \[f(x)=\mathbf{1}_{\Q \cap [0,1]}(x)=\begin{cases} 1 & \text{if } x \in \Q \cap [0,1] \\ 0 & \text{if } x \not\in \Q \cap [0,1]. \end{cases} \] Note that $f$ is clearly bounded, and since $\Q \cap [0,1]$ is countable (cf. Definition 1.16 & Example 1.17 i) it is a Borel set (cf. e.g. Exercise 1.9) and hence $f$ is measurable as well (cf. Exercise 2.3 i e.g.).

For this function, the Riemann integral $\int_0^1 f(x) \d x$ does not exist. This follows from Proposition 3.19 i, since $f$ is not continuous anywhere on $[0,1]$, or if you’d prefer a more direct argument: the “lower sums” and “upper sums” in the construction of the Riemann integral (see Wiki e.g.) do not converge: if you partition the domain $[0,1]$ into subintervals, then each subinterval always contains both a rational and irrational number (since $\Q$ is dense in $\R$, cf. Remark 1.19) so the “lower sum” will always be $0$ while the “upper sum” will always be $1$.

The Lebesgue integral on the other hand does exist and is in fact quite easy to work out: indeed $f$ has range $\{0,1\}$ and is therefore a non-negative step function, so that Definition 3.10 applies: \[\begin{align*} \int_{[0,1]} f \d\lambda &= 0 \cdot \lambda(f=0) + 1 \cdot \lambda(f=1) \\ &= 0 \cdot 0 + 1 \cdot 1 =1, \end{align*} \] where we used that the inverse image $\{f=1\}$ is the event $\Q \cap [0,1]$ which has Lebesgue measure $0$ as it’s countable (cf. Proposition 1.56), and then its complement automatically has Lebesgue measure $1$.

I think that this example also nicely illustrates that Lebesgue’s idea to approach expressing the average value/integral of a function via its range rather than via its domain was a brilliant one! :).

Exercises

You can now do Exercise 3.3.

3.4.6 Step 6: a bit more about Lebesgue integrals [⛔]

Obviously there is an awful lot more to say about Lebesgue integrals than the construction & link with Riemann we’ve seen in the above subsections: (more) properties, (more) special cases etc. We’ll see some essential properties for our purposes in Section 3.5 below, but just for those of you who are particularly interested I’d like to highlight in this subsection a few more of those landmark points that any half decent intro to Lebesgue integration should at least touch upon! For a source and further reading if you’re interested, have a look at Stroock (1994) e.g.

Point 1: Lebesgue integrals on measure spaces. First of all, our whole construction in the above subsections was based on functions living on a probability theory context. I chose to do it that way to keep the cognitive load of the single chapter hopefully a little bit within reason. But, as some of you may already have guessed and is actually quite easy to see: the whole construction works equally well to define the Lebesgue integral $\int_\Omega X \d\mu$ for a measurable real valued function $X:\Omega \to \R$ on any measure space $(\Omega,\mathcal{F},\mu)$ (cf. Definition 1.50) — indeed all results in Subsection 3.4.1–Subsection 3.4.3 can be directly transferred! The only actual change is that in order to keep our interpretation as “average value” of $X$ on track, as in the Riemann case in Proposition 3.7 we need to include a normalising constant (assuming that $\mu(\Omega) \in (0,\infty)$, the case $\mu(\Omega)=0$ is degenerate and the case $\mu(\Omega)=\infty$ needs a slightly more subtle formulation): \[\text{the "average value" of } X = \frac{1}{\mu(\Omega)} \int_\Omega X \d\mu \tag{3.29}\] Indeed in the case of a probability measure we have that $\mu(\Omega)=1$ and hence no normalisation needed!

Point 2: The Law of the Unconscious Statistician. Next up is what I think is a genuinely beautiful connection. :). On some probability space $(\Omega,\mathcal{F},\P)$ with measurable function/random variable $X$, we have argued how the Lebesgue integral $\int_\Omega X \dP$ can be interpreted as the “average value” that $X$ takes in its co-domain $\R$. Ok. But, that “average value” of $X$ is all about how $X$ “behaves” on its co-domain right — I mean, it’s loosely speaking about the values it takes there, and with what probabilities, that’s in fact all that matters no? We had a very similar idea in Section 2.4, where we discussed that how $X$ “behaves” on the real line can be fully captured in one object: a probability measure $\mu_X$ on the co-domain $\R$, which turns the co-domain into a probability space $(\R,\mathcal{B}(\R),\mu_X)$ (cf. Definition 2.20). In a Probability context we call $\mu_X$ the distribution of $X$, and in a more general measure theory context we call it the push forward of $\P$ under $X$. And if it is really true that $\mu_X$ fully describes the behaviour of $X$ on the real line, and that the “average value” of $X$ is determined by its behaviour on the real line i.e. by $\mu_X$, and that the “average value” of $X$ equals $\int_\Omega X \dP$, then it would make sense to expect that there is a way to express $\int_\Omega X \dP$ in terms of $\mu_X$ no?? Hold that thought!

Note that we now have two probability spaces: $(\Omega,\mathcal{F},\P)$ which is a structure around the domain of $X$, and $(\R,\mathcal{B}(\R),\mu_X)$ which is a structure around the co-domain of $X$. And we have the function $X$ forming a bridge between the two spaces if you will. Now I need you to free your mind a little bit, to realise the following: letting go of any interpretation and context, just because $(\R,\mathcal{B}(\R),\mu_X)$ is a probability space (fact!) and our construction of Lebesgue integrals works for any probability space (fact!), we can also talk about the Lebesgue integral of (Borel) measurable functions $h: \R \to \R$ that have $(\R,\mathcal{B}(\R),\mu_X)$ as their domain i.e. \[\int_\R h \d\mu_X\] (fact!).

Ok, now remember that thought from a paragraph ago, which turns out to be spot on: we can express a Lebesgue integral over the domain of $X$, so of the form $\int_\Omega \ldots \dP$, equally well as a Lebesgue integral over the co-domain of $X$ i.e. of the form $\int_\R \ldots \d\mu_X$! Gasp! Indeed! Here is the exact formulation. Let $h: \R \to \R$ be a Borel measurable function. Then we have that³ \[\int_\Omega h(X) \dP=\int_\R h \d\mu_X \tag{3.30}\] (provided that both integrals exist, recall from Definition 3.16 that that is not always the case). Note that in the left hand side, we mean that we are integrating the random variable $Y=h \circ X$ i.e. $Y: \Omega \to \R$ given by $Y(\omega)=h(X(\omega))$. Of course, the identity i.e. $h(x)=x$ is a very valid choice, in which case the left hand side in Equation 3.30 simplifies to $\int_\Omega X \dP$.

³ You may well want to see this as the Lebesgue analogue of the change of variable/integration by substitution technique for Riemann integrals!

In fact, Equation 3.30 is not very difficult to prove, but it’s easiest by means of the Monotone Convergence Theorem, which we haven’t discussed yet. Fun fact is that Equation 3.30 is commonly known under the name Law of the Unconscious Statistician (for reasons I’m not entirely conscious of…).

Point 3: The Radon-Nikodym theorem. For this point and the next one, let’s introduce a little bit of notation. On some measurable space $(\Omega,\mathcal{F})$ on which two measures $\mu$ and $\nu$ live, we use the following terminology:

$\nu$ dominates $\mu$ (notation: $\mu \leq \nu$): $\mu(A) \leq \nu(A)$ for all $A \in \mathcal{F}$,
$\mu$ is absolutely continuous with respect to $\nu$ (notation: $\mu \ll \nu$): for any $A \in \mathcal{F}$, if $\nu(A)=0$ then also $\mu(A)=0$. That is, any event that $\nu$ considers an irrelevant null event, $\mu$ has the same opinion about. Like two children where $\nu$ is the bossy one and $\mu$ the follower: if $\nu$ says “I don’t want to play with that event because it’s, like, STUPID!” then $\mu$ quickly follows with “Yes I know I also don’t like it!”.
$\mu$ and $\nu$ are singular (notation: $\mu \perp \nu$): an $\Omega_0 \in \mathcal{F}$ exists so that $\mu(\Omega_0)=0$ and $\nu(\Omega_0^c)=0$. So $\mu$ and $\nu$ utterly disagree about anything in life, like an old married couple who have divided up the house $\Omega$ in two domains: the wife $\nu$ lives only in the dining room ($=\Omega_0$) while the husband $\mu$ lives only in the kitchen ($=\Omega_0^c$). The wife couldn’t care less about anything in the kitchen (for any $A \in \mathcal{F}$ with $A \subseteq \Omega_0^c$, $\nu(A) \leq \nu(\Omega_0^c)=0$, cf. Theorem 1.51 i) while the husband $\mu$ has that same opnion about anything in the dining room.

Example 3.21 (Examples of relationships between measures [⛔]) First make an observation: for two measures $\mu, \nu$ on some measurable space $(\Omega,\mathcal{F})$, absolute continuity and singularity are effectively mutually exclusive relationships: if $\mu \perp \nu$ and $\mu \ll \nu$, then $\mu$ is the trivial measure $\mu=0$. Indeed, take any $A \in \mathcal{F}$. Since, with $\Omega_0$ as in the def of $\mu \perp \nu$, \[A= \big( A \cap \Omega_0 \big) \cup \big( A \cap \Omega_0^c \big),\] by countable additivity \[\mu(A)=\mu(A \cap \Omega_0)+\mu(A \cap \Omega_0^c). \tag{3.31}\] Note that $\mu(A \cap \Omega_0) \leq \mu(\Omega_0)=0$ (by $\mu \perp \nu$). Further $\mu \perp \nu$ also implies that $\nu(\Omega_0^c)=0$. Since $\mu \ll \nu$, then also $\mu(\Omega_0^c)=0$ and hence $\mu(A \cap \Omega_0^c) \leq \mu(\Omega_0^c)=0$. So Equation 3.31 indeed yields $\mu(A)=0$.

Now, take for instance $(\Omega,\mathcal{F})$ with $\Omega=\{\omega_1,\ldots,\omega_4\}$ and for $\mathcal{F}$ the powerset $\mathcal{F}=\mathcal{P}(\Omega)$. Recall that measures on $\mathcal{F}$ are fully determined by the value they assign to the singletons $\{\omega_1\}$, $\ldots$, $\{\omega_4\}$ (cf. Example 1.54). With that in mind, consider the measures $\mu$, $\nu$ given by \[\mu(\{\omega_1\})=1, \quad \mu(\{\omega_2\})=2, \quad \mu(\{\omega_3\})=0, \quad \mu(\{\omega_4\})=0\] and \[\nu(\{\omega_1\})=2, \quad \nu(\{\omega_2\})=1, \quad \nu(\{\omega_3\})=1, \quad \nu(\{\omega_4\})=0.\] Then $\mu \ll \nu$, but not the other way around: $\nu \not\ll \mu$. Further it’s clear that they are not singular, either by the above observation or just direct from the definition. Can you think of other examples of measures on this space that are singular?

As another example, consider $(\R,\mathcal{B}(\R))$. We call a measure $\mu$ on this space purely atomic if there exists a countable set $D \subseteq \R$ so that $\mu(\{x\})>0$ for all $x \in D$ while $\mu(\R \setminus D)=0$. Consider also the Lebesgue measure $\lambda$. Then $\mu \perp \lambda$, because $\lambda(D)=0$ (since $D$ is countable, cf. Proposition 1.56) and $\mu(\R \setminus D)=0$.

We discuss this point and the one below in the context of the measure space $(\R,\mathcal{B}(\R),\lambda)$, where $\lambda$ is the Lebesgue measure, but they can be formulated for more general measure spaces as well (under an extra technical condition).

Now, let $\mu$ be another measure on $(\R,\mathcal{B}(\R),\lambda)$ which is finite i.e. $\mu(\R)<\infty$. Then the seminal Radon-Nikodym theorem states that the following two statements are equivalent:

$\mu \ll \lambda$;
There exists a real valued, non-negative function $f$ with $(\R,\mathcal{B}(\R),\lambda)$ as domain which is Lebesgue integrable i.e. $\int_\R \lvert f \rvert \d\lambda<\infty$ (cf. Definition 3.16) and so that \[\mu(A)=\int_A f \d\lambda \quad \text{for all } A \in \mathcal{B}(\R)\] (recall that we introduced integrating over a subset of the domain in Definition 3.18).

This $f$ is then called the density of $\mu$ with respect to $\lambda$, and allows to transfer a Lebesgue integral in terms of $\mu$ into a Lebesgue integral in terms of $\lambda$: if $h: \R \to \R$ is Borel measurable and Lebesgue integrable, then \[\int_\R h \d\mu = \int_\R hf \d\lambda. \tag{3.32}\] Just for clarity, note a crucial conceptual difference with the Law of the Unconscious Statistician above: here we have equality between two integrals on the same space, while Equation 3.30 is about equality between two integrals on different spaces (well in principle at least!).

This guy is of crucial importance in many ways. For instance in MathFinance, a fundamentally important technique is to perform a “change of measure” from one probability measure to another (on the same space, each prob measure being absolutely continuous with respect to the other). The Radon-Nikodym theorem (in the slightly more general formulation than we’ve listed it here) then shows how to quantify the relationship between these two probability measures, and (the slightly more general form of) Equation 3.32 how to relate Lebesgue integrals (which, as we’ll see in Section 3.5, are also expectations!).

But also for our purposes this result has a very nice implication! Suppose that $X$ is a continuous random variable on some probability space $(\Omega,\mathcal{F},\P)$. Then by definition (cf. Subsection 2.4.3) it has a pdf $f_X: \R \to \R$ which is Borel measurable, Riemann integrable and non-negative, and so that the distribution $\mu_X$ of $X$ satisfies (cf. Equation 2.23 & Proposition 2.22) \[\mu_X((-\infty,x])=\int_{-\infty}^x f_X(y) \d y \quad \text{for all } x \in \R\] Using a slight extension of Proposition 3.19 (we need to apply some limit arguments, but we haven’t established the tools yet — we’ll see them later in this chapter) we can rewrite this as a Lebesgue integral: \[\mu_X((-\infty,x])=\int_{(-\infty,x]} f_X \d\lambda \quad \text{for all } x \in \R.\] Further we can now borrow the argument from Subsection 1.6.2 to argue that this relationship in fact fully determines $\mu_X$, in the sense that we in fact have \[\mu_X(A)=\int_A f_X \d\lambda \quad \text{for all } A \in \mathcal{B}(\R).\] And this means, by the Radon-Nikodym theorem above, that $\mu_X$ is absolutely continous with respect to the Lebesgue measure $\lambda$, with density $f_X$! Further Equation 3.32 shows that for any Borel measurable and Lebesgue integrable $h: \R \to \R$ it holds that \[\int_\R h \d\mu_X = \int_\R hf_X \d\lambda\] (assuming that these Lebesgue integrals indeed exist, which we’ll automatically assume for the other integrals below as well). If we now finally combine this with Equation 3.30 from the Law of the Unconscious Statistician, then we get that \[\int_\Omega h(X) \dP=\int_\R hf_X \d\lambda.\] If the product $hf_X$ is nice enough be to be Riemann integrable, then a similar slight extension of Proposition 3.19 as we used above allows to write the right hand side as a good old Riemann integral: \[\int_\Omega h(X) \dP=\int_{-\infty}^\infty h(x)f_X(x) \d x. \tag{3.33}\] Recall that our key goal in this chapter is to develop a new notion of expectation of a random variable. We’ll argue in the next section that we want to use $\E[h(X)]:=\int_\Omega h(X) \dP$. And then indeed Equation 3.33 is nothing but the formula for expectation of a function of a continuous random variable that you know so well from earlier Probability courses (cf. Subsection 2.4.3)! :).

Note that in the above, it is actually a bit unnatural to involve Riemann integrals at all — we first have to translate from Riemann to Lebesgue and then back again. Indeed it would be a bit more clean to just formulate the condition for a random variable to be continuous and the integral formula for expectation flowing from that in terms of Lebesgue integrals only. It’s just that in a typical study trajectory, you get to see continuous random variables way before Lebesgue integrals are introduced, hence most/all of you will have that Riemann integral perspective!

While on the topic of random variables anyway, let’s also have a quick look at a discrete random variable $X$ on a probability space $(\Omega,\mathcal{F},\P)$. Recall from Subsection 2.4.1 that this means that $X$ has a countable range in its co-domain $\R$, let’s denote that set by $R_X$. Its distribution $\mu_X$ has then the property that $\mu_X(\{x\})>0$ for all $x \in R_X$ (well, slight edge case: it could be true that some elements of $x$ of $R_X$ have $\mu_X(\{x\})=0$ but then for the sake of this argument we can just shrink $R_X$ by kicking these out without hurting any conclusions) while $\mu_X(\R \setminus R_X)=0$. Hence, $\mu_X$ is what we called in Example 3.21 a “purely atomic” measure, and as we saw in that example $\mu_X$ is singular to the Lebesgue measure.

Further, for a Borel measurable function $h: \R \to \R$, the random variable $h(X)$ (recall: the mapping $\omega \mapsto h(X(\omega))$) obviously has countable range \[R_{h(X)}=\{ h(x) \, | \, x \in R_X \}.\] As we did for the continuous random variable above, let’s already take a little peak forward to when we introduce the new definition $\E[h(X)]:=\int_\Omega h(X) \dP$. In this case it is a lot less deep to see what $\int_\Omega h(X) \dP$ will be and can e.g. be done with just a bit of algebra to combine elements from the construction of Lebesgue integrals! It’s a bit of a hassle, but I thought I’d show the details just so that you can see it in action if you’d like.

Assume for the moment that $R_X$ (and hence also $R_{h(X)}$) is a finite set and that $h$ is a non-negative function i.e. all the elements of $R_{h(X)}$ are non-negative, then $h(X)$ is nothing but a non-negative step function as addressed in Definition 3.10 and hence we get immediately from that def \[\int_\Omega h(X) \dP=\sum_{z \in R_{h(X)}} z \P(h(X)=z)=\sum_{x \in R_X} h(x) \P(X=x). \tag{3.34}\] Note that for the second equality, if $h$ is a one-to-one function then $R_X$ and $R_{h(X)}$ have the same number of elements and the equality is just direct by linking up the values in $R_X$ with the ones in $R_{h(X)}$ so to speak. If that is not the case, then $R_{h(X)}$ may have fewer elements than $R_X$. However, if you take any $z \in R_{h(X)}$ that is hit multiple times by $h$ i.e. $\{x_1,\ldots,x_n\} \subseteq R_X$ satisyfing $h(x_i)=z$ for all $i=1,\ldots,n$, then \[\begin{align*} z \P(h(X)=z) &= z \P(X \in \{x_1,\ldots,x_n\}) \\ &= z \big( \P(X=x_1)+\P(X=x_2)+\ldots+\P(X=x_n) \big) \\ &= \sum_{i=1}^n h(x_i) \P(X=x_i), \end{align*} \tag{3.35}\] where we used countable additivity again. And hence if you walk through all terms in the sum \[\sum_{z \in R_{h(X)}} z \P(h(X)=z)\] then for any $z$ you can write down the elements in $R_X$ that get mapped to $z$ by $h$ and, if it’s more than one, use Equation 3.35. Gathering all resulting terms after you have completed your walk yields the ultimate right hand side of Equation 3.34 ⁴.

⁴ Maybe you had realised, but this argument is essentially the same as we’ve used several times before: exploiting very nice properties of inverse images (cf. Lemma 2.5 etc.), in this case of $h$. I just thought that it might be clearer here to write it out and not muddy the waters more than needs be

Now, if we drop the assumptions that $R_X$ is finite and $h$ is non-negative, then $h(X)$ is no longer a step function nor non-negative (necessarily). Looking at Definition 3.16, we need to look at the random variables $h(X)^+$ and $h(X)^-$ and use that by definition \[\int_\Omega h(X) \dP=\int_\Omega h(X)^+ \dP-\int_\Omega h(X)^- \dP.\] Observe that by definition, cf. Equation 3.21, the ranges of these random variables are \[R_{h(X)^+}=\left\{ \max\{z,0\} \, \left| \, z \in R_{h(X)} \right. \right\}\] and \[R_{h(X)^-}=\left\{ \max\{-z,0\} \, \left| \, z \in R_{h(X)} \right. \right\}.\] For each of them, if the resulting range is finite then we are back in the territory of a non-negative step function and Definition 3.10 yields \[\begin{align*} \int_\Omega h(X)^+ \dP &= \sum_{z \in R_{h(X)^+}} z \P(h(X)^+=z) \\ &= \sum_{z \in R_{h(X)}} \max\{z,0\} \P(h(X)=z) \end{align*} \tag{3.36}\] and \[\begin{align*} \int_\Omega h(X)^- \dP &= \sum_{z \in R_{h(X)^-}} z \P(h(X)^-=z) \\ &=\sum_{z \in R_{h(X)}} \max\{-z,0\} \P(h(X)=z), \end{align*} \tag{3.37}\] where in both cases the second equality can be justified using the same argument as we used under Equation 3.34. Otherwise, if either/both of them have infinite range (but still countable of course) then they are at least non-negative and hence we have to go through the pain of using Definition 3.12. In these cases, we can follow exactly the same construction as in Example 3.13 to derive that Equation 3.36 and Equation 3.37 still hold up — but we have to keep in mind that it is now well possible that either or both of Equation 3.36 and Equation 3.37 equal $\infty$!

Finally then, following Definition 3.16: if one or both of Equation 3.36 and Equation 3.37 is infinite, then $\int_\Omega h(X) \dP$ is either $-\infty$, $\infty$ or does not exist. If they are both finite, then \[\begin{align*} \int_\Omega h(X) \dP &= \int_\Omega h(X)^+ \dP-\int_\Omega h(X)^- \dP \\ &= \sum_{z \in R_{h(X)}} \max\{z,0\} \P(h(X)=z) - \sum_{z \in R_{h(X)}} \max\{-z,0\} \P(h(X)=z) \\ &= \sum_{z \in R_{h(X)}} \big( \max\{z,0\}-\max\{-z,0\} \big) \P(h(X)=z) \\ &= \sum_{z \in R_{h(X)}} z \P(h(X)=z) \end{align*} \] i.e. Equation 3.34 indeed holds!

Point 4: Lebesgue’s decomposition theorem. Yet another big hitter, truly a reel with only golden classics this subsection! Recall that at the start of Point 2 we introduced three possible relationships between measures and did a little bit of work with them in Example 3.21. Further in the above Point 3 we saw how on the measure space $(\R,\mathcal{B}(\R),\lambda)$ with $\lambda$ the Lebesgue measure, while considering distributions of random variables i.e. probability measures $\mu$ on this space, if $X$ is a continuous random variable then $\mu_X \ll \lambda$ while if $X$ is discrete then $\mu_X \perp \lambda$. In particular, as you can check, this also implies that a distribution/probability measure corresponding to a continuous random variable will always be singular to any probability measure/probability measure corresponding to a discrete random variable. Intuitively, in the discrete case the “natural domain” of $\mu_X$ are singletons and countable collections of singletons, while in the continuous case $\mu_X$ feels that these guys are irrelevant null events and it needs at least a full interval (so, uncountable!) to start getting excited and assign it a strictly positive value.

Lebesgue’s decomposition theorem is a seminal result which we formulate in the context of $(\R,\mathcal{B}(\R),\lambda)$ for convenience but covers more general measure spaces. It states that if $\mu$ is any finite measure on this space, then it can be “decomposed” in two components: one component that is absolutely continuous with respect to the Lebesgue measure and one that is singular to the Lebesgue measure. Quite similar to how any complex number consists of a real and imaginary component for instance. Mathematically: we can (uniquely) write $\mu=\mu_c+\mu_s$, where $\mu_c, \mu_s$ are measures on this space satisfying $\mu_c \leq \mu$, $\mu_c \ll \lambda$ and $\mu_s \perp \lambda$.

Now, we know already how to think about $\mu_c$: the fact that $\mu_c \ll \lambda$ implies that $\mu_c$ has a density with respect to $\lambda$ which fully expresses $\mu_c$ in terms of $\lambda$, cf. Point 3 above. About $\mu_s$ we know less, we have only seen that the “purely atomic” measures are examples of measures that are singular to the Lebesgue measure (cf. Example 3.21). A natural question then arises: are all measures singular to the Lebesgue measure of that “purely atomic” form? It’s surely tempting to think so — I mean it’s not easy to intuitively imagine that there is some other form possible! Note that if the answer were “yes”, then at the level of random variables (recall that there for any probability measure on this space, there exists a random variable which has that measure as its distribution, cf. Exercise 2.8) this would mean that any random variable is either continuous (if $\mu_X \ll \lambda$), or discrete (then $\mu_X \perp \lambda$), or it would be one from the “other” category (cf. Subsection 2.4.4) but then by Lebesgue’s decomposition theorem it’s still composed of a continuous and discrete one in some way.

But, I’m afraid that those who can’t resist the urge to press the “yes” button will have a trap door open underneath their feet and slide all the way down to the Dungeon of the Uncountability Devil! :(. Indeed the answer is actually no: there do exist probablity measures on $(\R,\mathcal{B}(\R),\lambda)$ that are singular to the Lebesgue measure yet not “purely atomic”! A famous example is the probability measure associated with (via Proposition 2.22) to the Cantor function, also appropriate called “the Devil’s Staircase”! :).

Point 5: Fubini & Tonelli’s theorems. Final point then! We won’t need it in this course, but in general quite a useful tool and it’s nice to see another analogy with Riemann integrals! Recall that for suitable functions $f:\R^2 \to \R$, we can also define the two dimensional Riemann integral (using the same construction idea as on $\R$, suitably generalised) \[\int_{-\infty}^\infty \int_{-\infty}^\infty f(x,y) \d (x,y).\] We know, one of these really nice results, that under some mild conditions such a two-dimensional integral can also be done as a repeated one-dimensional integral, and that the order in which you do that does not matter: \[\begin{align*} \int_{-\infty}^\infty \int_{-\infty}^\infty f(x,y) \d (x,y) &= \int_{-\infty}^\infty \left( \int_{-\infty}^\infty f(x,y) \d x \right) \d y \\ &= \int_{-\infty}^\infty \left( \int_{-\infty}^\infty f(x,y) \d y \right) \d x. \end{align*} \] Of course, we can see $\R^2$ as the product space $\R \times \R$ (Cartesian product, recall from Definition 1.12).

Now, we have also (briefly only) talked about product spaces in our context, cf. Example 1.54: if $(\Omega_1,\mathcal{F}_1,\mu_1)$ and $(\Omega_2,\mathcal{F}_2,\mu_2)$ are two measure spaces, then (under some mild technical conditions) we can construct the product space $(\Omega,\mathcal{F},\mu)$, where $\Omega=\Omega_1 \times \Omega_2$ consisting of the pairs $(\omega_1,\omega_2)$ for $\omega_i \in \Omega_i$, $\mathcal{F}$ is the $\sigma$-algebra on $\Omega$ generated by product sets $A_1 \times A_2$ for $A_i \in \mathcal{F}_i$, and $\mu$ is the (unique) measure on $\mathcal{F}$ satisfying $\mu(A_1 \times A_2)=\mu_1(A_1) \mu_2(A_2)$ for $A_i \in \mathcal{F}_i$.

On this product space we can then consider real valued, measurable functions $f: \Omega \to \R$, i.e. $f(\omega_1,\omega_2) \in \R$ for all $(\omega_1,\omega_2) \in \Omega$. And following Point 1, we can consider the Lebesgue integral \[\int_\Omega f \d\mu=\int_\Omega f(\omega_1,\omega_2) \, \mu(\mathrm{d}(\omega_1,\omega_2))\] (recall this alternative notation from Definition 3.18). And, beautifully analogue to the Riemann case above, Tonelli’s theorem and Fubini’s theorem each formulate some mild conditions under which this “double” integral can be worked out as a repeated “single” integral, with also here the order not mattering: \[\begin{align*} \int_\Omega f(\omega_1,\omega_2) \, \mu(\mathrm{d}(\omega_1,\omega_2)) &= \int_{\Omega_1} \left( \int_{\Omega_2} f(\omega_1,\omega_2) \, \mu_2(\mathrm{d}\omega_2) \right) \, \mu_1(\mathrm{d}\omega_1) \\ &= \int_{\Omega_2} \left( \int_{\Omega_1} f(\omega_1,\omega_2) \, \mu_1(\mathrm{d}\omega_1) \right) \, \mu_2(\mathrm{d}\omega_2). \end{align*} \] Note that these repeated integrals work in the same way as in the Riemann case. For instance for the one on the first line, you first fix any $\omega_1 \in \Omega_1$ and consider the function $\omega_2 \mapsto f(\omega_1,\omega_2)$ as a real valued measurable function on $(\Omega_2,\mathcal{F}_2,\mu_2)$ which we can integrate over $\Omega_2$. Doing this for all $\omega_1 \in \Omega_1$ means that we create a real valued measurable function \[\omega_1 \mapsto \int_{\Omega_2} f(\omega_1,\omega_2) \, \mu(\mathrm{d}\omega_2)\] on $(\Omega_1,\mathcal{F}_1,\mu_1)$ which is then integrated over $\Omega_1$ to finally get the end result.

3.5 A fresh look at the expectation of a random variable

There you are, you see, you have survived this whole fancy new Lebesgue integral construction — well done you and you can collect your “Official Lebesgue Engineer” certificate at the exit! Now it’s time to firmly get back into the Probability Theory domain. :). Recall, as discussed in Section 3.1, that our primary motivation for developing Lebesgue integrals was to solve our problem that we do not have a concept to express the expectation of random variables that works across the board. But now the resolution is staring us in the face!

Indeed, let $X$ be a random variable on some probability space $(\Omega,\mathcal{F},\P)$ (recall from Definition 1.50 & Definition 2.13). What do we actually want the expectation or mean of $X$ to represent? Well yes, it should be an “average value” of the values that $X$ takes in its co-domain $\R$, properly accounting for with what probabilities that happens right? But, as we saw in Section 3.4, cf. Definition 3.18 & Definition 3.16, that is also exactly how we can interpret the value of the Lebesgue integral $\int_\Omega X \dP$. And so it only makes an awful lot of sense to define the expectation of $X$, which consistent with the past we still denote by $\E[X]$, as \[\E[X]:=\int_\Omega X \dP, \tag{3.38}\] at least provided that the Lebesgue integral exists.

Always good to keep both feet firmly on the ground: observe that this “$\E$” guy is nothing but a function/mapping, with as domain the collection of all random variables on $(\Omega,\mathcal{F},\P)$ and as co-domain $\R \cup \{-\infty,\infty\}$ (though admitting that some inputs/arguments don’t get a value because their Lebesgue integral doesn’t exist). Just because it’s such a central character in our story it gets the fancy, dedicated symbol “$\E$” rather than a more generic “$f$” or “$g$” or whatever we otherwise tend to use as name for functions, and for historical reasons we use “$[]$” to enclose the argument in rather “$()$”. In particular, the argument of this function/mapping must always be a random variable because that’s the only input/argument it understands!

Having said that, both the expectation operator and the Lebesgue integral can take as input some “compound” expression, consisting of multiple random variables and/or Borel measurable functions, as long as that expression describes/defines (in “pointwise mode”, cf. Subsection 2.3.1) a (single) random variable. See the several examples of such expressions in Example 2.16 for instance. For example, if $X$ and $Y$ are random variables and $h: \R \to \R$ a Borel measurable function, then we can write \[\E[h(XY^2)]=\int_\Omega h(XY^2) \dP, \tag{3.39}\] which means that we apply the expectation operator as well as the Lebesgue integral to the random variable $h(XY^2)$, which is “pointwise mode” notation for the “anonymous” function $\Omega \to \R$ given by \[\omega \mapsto h \big( X(\omega)Y(\omega)^2 \big),\] and the equality holds because the definition Equation 3.38 applies to any random variable, including $h(XY^2)$! Obviously, alternatively we could also give that anynomous function a name, say $Z$, i.e. we could define the random variable $Z$ as $Z=h(XY^2)$ (again “pointwise mode” notation) and then Equation 3.39 can be written as $\E[Z]=\int_\Omega Z \dP$.

It may well be that the above two paragraphs are all very obvious to you, but if not then it’s very important to keep in mind — we’ll be using such “compound” expressions to define/describe random variables, both as arguments for the expectation operator and more generally, an awful lot going forward!

Definition 3.22 (New definition of expectation of a random variable) Let $(\Omega,\mathcal{F},\P)$ be a probability space (cf. Definition 1.50). For any random variable (cf. Definition 2.13) $X$ on this space, we define its expectation (or mean) as the Lebesgue integral of $X$: \[\E[X]:=\int_\Omega X \dP, \tag{3.40}\] provided that the integral exists. If it doesn’t exist, then $\E[X]$ also does not exist/is undefined.

Some notes:

In situations where we want to allow $X$ to take the value $-\infty$ and/or $\infty$ as well, this definition still works perfectly fine, recall Definition 3.14.
As discussed above, be aware that this definition automatically also covers any “compound” expression that specifies a random variable, such as in Equation 3.39.
Good to reiterate I think: the value of the integral in Equation 3.40 and hence of $\E[X]$ depends on the probability measure $\P$, in the sense that if we would replace it by a different probability measure (there are always infinitely many available, recall from Example 1.52) while keeping $\Omega$, $\mathcal{F}$ and the mapping $X: \Omega \to \R$ all the same, then the value of $\E[X]$ will generally change as well! This is apparent from Definition 3.10 for instance, as $\P$ there explicitly feeds into the expression for the integral.
Recall that the Lebesgue integral in the right hand side of Equation 3.40 had its notation introduced in Definition 3.18 and its construction/definition in Definition 3.16. For convenience, let’s (directly!) translate Definition 3.16 to this expectation context. Given $X$, the expectation of the non-negative random variables $X^+, X^-, \lvert X \rvert: \Omega \to \R$ (cf. Equation 3.21) i.e. (see previous comment!) \[\E[X^+]=\int_\Omega X^+ \dP, \quad \E[X^-]=\int_\Omega X^- \dP, \quad \E[ \lvert X \rvert]=\int_\Omega \lvert X \rvert \dP\] are guaranteed to exist, with values in $[0,\infty]:=[0,\infty) \cup \{\infty\}$. Further we have the relationship (cf. Equation 3.23) \[\E[ \lvert X \rvert] = \E[X^+]+\E[X^-] \in [0,\infty] \tag{3.41}\]

This gives rise to the following three possible cases.
1. If $\E[ \lvert X \rvert]<\infty$, or equivalently $\E[X^+]<\infty$ and $\E[X^-]<\infty$ (cf. Equation 3.41), then we say that $X$ is integrable and we have that \[\E[X]=\E[X^+]-\E[X^-] \in \R.\]
2. If either $\E[X^+]=\infty$ or $\E[X^-]=\infty$ but not both, then $\E[ \lvert X \rvert]=\infty$ (cf. Equation 3.41) yet $\E[X]$ still exists, with \[\E[X]=\E[X^+]-\E[X^-] \in \{-\infty,\infty\}.\]
3. Otherwise, if $\E[X^+]=\infty$ and $\E[X^-]=\infty$, then again $\E[ \lvert X \rvert]=\infty$ (cf. Equation 3.41) but now $\E[X]$ does not exist.

So here we are: a shiny sparkly brand new definition for the expectation of any random variable! The question that immediately comes up: we already had expressions for the expectation in the special cases that $X$ is either discrete or continuous, so how does new compare with old? Reassuringly, as we will see next, our new definition is consistent with the old ones, so what we have here is truly an extension of our existing knowledge. Note how this is also a crucial bump up for the validity of our new definition: if we would have found inconsistencies with our old expressions, then we would have to go back to the drawing board and think really hard whether we have made the right choices along the way!

It’s also nice to realise, I think, that this means that the seemingly very different expressions we had (a summation for discrete, a Riemann integral for continuous) are actually special cases of a single concept namely Lebesgue integrals! We don’t discuss the proof because time and because the argument for a continuous random variable needs some deeper properties of Lebesgue integrals. However see Subsection 3.4.6 and in particular Equation 3.33 and Equation 3.34 if you’re interested.

Proposition 3.23 (Consistency with old expressions for expectation) In the context of Definition 3.22, let $h: \R \to \R$ be a Borel measurable function (recal the sufficient condition for this from Proposition 2.11 ii for instance). Then we have the following.

If $X$ is a discrete random variable with countable range $R_X$ (cf. Subsection 2.4.1), then consistent with Equation 2.18 we have that \[\E[h(X)]=\int_\Omega h(X) \dP=\sum_{x \in R_X} h(x) \P(X=x)\] provided that the summation is well defined/exists.
If $X$ is a continuous random variable with pdf $f_X$ (cf. Subsection 2.4.3), then consistent with Equation 2.24 we have that \[\E[h(X)]=\int_\Omega h(X) \dP=\int_{-\infty}^\infty h(x) f_X(x) \d x\] provided that the Riemann integral is well defined/exists.

Very neat! It goes without saying that one consequence is that going forward, if you encounter a random variable and need to compute its expectation, then first of all try to see if it’s discrete or continuous, because if it is then you can just throw your old trusted expressions at it rather than having to stare at the Lebesgue integral that now defines expectations!

Finally in this section, the following. While Proposition 3.23 does open up a computational route in the special cases of discrete and continuous random variables, otherwise all we have at this point to work with expectations is just the definition as Lebesgue integral from Definition 3.22. That’s not much for such a very important tool for Probabilists! What we clearly need to do is to write down and prove a number of properties of expectations, to keep these in a prominent spot in our toolbox ready for use as we march on!

You will probably have seen before a good number of the properties we list below. However, depending on how your earlier Probability courses decided to do things exactly, you most likely won’t have seen proper/general proofs yet — after all, you most likely didn’t have a general definition for the expectation of a random variable available in the first place! Further, since our new definition of expectation really just defers to Lebesgue integrals, all properties below are also properties of Lebesgue integrals, and the proofs are mostly about manipulating Lebesgue integrals!

Theorem 3.24 (Important properties of expectations) Consider all the below on some probability space $(\Omega,\mathcal{F},\P)$ (cf. Definition 1.50), where $X, X_1, X_2$ are all random variables (cf. Definition 2.13) and expectations are defined as in Definition 3.22.

Linearity: for any $\alpha,\beta \in \R$ \[\E[\alpha X_1 + \beta X_2]=\alpha \E[X_1]+\beta \E[X_2]\] (provided that these expectations exist).
Non-negativity: if $X \geq 0$ a.s. then $\E[X] \geq 0$ (note that this includes $\E[X]=\infty$ as possibility).
Almost sure equivalence: $X_1=X_2$ a.s. if and only if $\E[ \lvert X_2-X_1 \rvert]=0$.
Jensen’s inequality: if $X$ is integrable (i.e. $\E[\lvert X \rvert]<\infty$, cf. Definition 3.22) and $f: \R \to \R$ is a convex function then \[f(\E[X]) \leq \E[f(X)].\] See e.g. Wiki for a reminder about convexity. Convenient special case: if $f$ is twice continuously differentiable, then convexity is equivalent to $f'' \geq 0$.
Markov’s inequality: if $X \geq 0$ and $a>0$, \[\P(X \geq a) \leq \frac{1}{a} \E \left[ \mathbf{1}_{\{ X \geq a\}} X \right] \leq \frac{1}{a} \E[X].\]
Minkowski’s inequality: if $p \in [1,\infty)$, \[\big( \E\left[ \lvert X_1+X_2 \rvert^p \right] \big)^{1/p} \leq \big( \E\left[ \lvert X_1 \rvert^p \right] \big)^{1/p} + \big( \E\left[ \lvert X_2 \rvert^p \right] \big)^{1/p}\] (provided the two expectations in the right hand side exist and are real valued). Note that for the case $p=1$ this reads as \[\E[\lvert X_1+X_2 \rvert] \leq \E[\lvert X_1 \rvert]+\E[\lvert X_2 \rvert],\] which is typically known as the triangle inequality.
Hölder’s inequality: if $p,q \in (1, \infty)$ so that $1/p+1/q=1$, \[\E[ \lvert X_1 X_2 \rvert] \leq \big( \E\left[ \lvert X_1 \rvert^p \right] \big)^{1/p} \big( \E\left[ \lvert X_1 \rvert^q \right] \big)^{1/q}\] (provided the two expectations in the right hand side exist and are real valued).
Independence: $X_1$ and $X_2$ are independent (recall from Section 2.5) if and only if for all bounded, Borel measurable functions $f,g: \R \to \R$ \[\E[f(X_1)g(X_2)]=\E[f(X_1)]\E[g(X_2)]. \tag{3.42}\] Further, if $X_1$ and $X_2$ are independent then Equation 3.42 holds for any Borel measurable functions $f,g: \R \to \R$, provided that all expectations exist.

Some notes:

Note that properties ii & iii make use of the “almost sure mode” we discussed in Subsection 2.3.1. For instance the statement “$X \geq 0$ a.s.” in part ii means that $\P(X \geq 0)=1$ i.e. that the inverse image $\{X \geq 0\}$ is an almost sure event (cf. Definition 1.55).
Two useful immediate consequences from properties i & ii are \[\text{if } X_1 \leq X_2 \text{ a.s., then } \E[X_1] \leq \E[X_2] \tag{3.43}\] (provided the expectations exist) and \[\text{if } X_1 = X_2 \text{ a.s., then } \E[X_1] = \E[X_2] \tag{3.44}\] (again provided the expectations exist). For Equation 3.43, observe that $X_1 \leq X_2$ a.s. implies that $X_2 - X_1 \geq 0$ a.s. Then property ii yields $\E[X_2 - X_1] \geq 0$ and property i that $\E[X_2] - \E[X_1] \geq 0$. For Equation 3.44, just apply Equation 3.43 twice to derive that $\E[X_1] \leq \E[X_2]$ and $\E[X_2] \leq \E[X_1]$ both hold.
Property iii, almost sure equivalence, may seem rather unassuming but it is actually expressing a fundamental fact. Recall that we argued earlier, in Subsection 2.7.3, that the non-negative number $\E [\lvert X_2-X_1 \rvert]$ can be seen as a “measure of distance” between the random variables $X_1$ and $X_2$, as judged by $\E$: the smaller the number, the smaller the distance (according to this measure/$\E$). With that interpration, property iii can be read as follows: $X_1$ and $X_2$ are identical in the eyes of $\E$ (because they have zero distance between them), if and only if $X_1=X_2$ a.s..

Of course, if $X_1$ and $X_2$ are (genuinely) identical i.e. $X_1=X_2$ pointwise, then we measure a distance of zero as you’d want: $\E [\lvert X_2-X_1 \rvert]=\E[0]=0$ (cf. Exercise 3.5). However this property shows that for $\E$ to consider them identical (zero distance), only a.s. equality i.e. only $\P(X_1=X_2)=1$ is needed. That is to say, $\E$ considers random variables that differ on a null event only as identical/interchangeable. And of course property ii, with Equation 3.43 and Equation 3.44, are manifestations of this same phenomenom.

This is also a key aspect motivating the practice in (advanced) Probability Theory to work in “almost sure mode” (as discussed in Subsection 2.3.1) pretty much by default: $\E$ as the central operator doesn’t care about more accuracy than that. It’s a bit like doing a long numerical computation, where you use a calculator that can do say 10 digit numbers only for part of it. Of course you could do the computation by hand with 20 or 30 or whatever number of decimals, but then when you switch to your calculator you lose that extra accuracy anyway!
Jensen’s inequality, property iv, is a very nice and handy tool. For instance, as you can readily check both $f(x)=x^2$ and $f(x)=\lvert x \rvert$ are convex functions, and plugging these in yield the regularly useful inequalities \[\big( \E[X] \big)^2 \leq \E \left[ X^2 \right] \quad \text{and} \quad \lvert \E[X] \rvert \leq \E[\lvert X \rvert]. \tag{3.45}\] Note that in Jensen’s inequality, integrability of $X$ is needed to guarantee that $\E[X] \in \R$ (cf. Definition 3.22) so that we know for sure that the expression $f(\E[X])$ is well defined. The other expectation in the inequality, $\E[f(X)]$, is guaranteed to exist but may well be $\infty$.

Proof. We’ll (fully) prove only a selection of the properties here, otherwise things would get way too much out of hand!

Property i. This is about linearity of Lebesgue integrals (using Definition 3.22): \[\int_\Omega \alpha X_1 + \beta X_2 \dP=\alpha \int_\Omega X_1 \dP + \beta \int_\Omega X_2 \dP\] (which is one the properties we said in Definition 3.1 that every sensible integral should have!). It is probably easiest to prove two separate statements, namely \[\int_\Omega \alpha X_1 \dP=\alpha \int_\Omega X_1 \dP\] and \[\int_\Omega X_1+X_2 \dP=\int_\Omega X_1 \dP+\int_\Omega X_2 \dP.\] Both statements can be proven from the very definition/construction of Lebesgue integrals. In each case, start working with non-negative step functions so that Definition 3.10 gives you an explicit expression for the Lebesgue integral to show your linearity result. Then move to non-negative random variables and use that linearity remains preserved through the limit in Definition 3.12 (see e.g. Wiki). Finally make the step to general random variables using Definition 3.16, where the key point is that Equation 3.24 preserves linearity.

Next we prove two “intermediate claims” that are helpful in the proofs of properties ii & iii. We use in this part several times the type of “probabilistic algebra” we practised in Exercise 3.3!

Intermediate claim 1. If $Z$ is a random variable which is a non-negative step function (as in Definition 3.10), then $Z=0$ a.s. if and only if $\E[Z]=0$.

Proof intermediate claim 1. Recall from Definition 3.10 that this $Z$ has a finite range $\{z_1,\ldots,z_n\} \subseteq [0,\infty)$ and that \[\E[Z]=\sum_{i=1}^n z_i \P(Z=z_i). \tag{3.46}\]

Let’s do this by proving both implications (cf. Definition 1.2 iv).

“$\implies$”: assume that $Z=0$ a.s. i.e. $\{Z=0\}$ is an almost sure event. Then its complement $\{Z>0\}$ (using that $Z$ is non-negative) is a null event (cf. Definition 1.55). Fix any $i=1,\ldots,n$ and consider $z_i \P(Z=z_i)$. If $z_i=0$ then $z_i \P(Z=z_i)=0$. If $z_i>0$, then $\{Z=z_i\} \subseteq \{Z>0\}$ and hence Theorem 1.51 i yields that $\P(Z=z_i) \leq \P(Z>0)=0$, so again $z_i \P(Z=z_i)=0$. In conclusion, each term in the summation in Equation 3.46 is zero, and hence $\E[Z]=0$.

“$\impliedby$”: now assume that $\E[Z]=0$. Because each term in the summation Equation 3.46 is non-negative, this implies that $z_i \P(Z=z_i)=0$ for all $i=1,\ldots,n$. And so, for any $i=1,\ldots,n$, if $z_i>0$ then $\P(Z=z_i)=0$. But since the probabilities in the summation must add up to $1$, it must mean that $0$ is an element of the range of $Z$ and that $\P(Z=0)=1$.

Intermediate claim 2. For any random variable $Y$ we have that \[Y \geq 0 \text{ a.s.} \iff \E[Y^-]=0. \tag{3.47}\]

Proof intermediate claim 2. Recall from Equation 3.21 that $Y^-$ is a non-negative random variable given by $Y^-=\max\{-Y,0\}$. Define the inverse image/event $A=\{Y<0\}$ and observe that for $\omega \in A$ we have that \[Y^-(\omega)=\max\{-Y(\omega),0\}=-Y(\omega)>0\] while for $\omega \in A^c$ we have that \[Y^-(\omega)=\max\{-Y(\omega),0\}=0.\] So in particular, we also have that $A=\{Y^->0\}$.

Further recall that since $Y^-$ is non-negative, Definition 3.12 tells us that there exists a sequence of non-negative step functions, say $Y^{(1)} \leq Y^{(2)} \leq \ldots$, that converge to $Y^-$ pointwise and that \[\E[Y^-]=\lim_{n \to \infty} \E \left[ Y^{(n)} \right]. \tag{3.48}\]

Let’s again prove both implications (cf. Definition 1.2 iv).

“$\implies$”: assume that $Y \geq 0$ a.s., which means that $A=\{Y<0\}=\{Y^->0\}$, as complement of the almost sure event $\{Y \geq 0\}$, is a null event (cf. Definition 1.55). In the approximation Equation 3.48, fix some $n$. Observe that for any $\omega \in A^c$, $Y^{(n)}(\omega) \geq 0$ because it’s non-negative but also $Y^{(n)}(\omega) \leq Y^-(\omega)= 0$, so $Y^{(n)}(\omega)=0$. That is, $A^c \subseteq \{Y^{(n)}=0\}$ and hence $\P(Y^{(n)}=0) \geq \P(A^c)=1$ (cf. Theorem 1.51 i). But then intermediate claim 1 above yields $\E [Y^{(n)}]=0$, and hence \[\E[Y^-]=\lim_{n \to \infty} 0=0.\]

“$\impliedby$”: now assume that $\E[Y^-]=0$. The sequence of real numbers $\E [Y^{(1)}]$, $\E [Y^{(2)}]$, $\ldots$ is non-negative (by construction, cf. Definition 3.10) and non-decreasing (cf. Proposition 3.11). So if any term in the sequence were strictly positive, then its limit would also be strictly positive, but we know that its limit is zero by our assumption. So it follows that $\E [Y^{(n)}]=0$ for all $n=1,2,\ldots$.

By intermediate claim 1, this means that $Y^{(n)}=0$ a.s. for each $n=1,2,\ldots$. Consider the almost sure events $A^{(n)}:=\{Y^{(n)}=0\}$ for all $n=1,2,\ldots$, and define their intersection to be \[\widehat{A}=\bigcap_{n=1}^\infty A^{(n)}.\] It follows from Exercise 1.12 that $\widehat{A}$ is an almost sure event as well. Observe that for any $\omega \in \widehat{A}$, we have that $Y^{(n)}(\omega)=0$ for all $n=1,2,\ldots$ and hence also \[Y^-(\omega)=\lim_{n \to \infty} Y^{(n)}(\omega)=0.\] That is, $\widehat{A} \subseteq \{Y^-=0\}$. By Theorem 1.51 i, $\{Y^-=0\}$ is an almost sure event since $\widehat{A}$ is. Recalling that $Y^-$ is non-negative, it follows that its complement i.e. $A=\{Y<0\}=\{Y^->0\}$ is a null event, and hence $\{ Y \geq 0\}$ indeed an almost sure event.

Property ii. It follows from Equation 3.47 (with $Y=X$) that $\E[X^-]=0$. Hence in Definition 3.22 we must be in case 1 or case 2, with \[\E[X]=\E[X^+]-\E[X^-]=\E[X^+] \in [0,\infty].\]

Property iii. For notational convenience, note that we may equivalently formulate it as \[X=0 \text{ a.s.} \iff \E[\lvert X \rvert]=0. \tag{3.49}\]

“$\implies$”: assume that $X=0$ a.s. Then also $X \geq 0$ a.s. (because $\{X=0\} \subseteq \{X \geq 0\}$), and hence by Equation 3.47 $\E[X^-]=0$. Further also $X \leq 0$ a.s. Setting $Y=-X$, we get $Y \geq 0$ a.s. and hence $\E[Y^-]=0$ from Equation 3.47. But clearly (recall Equation 3.21) \[Y^-=\max\{-Y,0\}=\max\{-(-X),0\}=\max\{X,0\}=X^+,\] so in fact $\E[X^+]=0$. Finally then from Equation 3.41 \[\E[\lvert X \rvert]=\E[X^+]+\E[X^-]=0+0=0.\]

“$\impliedby$”: assume that $\E[\lvert X \rvert]=0$. Since $\E[X^+],\E[X^-] \in [0,\infty]$, it follows from Equation 3.41 that $\E[X^+]=\E[X^-]=0$. Using that $\E[X^-]=0$ it follows from Equation 3.47 that $X \geq 0$ a.s. Set $Y=-X$. Using that $Y^-=X^+$ as seen in the previous step, we see that $\E[Y^-]=\E[X^+]=0$. And hence from Equation 3.47 we get that $Y \geq 0$ a.s. i.e. $X \leq 0$ a.s. So both $X \geq 0$ a.s. and $X \leq 0$ a.s. holds, which indeed implies that $X=0$ a.s. because \[\{X=0\} = \{X \geq 0\} \cap \{X \leq 0\}\] and the intersection of two almost sure events is again an almost sure event (cf. Exercise 1.12).

Property iv. Recall (e.g. from Wiki) that a convex function on $\R$ is continuous, differentiable in at least almost all points, and in points where it’s not differentiable its left and right derivative do exist. Fix some $x_0 \in \R$. If $f$ is differentiable in $x_0$, then convexity implies that $f$ dominates the tangent line in $(x_0,f(x_0))$ i.e. \[f(x) \geq f(x_0)+a(x-x_0) \quad \text{for all } x \in \R \tag{3.50}\] where $a=f'(x_0)$. If $f$ is not differentiable in $x_0$, then convexity implies that $f$ has an “upward kink” i.e. that its right derivative is strictly larger than its left derivative, in which case $f$ dominates any line through $(x_0,f(x_0))$ with slope between that left and right derivative, i.e. Equation 3.50 holds for multiple choices of $a$.

Now choose $x_0=\E[X]$ and some $a \in \R$ that makes Equation 3.50 work. Then it follows from Equation 3.50 that we have pointwise inequality between the random variables $f(X)$ and $f(x_0)+a(X-x_0)$: \[f(X) \geq f(x_0)+a(X-x_0)\] (indeed, for any $\omega \in \Omega$, just plug $X(\omega)$ for $x$ into Equation 3.50). Hence by non-negativity (property ii) \[\E[f(X)] \geq \E[f(x_0)+a(X-x_0)]\] and then by linearity (property i) \[\E[f(X)] \geq \E[f(x_0)]+a \big( \E[X]-\E[x_0] \big)\] and using Exercise 3.5 this simplifies to \[\E[f(X)] \geq f(x_0)+a \big( \E[X]-x_0 \big).\] Finally, plugging in that $x_0=\E[X]$ we indeed arrive at \[\E[f(X)] \geq f(\E[X]).\]

Property v. Leaving this one up to you, in Exercise 3.7!

Property vi & vii. These are not important enough in our course (they are in other contexts, hence I thought it’d be good to include them) to warrant a lengthy and detailed proof I think. If you’re interested, see e.g. Theorems 6.1.4 & 6.1.5 in Stroock (1994). Note that the triangle inequality \[\E[\lvert X_1+X_2 \rvert] \leq \E[\lvert X_1 \rvert]+\E[\lvert X_2 \rvert], \tag{3.51}\] which is one of those guys that is useful always and everywhere, is easy to prove independently. Indeed for any $\omega \in \Omega$ the triangle inequality for real numbers yields that \[\lvert X_1(\omega)+X_2(\omega) \rvert \leq \lvert X_1(\omega) \rvert+\lvert X_2(\omega) \rvert\] i.e. we have pointwise inequality between the random variables: \[\lvert X_1+X_2 \rvert \leq \lvert X_1 \rvert+\lvert X_2 \rvert.\] Then by non-negativity (property ii) and linearity (property i), Equation 3.51 follows.

Property viii. We prove the equivalence by proving both implications:

“$\impliedby$”: assume that \[\E[f(X_1)g(X_2)]=\E[f(X_1)]\E[g(X_2)]\] for all bounded, Borel measurable functions $f,g: \R \to \R$. Let $B_1, B_2$ be any Borel sets. Choose $f,g: \R \to \R$ as follows: \[f(x)=\begin{cases} 1 & \text{if } x \in B_1 \\ 0 & \text{if } x \not\in B_1 \end{cases} \quad \text{and} \quad g(x)=\begin{cases} 1 & \text{if } x \in B_2 \\ 0 & \text{if } x \not\in B_2. \end{cases} \] They are clearly bounded, and also Borel measurable — after all, they are indicator functions and so the arguments in Subsection 2.4.2 apply (be aware that the domain of these functions is $(\R,\mathcal{B}(\R))$, and that should hence be taken as $(\Omega,\mathcal{F})$ in the context of that section). So our assumption implies that with these choices we do have that \[\E[f(X_1)g(X_2)]=\E[f(X_1)]\E[g(X_2)]. \tag{3.52}\] Now observe that for any $\omega \in \Omega$ \[\begin{align*} f(X_1(\omega)) &= \begin{cases} 1 & \text{if } X_1(\omega) \in B_1 \\ 0 & \text{if } X_1(\omega) \not\in B_1 \end{cases} \\ &= \begin{cases} 1 & \text{if } \omega \in \{ X_1 \in B_1\} \\ 0 & \text{if } \omega \not\in \{ X_1 \in B_1\} \end{cases} \\ &= \mathbf{1}_{\{ X_1 \in B_1\}}(\omega) \end{align*} \] the second equality just by very definition of the inverse image $\{ X_1 \in B_1\}=X_1^{-1}(B_1)$ — which is an event i.e. an element of $\mathcal{F}$ since $X_1$ is a random variable and hence measurable (cf. Definition 2.13), and the final equality just by definition of the indicator function of that event (cf. Subsection 2.4.2).

So the random variables $f(X_1)$ and $\mathbf{1}_{\{ X_1 \in B_1\}}$ are identical. Similarly $g(X_2)$ and $\mathbf{1}_{\{ X_2 \in B_2\}}$ are identical. So Equation 3.52 implies that \[\E \left[ \mathbf{1}_{\{ X_1 \in B_1\}} \mathbf{1}_{\{ X_2 \in B_2\}} \right]=\E \left[ \mathbf{1}_{\{ X_1 \in B_1\}} \right] \E \left[ \mathbf{1}_{\{ X_2 \in B_2\}} \right]. \tag{3.53}\]

For notational convenience, define the event $A:=\{ X_1 \in B_1\} \cap \{ X_2 \in B_2\}$. Observe that for any $\omega \in \Omega$ \[\begin{align*} \mathbf{1}_{\{ X_1 \in B_1\}}(\omega) \mathbf{1}_{\{ X_2 \in B_2\}}(\omega) &= \begin{cases} 1 & \text{if } \omega \in \{ X_1 \in B_1\} \text{ and } \omega \in \{ X_2 \in B_2\} \\ 0 & \text{otherwise} \end{cases} \\ &= \begin{cases} 1 & \text{if } \omega \in A \\ 0 & \text{if } \omega \not\in A \end{cases} \\ &= \mathbf{1}_{A}(\omega) \end{align*} \] i.e. the random variables $\mathbf{1}_{\{ X_1 \in B_1\}} \mathbf{1}_{\{ X_2 \in B_2\}}$ and $\mathbf{1}_{A}$ are identical. So Equation 3.53 simplifies to \[\E \left[ \mathbf{1}_{A} \right]=\E \left[ \mathbf{1}_{\{ X_1 \in B_1\}} \right] \E \left[ \mathbf{1}_{\{ X_2 \in B_2\}} \right].\] Finally, using that the expectation of the indicator function of some event is just the probability of that event (cf. Equation 2.21, and recall that that “old” expression for expectation is still valid due to Proposition 3.23!) this translates to \[\P(A)=\P(X_1 \in B_1) \P(X_2 \in B_2).\] Since $B_1, B_2$ were arbitrary Borel sets, it indeed follows that $X_1$ and $X_2$ are independent (cf. Equation 2.31).

“$\implies$”: assume now that $X_1$ and $X_2$ are independent, and let $f,g: \R \to \R$ be Borel measurable. We need to show that \[\E[f(X_1)g(X_2)]=\E[f(X_1)]\E[g(X_2)], \tag{3.54}\] provided that all expectations exist.

For this, we’ll need to follow a constructive proof again, where we start with basic enough functions $f$ and $g$ and then gradually build that out to the general case. First assume that $f$ has range $\{y_1,\ldots,y_n\} \subseteq [0,\infty)$ and $g$ has range $\{z_1,\ldots,z_m\} \subseteq [0,\infty)$. Then the random variables $f(X_1)$ and $g(X_2)$ are non-negative step functions in the sense of Definition 3.10, and hence we get from that definition that \[\E[f(X_1)]=\sum_{i=1}^n f(y_i) \P(f(X_1)=y_i)\] and \[\E[g(X_2)]=\sum_{j=1}^m g(z_j) \P(g(X_2)=z_j).\] Further also the random variable $f(X_1)g(X_2)$ is a non-negative step function, whose range consists of all possible products of an element from the range of $f$ and one from the range of $g$, which yields according to Definition 3.10 \[\E[f(X_1)g(X_2)]=\sum_{i=1}^n \sum_{j=1}^m f(y_i)g(z_j) \P(f(X_1)=y_i \text{ and } g(X_2)=z_j).\] The key point now is that from the assumed independence it follows that (cf. Equation 2.31) \[\begin{align*} \P(f(X_1)=y_i \text{ and } g(X_2)=z_j) &= \P \big( X_1 \in f^{-1}(\{y_i\}) \text{ and } X_2 \in g^{-1}(\{z_j\}) \big) \\ &= \P \big( X_1 \in f^{-1}(\{y_i\}) \big) \P \big( X_2 \in g^{-1}(\{z_j\}) \big) \\ &= \P(f(X_1)=y_i) \P(g(X_2)=z_j) \end{align*} \] so that we may also write \[\E[f(X_1)g(X_2)]=\sum_{i=1}^n \sum_{j=1}^m f(y_i)g(z_j) \P(f(X_1)=y_i) \P(g(X_2)=z_j).\] Now Equation 3.54 indeed follows, using the general fact that for any sequences of numbers $a_1,\ldots,a_n$ and $b_1,\ldots,b_m$ it holds that \[\begin{align*} \left( \sum_{i=1}^n a_i \right) \cdot \left( \sum_{j=1}^m b_j \right) &= a_1 \cdot \sum_{j=1}^m b_j \\ &\qquad + a_2 \cdot \sum_{j=1}^m b_j + \ldots + a_n \cdot \sum_{j=1}^m b_j \\ &= \sum_{i=1}^n a_i \sum_{j=1}^m b_j \\ &= \sum_{i=1}^n \sum_{j=1}^m a_i b_j. \end{align*} \]

Then it is a matter of going through Definition 3.12 and Definition 3.16 to generalise Equation 3.54 to the general case.

Exercises

You can now do Exercise 3.4–Exercise 3.7.

3.6 The classic convergence theorems

Finally in this chapter, after all that Lebesgue integral and expectation violence, let us revisit the question of convergence of random variables. The setting is a probability space $(\Omega,\mathcal{F},\P)$ on which a sequence $X_1, X_2, \ldots$ of random variables is defined. Recall that in Section 2.7 we already established the main modes of convergence: different ways to give meaning to the idea of that sequence of random variables converging to some limit random variable $X$. Recall that we in particular discussed the “modes” pointwise convergence (cf. Definition 2.33), almost sure convergence (cf. Definition 2.34), and convergence in $L^r$ for some $r \geq 1$ (cf. Definition 2.37). Recall that this final guy involves the expectation operator — obviously from now onwards we understand in that definition, as well as in the results below plus indeed in the rest of our course going forward any mention of expectation/$\E$ to mean our shiny new one from Definition 3.22!

Since the expectation operator plays such a crucially important role in Probablity Theory, we often end up in a situation where we have such a sequence of random variables $X_1, X_2, \ldots$, we have a (candidate) limit random variable $X$, and we need to deduce that there is convergence of expectations (as well), while we are not (easily) able to verify convergence of expectations by direct computation. In that context fit the classic/seminal results of this section: they formulate certain conditions under which convergence of expectations is guaranteed to happen.

First let’s make the following observation. There are (at least) two possible ways to give meaning to that “convergence of expectations”, namely:

\[\lim_{n \to \infty} \E[ \lvert X_n-X \rvert ]=0 \tag{3.55}\] (i.e. $X_n \stackrel{L^1}{\longrightarrow} X$ as $n \to \infty$, as seen before in Definition 2.37);
\[\lim_{n \to \infty} \E[X_n]=\E[X], \tag{3.56}\] or, equivalently (just a sequence of real numbers converging to a real number, so Equation 2.34 applies) \[\lim_{n \to \infty} \big\lvert \E[X_n]-\E[X] \big\rvert=0.\]

A common mistake is to think that these two are equivalent/interchangeable, but that is not the case! By linearity (cf. Theorem 3.24 i) and Jensen’s inequality (cf. Equation 3.45) we have that \[\big\lvert \E[X_n]-\E[X] \big\rvert=\big\lvert \E[X_n-X] \big\rvert \leq \E[ \lvert X_n-X \rvert ],\] which shows that Equation 3.55 implies Equation 3.56, but the converse implication is not generally true!

Here is our first set of conditions (for a proof, see e.g. Theorem 3.3.2 in Stroock (1994)):

Theorem 3.25 (Monotone Convergence Theorem (MCT)) On some probability space $(\Omega,\mathcal{F},\P)$ (cf. Definition 1.50), let $X_1, X_2, \ldots$ be a sequence of non-negative random variables (cf. Definition 2.13) which is pointwise non-decreasing i.e. $0 \leq X_1 \leq X_2 \ldots$. Let $X:=\lim_{n \to \infty} X_n$ be its (pointwise) limit. Then $X$ is also a random variable and we have that \[\lim_{n \to \infty} \E[X_n]=\E[X].\]

Some notes:

Note that by assumption, we have for any $\omega \in \Omega$ that $X_1(\omega) \leq X_2(\omega) \ldots$ forms a non-decreasing sequence of real numbers which is indeed guaranteed to have a limit which we then define $X(\omega)$ to be. Keep in mind though that this limit may well be $\infty$, and hence to enjoy the full power of the MCT we do explicitly allow the limit $X$ to take value $\infty$ as well. This is no particular problem when it comes to $\E[X]$: since the $X_n$’s are non-negative, so will $X$ be, and hence Definition 3.14 tells us that if $\P(X=\infty)>0$ then $\E[X]=\infty$.
Although our formulation of the non-decreasing and limiting properties are in “pointwise mode”, they can be weakened to “almost sure mode” (cf. Subsection 2.3.1). See Exercise 3.12.

Example 3.26 (Simple example of the MCT resulting in infinities) Take some probability space $(\Omega,\mathcal{F},\P)$ and define constant random variables $X_1, X_2, \ldots$ as $X_n:=n$ for all $n=1,2,\ldots$. Then clearly all conditions in the MCT above are satisfied. Observe that for any $\omega \in \Omega$, \[X(\omega):=\lim_{n \to \infty} X_n(\omega)=\lim_{n \to \infty} n=\infty.\] That is, the limit $X$ is a random variables that even takes only the value $\infty$! Clearly, from Definition 3.14 we get that $\E[X]=\infty$.

No problem though — since $\E[X_n]=n$ (cf. Exercise 3.5) the conclusion of the MCT indeed holds.

Of course, a lot of the time you won’t be lucky enough to be staring at a non-decreasing sequence of random variables. In such situations, we have the following magnificent beast (for a proof, see e.g. Theorem 3.3.4 in Stroock (1994)):

Theorem 3.27 (Dominated Convergence Theorem (DCT)) On some probability space $(\Omega,\mathcal{F},\P)$ (cf. Definition 1.50), let $X_1, X_2, \ldots$ be a sequence of random variables and $X$ another random variable (cf. Definition 2.13) satisfying:

$X_n \stackrel{\text{a.s.}}{\longrightarrow} X$ as $n \to \infty$ (cf. Definition 2.34),
there exists an integrable random variable $Y$ (i.e. $\E[\lvert Y \rvert]<\infty$, cf. Definition 3.22) so that $\lvert X_n \rvert \leq Y$ a.s. for all $n=1,2,\ldots$.

Then we have that (cf. Definition 2.37) \[X_n \stackrel{L^1}{\longrightarrow} X \text{ as } n \to \infty,\] and hence in particular also \[\lim_{n \to \infty} \E[X_n]=\E[X].\]

Some notes:

If the conditions above hold, in particular condition ii, then this guarantees that all of $X_1, X_2, \ldots$ and $X$ are integrable random variables, i.e. all their expectations exist and are finite (cf. Definition 3.22).
Read condition ii above properly: we need to supply a single integrable random variable $Y$, so in particular also independent of $n$, which satisfies $\lvert X_n \rvert \leq Y$ a.s. for all $n=1,2,\ldots$. You can think about $Y$ as a sole kindergarten teacher preventing a whole class full of toddlers from going full anarchy and taking off their nappies to smear these in each other’s faces. Indeed the condition translates to, for any $n=1,2,\ldots$ \[-Y \leq X_n \leq Y \text{ a.s.}.\] So $Y$ provides some guard rails, a lowerbound and upperbound our sequence should stay in between (allowing some spillover as long as that happens on null events only). This prevents a situation in which as $n$ grows, the sequence attains increasingly large negative and positive values which can cause the conclusion above indeed to fail (for an example of this, see e.g. Exercise 3.9, in particular part v). Of course, this is not unrelated to the Lebesgue integral’s and hence our new definition of expectation’s vulnerability in such extreme scenarios, as earlier seen in Example 3.17 e.g.!
It’s nice, I think, to observe that we see here the “$\E$ can’t see differences between random variables that are limited to null events only” from Theorem 3.24 (“almost sure equivalence”, property iii) manifest itself in several ways:
- the conditions/assumptions, both the convergence and the integrable bound, are in “almost sure mode” rather than the stronger “pointwise mode”. This is good enough, because the conclusion is in terms of expectations;
- recall from the discussion in Subsection 2.7.2 that in the case of almost sure convergence, like we have in condition i above, the limit random variable is not at all uniquely defined. The random variable $X$ we’re using in condition i is only a limit, one of the many possible ones! So how is that even possible, given that we are using its expectation $\E[X]$ in the conclusion?? Well, we know from Exercise 2.14 that any other possible limit is a.s. equal to $X$. And because of that selective blindness of $\E$, any other possible limit has the same expectation as $X$ (cf. Equation 3.44). So it simply doesn’t matter, the statement of the theorem holds no matter which limit random variable you decide to use!

We conclude with a final convergence result, for your information only, that is less of a shiny limelight boy and more of an anonymous hard worker in the background: Fatou’s Lemma. What makes it stand out is that it requires very minimal assumptions. Of course the conclusion then also looks a bit less impressive, but it can be an invaluable tool especially in technical proofs. Before we formulate the result, a quick reminder (or introduction) of the concepts limit inferior and limit superior.

Remark 3.28. [⛔] Consider a sequence of numbers $a_1, a_2, \ldots$. Imagine walking through the sequence, and that you have arrived at step $n$. You look at the numbers still ahead of you (incl. the one you are currently at) i.e. $a_n, a_{n+1}, a_{n+2}, \ldots$. You are particularly interested in the supremum (smallest upperbound) and infimum (largest lowerbound) of the numbers you see ahead of you: \[s_n := \sup_{m \geq n} a_m \in (-\infty,\infty] \quad \text{and} \quad i_n := \inf_{m \geq n} a_m \in [-\infty,\infty). \tag{3.57}\]

By doing this at every step, i.e. for each $n=1,2,\ldots$, we create two new sequences: $s_1, s_2, \ldots$ and $i_1, i_2, \ldots$. Compare standing at position $n+1$ with standing at position $n$. At position $n$ you see one extra number ahead of you, namely $a_n$. That extra number can make your supremum larger (if $a_n$ is larger than all of the later ones) but certainly not smaller. That is: $s_n \geq s_{n+1}$. So $s_1, s_2, \ldots$ is a non-increasing sequence and therefore is guaranteed to have a limit, say $s_\infty \in (-\infty,\infty]$. Similarly $i_1, i_2, \ldots$ is a non-decreasing sequence and therefore also has a limit, say $i_\infty \in [-\infty,\infty)$.

For example:

Consider the sequence given by $a_n=(-1)^n$, so alternating $-1$ and $1$. Then no matter at which position $n$ you stand, ahead of you you still see numbers alternating between $-1$ and $1$. Therefore we have that $s_n=1$ for all $n=1,2,\ldots$ and hence $s_\infty=1$, while $i_n=-1$ for all $n=1,2,\ldots$ and hence $i_\infty=-1$.
Consider the sequence given by $a_n=1/n$. Standing at position $n$, you see ahead of you a decreasing sequence of numbers: $1/n, 1/(n+1), \ldots$. There is a largest one, namely $1/n$, but there is no smallest: the largest lower bound is $0$. So we have that $s_n=1/n$ for all $n=1,2,\ldots$ and hence $s_\infty=0$, while $i_n=0$ for all $n=1,2,\ldots$ and hence $i_\infty=0$.

Here’s the key point. As is clear from Equation 3.57, we have $s_n \geq i_n$ for all $n=1,2,\ldots$ and hence also $s_\infty \geq i_\infty$. Intuitively, if the sequence $a_1, a_2, \ldots$ would converge to a limit $a$, then the further along the sequence you walk, the closer the remaining values are concentrated near $a$. In particular they are also concentrated close to each other, and hence there is not so much difference between their supremum and infimum. That is, $s_n$ and $i_n$ get ever closer together, ultimately to leading to the same limit $s_\infty=i_\infty$.

It turns out that this is indeed exactly true. Keeping in mind that in general always $s_\infty \geq i_\infty$, it holds that the sequence $a_1, a_2, \ldots$ has a limit if and only if $s_\infty = i_\infty$. Further, if $s_\infty = i_\infty$ then the limit of $a_1, a_2, \ldots$ is $s_\infty = i_\infty$.

Indeed we can verify this for the above two examples: in the first one we have $s_\infty > i_\infty$ and the sequence has no limit, while in the second one we have $s_\infty=i_\infty=0$ and the sequence has $0$ as its limit.

This way of looking at sequences of numbers is important enough that $s_\infty$ and $i_\infty$ have their own names and symbols: $s_\infty$ is called the limit superior (or: limsup) of $a_1, a_2, \ldots$ and is denoted by \[\limsup_{n \to \infty} a_n,\] and $i_\infty$ is called the limit inferior (or: liminf) of $a_1, a_2, \ldots$ and is denoted by \[\liminf_{n \to \infty} a_n.\]

Theorem 3.29 (Fatou’s Lemma [⛔]) On some probability space $(\Omega,\mathcal{F},\P)$ (cf. Definition 1.50), let $X_1, X_2, \ldots$ be a sequence of non-negative random variables (cf. Definition 2.13). Define the random variable $X$ as (pointwise) \[X:=\liminf_{n \to \infty} X_n.\]

Then we have that: \[\E[X] \leq \liminf_{n \to \infty} \E[X_n].\]

Under an additional condition, there also exists an “inverse” of this result, in which liminfs get replaced by limsups and the inequality flips as well. For this, as well as for a proof of this result, see Theorem 3.3.3 in Stroock (1994) e.g.

Exercises

You can now do Exercise 3.8–Exercise 3.12.

3.7 Some exercises

About the exercises

Each exercise has a (rough) indication of its difficulty, as follows:

*	easier: can be solved by (almost) only using relevant definitions/results,
**	medium: in addition to relevant definitions/results, needs a limited amount of work/creativity,
*******	harder: in addition to relevant definitions/results, needs a larger amount of work/serious creativity,
⛔	non-examinable: either very challenging or too much out of scope — these exercises are intended for those of you who relish a bit extra!

The exam consists of mostly ** and *** level questions, with some * thrown in there for good measure.

A bit of preaching: it is an incredibly important part of the study process to try and work on the exercises as much as possible before the tutorials/without looking at the solutions, both for your understanding and to secure yourself a juicy exam mark! You will get stuck at times, that’s the name of the game for all of us — as the famous mathematician Paul Erdős used to say: a problem worthy of attack, proves its worth by fighting back. :). Use the hints given with many questions/talk to classmates/ask me and try again!

Full & detailed model solutions for the exercises will become available, below the exercises on this page, immediately after our tutorial.

Exercise 3.1 [⛔] In Definition 3.1, show that properties 1 and 2 indeed follow from the defining properties i–iii.

Hint (click to view)

For property 1, consider making use of linearity and non-negativity. For property 2, consider defining $g_n := f-f_n$ and inspect what properties the sequence of $g_n$’s has.

Full solution (click to view)

For property 1, note that $g-f \geq 0$ and hence by non-negativity (property ii), $I(g-f) \geq 0$. From linearity (property i) we get that $I(g-f)=I(g)-I(f)$ and hence indeed $I(g) \geq I(f)$.

Now for property 2. Define $g_n := f-f_n \geq 0$. Note that the $g_n$’s form a non-increasing sequence of non-negative functions. We get from continuity (property iii) that $I(g_n) \downarrow 0$ as $n \to \infty$. Since by linearity (property ii) $I(g_n)=I(f)-I(f_n)$ it follows that $I(f_n) \uparrow I(f)$ as $n \to \infty$.

Exercise 3.2 [*] For both functions below, sketch their graphs and try to intuitively deduce what their “average value” should be. Then check that your intuition is consistent (hopefully!) with Proposition 3.7.

$f:[0,2\pi] \to [0,\infty)$ given by $f(x)=2+\sin(x)$,
$f:[0,2] \to [0,\infty)$ given by \[f(x)=\begin{cases} x & \text{if } x \in [0,1] \\ 2-x & \text{if } x \in (1,2]. \end{cases} \]

Hint (click to view)

Not much in the way of hints I can do here — I can’t speak for your intuition ;) and I’m sure that you can work out the integral of these functions by yourself!

Full solution (click to view)

For part i, if you sketch the graph of $f$ you get of course the usual sine curve, curling around the level $y=2$. On the domain $[0,2\pi]$ we get exactly one period of the sine function. We also see that the range of $f$ is $[1,3]$. Because the graph is symmetric in the level $y=2$, or because every value $2+h$ in the range is compensated for by equally many visits to the value $2-h$ to put it like that, intuitively it makes a lot of sense to expect that the average value is $2$ no? Indeed, since $\int_0^{2\pi} f(x) \d x=4\pi$, Proposition 3.7 also yields an average value of $4\pi/(2\pi)=2$.

For part ii, a sketch shows a “tent shape”, consisting of a straight line from the point $(0,0)$ to the point $(1,1)$, and then a stright line again from the point $(1,1)$ to the point $(2,0)$. So the range of $f$ is $[0,1]$. Further each value in the range is visited twice (well except for $(1,1)$), once on the line up and once on the line down, suggesting they all have the same “weight”. So on these grounds it makes some sense to expect that the average value should be in the middle of the range i.e. the value $1/2$ no? Indeed again also what Proposition 3.7 yields!

Exercise 3.3 [**/***] This exercise intends to give you some (more) practice with ways of reasoning that are very common and useful in (advanced) Probability Theory. We’ll see it also pop up in the proof of Theorem 3.24 for instance, and later in the course often enough as well. I thought that you might find it helpful to see some of these on their own, without context that further competes for your attention.

Take as setting a random variable $X$ on a probablity space $(\Omega,\mathcal{F},\P)$. Your key tools for the below are:

inverse images of random variables also being events, cf. Definition 2.13;
the “almost sure mode” as discussed in Subsection 2.3.1, and associated with that the notion of almost sure events and null events, cf. Definition 1.55, as well as the key Exercise 1.12 for manipulating almost sure and null events;
properties of $\P$ as seen in Definition 1.50 & Theorem 1.51.

Try to work carefully through the below, they are exercises in “probabilistic algebra” — several statements may well be obviously true to you but that’s not the point, the challenge is rather to write down a water tight little proof in which you rigorously justify each and every step from assumption to conclusion, no magic/handwaving of any type allowed!

Show that if $X \geq 0$, then also $X \geq 0$ a.s.
Show that if $X > 0$ a.s., then also $X \geq 0$ a.s. Is the converse also true, i.e. that if $X \geq 0$ a.s. then also $X > 0$ a.s.?
Show that if $X \leq 0$ a.s. and $X \geq 0$ a.s., then $X=0$ a.s.
Show that $\lvert X \rvert \leq 1/n$ a.s. for all $n=1,2,\ldots$ if and only if $X=0$ a.s.

Hint (click to view)

In general, try to write down what the assumption means in basic terms, do the same for the conclusion we’re after, and then use any of the tools listed to carefully connect them. To give you an idea of what that could look like, here is the mode solution for part i — now you try to solve the other parts aong the same lines as well!

For part i, note that the assumption $X \geq 0$ does not come with the qualifier “a.s.” (or any other), so it is a statement in “pointwise mode” (cf. Subsection 2.3.1) i.e. it translates to $X(\omega) \geq 0$ for all $\omega \in \Omega$. Defining the inverse image/event \[A:=\{X \geq 0\}=\{ \omega \in \Omega \, | \, X(\omega) \geq 0\},\] the assumption we made simply means that $A=\Omega$.

What we need to show, $X \geq 0$ a.s., means that $\P(X \geq 0)=1$ i.e. that $\{X \geq 0\}$ is an almost sure event. But, by assumption that event is $A=\Omega$, and indeed $\P(\Omega)=1$ by def (cf. Definition 1.50).

Full solution (click to view)

For part ii, observe that for any $\omega \in \Omega$, if $X(\omega)>0$ then also $X(\omega) \geq 0$. In terms of inverse image/events this gives the relationship \[\{X > 0\}=\{ \omega \in \Omega \, | \, X(\omega) > 0\} \subseteq \{ \omega \in \Omega \, | \, X(\omega) \geq 0\}=\{ X \geq 0 \}.\] By Theorem 1.51 i, this inclusion yields the following relationship between the probabilities of these events: $\P(X>0) \leq \P(X \geq 0)$.

Now, by assumption $\{X > 0\}$ is an almost sure event i.e. it has probablity $\P(X>0)=1$. Then $\P(X \geq 0) \geq \P(X>0)=1$ i.e. $\{X \geq 0\}$ is also an almost sure event.

The converse, if $X \geq 0$ a.s. then also $X > 0$ a.s., is not generally true. For any $\omega \in \Omega$, if $X(\omega) \geq 0$ then either $X(\omega) >0$ or $X(\omega) =0$. In terms of inverse images/events: \[\{ X \geq 0 \}=\{ X>0\} \cup \{X=0\}.\] Since $\{ X>0\}$ and $\{X=0\}$ are disjoint events, we get from countable additivities (cf. Definition 1.50) that $\P(X \geq 0)=\P(X>0)+\P(X=0)$.

So, even if by assumption we know that $\P(X \geq 0)=1$ then this doesn’t imply that $\P(X>0)=1$. Indeed this depends on what $\P(X=0)$ is, which can be anything from $0$ to $1$!

For part iii, this one involves three inverse images/events: $\{X \leq 0\}$, $\{X \geq 0\}$ and $\{X=0\}$. How do these relate to each other? Well, for any $\omega \in \Omega$, if $X(\omega) \leq 0$ and $X(\omega) \geq 0$ both hold, then $X(\omega)=0$. And if $X(\omega)=0$, then $X(\omega) \leq 0$ and $X(\omega) \geq 0$ both hold. That is to say, we have that \[\{X=0\}=\{X \leq 0\} \cap \{X \geq 0\}.\] Further, we’re assuming that $\{X \leq 0\}$ and $\{X \geq 0\}$ are both almost sure events. This is a beautiful opportunity to put Exercise 1.12 to work: it tells us (among others) that the intersection of any two almost sure events is again an almost sure event. So it indeed follows that $\{X=0\}$ is an almost sure event as well!

For part iv, we can do this one for instance by proving both implications (cf. Definition 1.2 iv).

“$\impliedby$”: is the easy direction: assume that $X=0$ a.s. i.e. that $\{X=0\}$ is an almost sure event. Fix some $n=1,2,\ldots$ and observe that for any $\omega \in \Omega$, if $X(\omega)=0$ then also $\lvert X(\omega) \rvert \leq 1/n$. That is, $\{X=0\} \subseteq \{ \lvert X \rvert \leq 1/n\}$. But then, since $\{X=0\}$ is an almost sure event so is $\{ \lvert X \rvert \leq 1/n\}$, by Theorem 1.51 i.

“$\implies$”: the harder one! Assume that $\lvert X \rvert \leq 1/n$ a.s. for all $n=1,2,\ldots$. Now we need to find a way to express the inverse image/event $\{X=0\}$ in terms of the events $\{ \lvert X \rvert \leq 1/n \}$. Intuitively, as $n$ grows and $1/n$ gets smaller, the events $\{ \lvert X \rvert \leq 1/n \}$ “shrinks” and gets ever closer to the event $\{X=0\}$. So what we need here is a sense of limit of events, of which there are two basic types: infinite intersections and infinite unions (cf. Definition 1.10). In general, you want a union if your events get larger and grow towards a limit, and an intersection of your events get smaller and shrink towards a limit. So in this case, we’re intuitively expecting/hoping that \[\{X=0\} = \bigcap_{n=1}^\infty \{ \lvert X \rvert \leq 1/n \}. \tag{3.58}\] If we can show this, then it follows from our assumption and Exercise 1.12 that indeed $\{X=0\}$ is an almost sure event as well.

Now, in order to show Equation 3.58 we can for instance prove both inclusions (cf. Remark 1.9):

“$\subseteq$”: assume that $\omega \in \{X=0\}$ i.e. that $X(\omega)=0$. For any $n=1,2,\ldots$ fixed, then also $\lvert X(\omega) \rvert \leq 1/n$ i.e. $\omega \in \{ \lvert X \rvert \leq 1/n \}$. Hence $\omega$ is also an element of the right hand side of Equation 3.58 (recall Equation 1.5 e.g.).

“$\supseteq$”: assume that $\omega$ is an element of the right hand side of Equation 3.58. This implies (cf. Equation 1.5 e.g.) that $\omega \in \{ \lvert X \rvert \leq 1/n \}$ for all $n=1,2,\ldots$ i.e. that $\lvert X(\omega) \rvert \leq 1/n$ for all $n=1,2,\ldots$. Since $1/n \downarrow 0$ as $n \to \infty$, this implies that $\lvert X(\omega) \rvert \leq 0$ and hence $X(\omega)=0$. (Because weak inequalities are preserved through the a limit, or argue that $\lvert X(\omega) \rvert > 0$ leads to a contradiction because then there exsists an $n=1,2,\ldots$ so that $\lvert X(\omega) \rvert > 1/n$). So indeed $\omega \in \{X=0\}$.

Note: for completeness/clarity, this proof is a bit longer than needs be. Indeed it is sufficient to establish Equation 3.58 with a “$\supseteq$” only and appeal to Theorem 1.51 i. Further, in this case you could also use Theorem 1.51 v rather than Exercise 1.12 because we have here a non-increasing/“shrinking” sequence of almost sure events. The result from Exercise 1.12 is in particular powerful because it doesn’t need any such relationship, an intersection of any almost sure events will do!

Exercise 3.4 [*] On some probability space $(\Omega,\mathcal{F},\P)$, let $X$ and $Y$ be two random variables such that $\E [\lvert X-Y \rvert]=0$. Explain how it immediately follows from Theorem 3.24 that $\E[X]=\E[Y]$ (provided their expectations exist).

Hint (click to view)

Really, all you need is readily available in Theorem 3.24!

Full solution (click to view)

From almost sure equivalence, property iii in Theorem 3.24, we get that $X=Y$ a.s., and then Equation 3.44 implies that $\E[X]=\E[Y]$.

Exercise 3.5 [**] On some probability space $(\Omega,\mathcal{F},\P)$, let $X$ be a constant random variable i.e. $X=c$ for some $c \in \R$.

Use Proposition 3.23 to show that $\E[X]=c$.
Now show that $\E[X]=c$ using Definition 3.22 and earlier results only.

Hint: consider the cases $c<0$ and $c \geq 0$ separately.
Suppose that we weaken the assumption somewhat, to $X=c$ a.s. Is $X$ now a discrete, a continuous, or an “other” random variable (mind the gap! ;))? Show that we still have $\E[X]=c$.

Hint (click to view)

For part i, once you realise that the range of $X$ is $\{c\}$ the result follows easily!

For part ii, observe that Definition 3.22 expresses the expectation as a Lebesgue integral: $\E[X]=\int_\Omega X \dP$. So it’s all about working out this integral! The easier case is $c \geq 0$, because then you can immediately apply Definition 3.10. The case $c<0$ is a little bit more work but also not too bad. Now we’ll need to follow the lead from Definition 3.16: first work out what the positive and negative parts of $X$ are (recall their defs from Equation 3.21), then work out their Lebesgue integrals for which you can use Definition 3.10 again, and then put these integrals together to get the integral of $X$ as Definition 3.16 prescribes.

For part iii, you may be struggling to answer the question what type of random variable it is — for good reason! ;). Its expectation is not so hard though if you’re a little bit smart about it. From part i we know what the expectation of a constant random variable is. And in Theorem 3.24 we have several properties expressing the sentiment that $\E$ considers random variables that are almost surely equal as the same thing, just pick the most convenient one from those to be done in one line!

Full solution (click to view)

For part i, observe that $X$ has range $\{c\}$ and is therefore definitely a discrete random variable as defined in Subsection 2.4.1. Applying Proposition 3.23 i, with for $h$ the identity function $h(x)=x$ (or use Equation 2.19 of course) we get that \[\E[X]=c \P(X=c)=c.\] We used that $\P(X=c)=1$, which you may consider obvious enough given the definition of $X$, but otherwise observe that the inverse image/event $\{X=c\}$ equals $\Omega$, so that by def (cf. Definition 1.50) $\P(X=c)=\P(\Omega)=1$.

For part ii, we now need to use Definition 3.22 i.e. that \[\E[X]=\int_\Omega X \dP.\]

First assume that $c \geq 0$. In order to work out the Lebesgue integral, looking back through the construction phase (non-negative step function, non-negative function, any function, cf. Subsection 3.4.1–Subsection 3.4.3), $X$ is clearly a non-negative step function as discussed in Subsection 3.4.1. Hence from Definition 3.10, again using that the range of $X$ is simply $\{c\}$, we get that \[\E[X]=\int_\Omega X \dP=c \P(X=c)=c.\]

Next assume that $c<0$. Now $X$ is no longer non-negative, and hence we’ll need to go with “any function” from Subsection 3.4.3. In particular, following Definition 3.16, we first need to determine the positive and negative parts $X^+,X^-$ of $X$. Applying their definitions from Equation 3.21, we get for any $\omega \in \Omega$ that \[X^+(\omega)=\max\{X(\omega),0\}=\max\{c,0\}=0\] and \[X^-(\omega)=\max\{-X(\omega),0\}=\max\{-c,0\}=-c.\] So both of these are constant, non-negative random variables, and hence to work out their Lebesgue integrals we can repeat the arguments from the case $c \geq 0$ and apply to Subsection 3.4.1: \[\int_\Omega X^+ \dP=0 \quad \text{and} \quad \int_\Omega X^- \dP=-c.\] Putting the pieces together, we get from Definition 3.16 that \[\E[X]=\int_\Omega X \dP=\int_\Omega X^+ \dP-\int_\Omega X^- \dP=0--c=c.\]

For part iii, note that what we know now is that $X=c$ a.s. i.e. that the inverse image/event $A:=\{X=c\}$ is an almost sure event (cf. Subsection 2.3.1 and Definition 1.55). So the picture is that $\Omega$ consists of two parts, $A$ and $A^c$, with $X(\omega)=c$ for all $\omega \in \Omega$ but we know nothing about what $X(\omega)$ is for $\omega \in A^c$! This also means that we can’t actually decide what type of random variable this is. It may be discrete, for instance if actually $A=\Omega$ or if $X$ is also constant on $A^c$ but there is no guarantee. The only thing we know for sure is that $X$ cannot be continuous, because we have that $\P(X=c)=1$ which is not possible for continuous random variables (recall from Exercise 2.10 e.g.).

What about $\E[X]$? Well we could now for instance use the non-negativity property, cf. Theorem 3.24 ii and in particular Equation 3.44: defining $X_1:=X$ and $X_2:=c$ (so a constant random variable), then we have by assumption that $X=X_2$ a.s. and it follows that $\E[X]=\E[X_2]$. But we know from part i above that $\E[X_2]=c$.

Exercise 3.6 [**] Let $X$ and $Y$ be random variables on some probability space $(\Omega,\mathcal{F},\P)$. Consider the statement \[\E[f(X)]=\E[f(Y)] \text{ for all Borel measurable functions } f: \R \to \R \tag{3.59}\] (provided both expectations exist).

Show that if $X=Y$ a.s., then Equation 3.59 holds as well.

It may be quite tempting to think that the converse implication must also be true, but that is actually not the case! Let’s investigate a little bit.

Show that if Equation 3.59 holds, then $X$ and $Y$ have the same distribution at least.
Consider the explicit example of tossing a fair coin, with $\Omega=\{T,H\}$, $\mathcal{F}$ the powerset of $\Omega$ and $\P$ determined by $\P(\{T\})=\P(\{H\})=1/2$ (recall from Example 1.54 that this works for “simple” experiments). Define the random variables $X,Y: \Omega \to \R$ as \[X(H)=0, \ X(T)=1; \quad Y(H)=1, \ Y(T)=0.\] Verify that Equation 3.59 holds but that $X=Y$ a.s. fails to hold.

Hint (click to view)

For part i, this is actually pretty straightforard if you have the discussion about how it’s fully natural to put “compound” expressions in your expectation/Lebesgue integral still on your mind, cf. Equation 3.39! From that perspective, define the random variables $U:=f(X)$ and $V:=f(Y)$. Show that $U=V$ a.s. and then go find a nice property in Theorem 3.24 that allows you to conclude equality of expectations as well.

For part ii, recall that the distribution of random variables was discussed in Definition 2.20, and that you hence need to show that $\P(X \in B)=\P(Y \in B)$ for Borel sets $B$. How to get there? An expectation becoming a probability rings one bell very loudly: the one of indicator functions, cf. Subsection 2.4.2. Fix some Borel set $B$, and try to identify a Borel measurable function $f: \R \to \R$ which yields $f(X)=\mathbf{1}_{\{ X \in B \}}$.

For part iii, this is all pretty straightforward working stuff out. Observe that $X$ and $Y$ are discrete and therefore Proposition 3.23 i can be used to work out expectations.

Full solution (click to view)

For part i, let $f: \R \to \R$ be Borel measurable. The trick is to consider the random variables $U:=f(X)$ and $V:=f(Y)$. Recall the discussion around Equation 3.39 about such “compound” expressions!

Now, note that for any $\omega \in \Omega$, if $X(\omega)=Y(\omega)$ then also \[U(\omega)=f(X(\omega))=f(Y(\omega))=V(\omega).\] In terms of inverse images/events, this means that we have that $\{X=Y\} \subseteq \{U=V\}$. Using that $\{X=Y\}$ is an almost sure event and Theorem 1.51 i, $\P(U=V) \geq \P(X=Y)=1$, so we also have that $U=V$ a.s. Now we can apply Equation 3.44 to deduce that $\E[U]=\E[V]$ i.e. that $\E[f(X)]=\E[f(Y)]$.

For part ii, recall from Definition 2.20 that $X$ and $Y$ have the same distribution if it holds that \[\P(X \in B)=\P(Y \in B) \quad \text{for all Borel sets }B\] (you could also go via their cdf’s i.e. Proposition 2.22 if you prefer, it doesn’t make any difference here). So somehow, we need to translate the assumed equality between expectations in Equation 3.59 to equality in probabilities. Which object does exactly that again? Well indeed, your loyal work horse the indicator function (cf. Subsection 2.4.2)!

Fix some Borel set $B$, and define the function $f: \R \to \R$ as $f=\mathbf{1}_B$ i.e. \[f(x)=\mathbf{1}_B(x)=\begin{cases} 1 & \text{if } x \in B \\ 0 & \text{if } x \not\in B. \end{cases} \] Observe that we then get, for any $\omega \in \Omega$, \[f(X(\omega))=\begin{cases} 1 & \text{if } X(\omega) \in B \\ 0 & \text{if } X(\omega) \not\in B \end{cases} = \mathbf{1}_{\{ X \in B \}}(\omega) \] i.e. the indicator function of the inverse image/event $\{ X \in B\}$. Similarly $f(Y)=\mathbf{1}_{\{ Y \in B \}}$.

Now, we have by assumption that $\E[f(X)]=\E[f(Y)]$, so \[\E \left[ \mathbf{1}_{\{ X \in B \}} \right]=\E \left[ \mathbf{1}_{\{ Y \in B \}} \right].\] Using Equation 2.21 (recall, all “old” expressions for expectations remain valid per Proposition 3.23) this immediately translates to $\P(X \in B)=\P(Y \in B)$.

For part iii, let $f: \R \to \R$ be any Borel measurable function. Observe that $X$ and $Y$ both have range $\{0,1\}$, which makes them discrete random variables (cf. Subsection 2.4.1) and we get from Proposition 3.23 i that \[\E[f(X)]=f(0)\P(X=0)+f(1)\P(X=1)=\frac{1}{2} \big( f(0)+f(1) \big)\] as well as \[\E[f(Y)]=f(0)\P(Y=0)+f(1)\P(Y=1)=\frac{1}{2} \big( f(0)+f(1) \big).\] So indeed $\E[f(X)]=\E[f(Y)]$, while $X(\omega) \not= Y(\omega)$ for all $\omega \in \Omega$ i.e. $\{X=Y\}=\emptyset$, the most extreme opposite of an almost sure event!

Exercise 3.7 [*/**] Let $(\Omega,\mathcal{F},\P)$ be a probability space. Recall Markov’s inequality from Theorem 3.24 v.

Prove that it indeed holds.

Hint: write the probability in Markov’s inequality also as an expectation, using a suitable indicator function. Then stare at the random variable inside each of the three expectations, and observe that they can be ordered pointwise!
Explain why the assumption in Markov’s inequality can be weakened to $X \geq 0$ a.s.

Hint (click to view)

For part i, the key point is to prove that the following inequalities hold pointwise: \[\mathbf{1}_{\{X \geq a\}} \leq \mathbf{1}_{\{X \geq a\}} \frac{1}{a} X \leq \frac{1}{a} X,\] for which you maybe want to consider $\omega \in \Omega$ that satisfy $X(\omega)<a$ and $X(\omega) \geq a$ separately.

For part ii, where in your argument in part i did you actually exactly use that $X \geq 0$, and to what extent does that argument still work under the weaker assumption that $X \geq 0$ a.s.?

Full solution (click to view)

For part i, let $X$ be a non-negative random variable and $a>0$. Recall from Equation 2.21 (recall, all “old” expressions for expectations remain valid per Proposition 3.23) that we have \[\P(X \geq a)=\E \left[ \mathbf{1}_{\{X \geq a\}} \right].\] Looking at Theorem 3.24 v, also using linearity (cf. Theorem 3.24 i), we need to prove that \[\E \left[ \mathbf{1}_{\{X \geq a\}} \right] \leq \E \left[ \mathbf{1}_{\{X \geq a\}} \frac{1}{a} X \right] \leq \E \left[ \frac{1}{a} X \right].\] Now, consider the three random variables \[\mathbf{1}_{\{X \geq a\}}, \quad \mathbf{1}_{\{X \geq a\}} \frac{1}{a} X \quad \text{and} \quad \frac{1}{a} X.\] We claim that these are actually ordered pointwise. Indeed, if $\omega \in \Omega$ satisfies $X(\omega)<a$ then \[\mathbf{1}_{\{X \geq a\}}(\omega)=0\] (by very def, recall from Subsection 2.4.2), \[\mathbf{1}_{\{X \geq a\}}(\omega) \frac{1}{a} X(\omega)=0\] and, since $X$ is non-negative \[\frac{1}{a} X(\omega) \geq 0.\] On the other hand, if $\omega \in \Omega$ satisfies $X(\omega) \geq a$ then \[\mathbf{1}_{\{X \geq a\}}(\omega)=1,\] \[\mathbf{1}_{\{X \geq a\}}(\omega) \frac{1}{a} X(\omega)=1 \cdot \frac{1}{a} X(\omega)=\frac{1}{a} X(\omega) \geq 1.\] So indeed we have pointwise \[\mathbf{1}_{\{X \geq a\}} \leq \mathbf{1}_{\{X \geq a\}} \frac{1}{a} X \leq \frac{1}{a} X,\] and applying Equation 3.43 then yields the statement (just in case you feel any confusion about Equation 3.43 requiring an almost sure inequality, recall that a pointwise inequality implies an almost sure one, see e.g. Exercise 3.3 i).

For part ii, let’s now assume that $X \geq 0$ a.s., and write $A:=\{ X geq 0\}$ which is then an almost sure event/inverse image. For any $\omega \in A$, the argument in part i above to establish the inequalities there still hold. So we get that \[\mathbf{1}_{\{X \geq a\}} \leq \mathbf{1}_{\{X \geq a\}} \frac{1}{a} X\] on the event $A$ i.e. almost surely, and that \[\mathbf{1}_{\{X \geq a\}} \frac{1}{a} X \leq \frac{1}{a} X\] on the event $A$ i.e. almost surely. Since Equation 3.43 only requires almost sure inequalities, the result remains true.

Exercise 3.8 [*/**] Comparing the conclusions from the MCT and DCT, the DCT concludes that \[X_n \stackrel{L^1}{\longrightarrow} X \text{ as } n \to \infty \tag{3.60}\] from which it then follows that \[\lim_{n \to \infty} \E[X_n]=\E[X],\] while the MCT concludes \[\lim_{n \to \infty} \E[X_n]=\E[X]\] only. Indeed, why does Equation 3.60 not always hold in the MCT setting?

Hint (click to view)

The problem is that in the MCT, we explicitly allow for the limit random variable to take the value $\infty$. In examples where that happens, Equation 3.60 can fail to hold. So try to create an example of this, for instance based on Example 3.26. Don’t forget about Definition 3.14!

Full solution (click to view)

In the MCT setting, it is possible that there exists an event $A$, with $\P(A)>0$, so that for $\omega \in A$ we have $X_n(\omega)<\infty$ for all $n=1,2,\ldots$ yet \[X(\omega):=\lim_{n \to \infty} X_n(\omega)=\infty.\] Inded see Example 3.26 for instance.

If we now look at the non-negative random variable $\lvert X_n-X \rvert$, then we have for any $\omega \in A$ and any $n=1,2,\ldots$ that \[\lvert X(\omega)-X_n(\omega) \rvert=\lvert \infty-X_n(\omega) \rvert=\infty\] and hence by Definition 3.14 we get that \[\E \left[ \lvert X_n-X \rvert \right]=\infty \quad \text{for all } n=1,2,\ldots.\] So \[\lim_{n \to \infty} \E \left[ \lvert X_n-X \rvert \right]=\infty\] and hence by def (cf. Definition 2.37), Equation 3.60 indeed does not hold.

Exercise 3.9 [**] Consider the probability space $((0,1),\mathcal{B}((0,1)),\P)$, where $\P$ is the Lebesgue measure. On this space, define the sequence of random variables $X_1, X_2, \ldots$ given by, for all $n=1,2,\ldots$ \[X_n(\omega)=\begin{cases} n^{\alpha} & \text{if } \omega \in (0,1/n) \\ 0 & \text{if } \omega \in [1/n,1), \end{cases} \] where $\alpha>0$ is a parameter.

We are interested in investigating whether this sequence has a limit, and if so whether there is convergence of expectations as well.

Why is the MCT of no use in this situation?
Show that \[X_n \stackrel{\text{a.s.}}{\longrightarrow} 0 \quad \text{as } n \to \infty.\]
By explicitly computing $\E[X_n]$, determine the limit $\lim_{n \to \infty} \E[X_n]$.
Define the random variable $Y$ by $Y(\omega)=\omega^{-\alpha}$ for all $\omega \in (0,1)$. For which parameter values $\alpha>0$ is $Y$ integrable?

Hint: Equation 3.28 from Proposition 3.19 still applies, despite the fact we’re working here on the universe $(0,1)$ rather than $[0,1]$!
Investigate for which parameter values the DCT can be applied, and how this compares to the result from part iii.
Let $\varphi: (0,1) \to \R$ be a bounded, Borel measurable function. Consider a second sequence of random variables on this space, $Y_1, Y_2, \ldots$ given by \[Y_n(\omega)=\begin{cases} \varphi(\omega) & \text{if } \omega \in (0,1/n) \\ 0 & \text{if } \omega \in [1/n,1) \end{cases} \] for all $n=1,2,\ldots$. Prove that $\lim_{n \to \infty} \E[Y_n]=0$.

Hint (click to view)

For part i, clearly the question implies that the conditions of the MCT (cf. Theorem 3.25) somehow fail to hold. So the sequence is probably not non-decreasing?

For part ii, recall the definition from Definition 2.34. The limit is already suggested/given, namely the constant random variable $X:=0$. As in Exercise 2.12 & Exercise 2.13 for instance, fix some $\omega \in (0,1)$ and investigate what $\lim_{n \to \infty} X_n(\omega)$ is. That’ll show you all you need!

For part iii, for any $n=1,2,\ldots$ fixed, observe that $X_n$ is a discrete random variable and hence computing its expectation is still as easy as it was in the past, cf. Proposition 3.23! Then when you evaluate the limit, you should three different regimes/cases: $\alpha \in (0,1)$, $\alpha=1$ and $\alpha>1$.

For part iv, recall that integrability was defined in Definition 3.22 and entails that $\E[\lvert Y \rvert]<\infty$. Using that $Y$ is non-negative, finding this expectation is a straightforward computation if you make use of the fact that $Y$ Definition 3.22 to write the expectation as a Lebesgue integral and then apply Proposition 3.19 to translate it into a Riemann integral! You should find that $Y$ is integrable if and only if $\alpha \in (0,1)$.

For part v, looking at the two conditions in the DCT (cf. Theorem 3.27), we know from part ii that the first one, the almost sure convergence, holds with as limit the constant random variable $X:=0$. So it’s all about the second condition, the existence of an integrable bound. Hmm, maybe using the random variable $Y$ from part iv for that purpose would make sense, at least for some parameter values $\alpha>0$?

For part vi, again try to employ the DCT (cf. Theorem 3.27). First find an almost sure limit random variable. Shaping your search in the same way as in Exercise 2.12 & Exercise 2.13 e.g. you should find that the constant random variable $Y:=0$ is the natural candidate limit, and then the same arguments as in part ii allow you to make this firm. Then we need to look at condition ii in the DCT again: a single random variable that forms an integrable bound. If You’re not sure, then consider using the random variable $Z:=\lvert \varphi \rvert$ for that purpose!

Full solution (click to view)

Recall that we have earlier worked on this space and looked at convergence questions, recall e.g. Example 2.36. A nice aspect is that random variables are here (also, just) functions from $(0,1)$ to $\R$ and hence you can sketch their graphs to investigate what’s going on, as in Figure 2.10 e.g.!

For part i, the MCT (cf. Theorem 3.25) requires a non-decreasing sequence of random variables. However that’s not the case here, due to the “shrinking” of the intervals $(0,1/n)$ as $n$ grows. Indeed for any $n=1,2,\ldots$ fixed, for $\omega \in [1/(n+1),1/n)$ we have that \[X_{n+1}(\omega)-X_n(\omega)=0-n^\alpha<0.\]

For part ii, recall from Definition 2.34 that we need to show that there exists an almost sure event $\Omega_0$ so that \[\lim_{n \to \infty} X_n(\omega)=0 \quad \text{for all } \omega \in \Omega_0. \tag{3.61}\] For this, have a good stare at the universe $(0,1)$ & hence the domain of our $X_n$’s, and how these guys behave on it (as pointed out at the start of this solution, sketching some graphs also goes a long way!). Note that the interval $[1/n,1)$ gradually “grows” to ultimately fill the whole universe $(0,1)$, and clearly our argument should be based on that. In fact there are no points in $(0,1)$ that escape getting swallowed up by the $[1/n,1)$ eventually, so we claim that Equation 3.61 even holds with $\Omega_0=(0,1)$ i.e. that we have in fact pointwise convergence (cf. Definition 2.33)!

Indeed, fix any $\omega \in (0,1)$. Then, since $\omega>0$, there exists an $N$ so that $1/n \leq \omega$ and hence $X_n(\omega)=0$ for all $n \geq N$. It follows that $\lim_{n \to \infty} X_n(\omega)=0$. So indeed Equation 3.61 holds.

For part iii, for any $n=1,2,\ldots$ fixed, observe that $X_n$ has range $\{0,n^\alpha\}$ and is therefore a discrete random variable (cf. Subsection 2.4.1). Hence we can apply Equation 2.19 (via Proposition 3.23) to compute \[\E[X_n]=0 \cdot \P(X=0)+n^\alpha \P \left( X=n^\alpha \right).\] Clearly the inverse image/event $\{X=n^\alpha\}$ equals the interval $(0,1/n)$, so that \[\P \left( X=n^\alpha \right)=\P \left( \left( 0,\frac{1}{n} \right) \right)=\frac{1}{n}\] (for the final step, recall Proposition 1.56) and hence $\E[X_n]=n^{\alpha-1}$. (Alternatively, you could recognise that this is actually an indicator function and immediately jump to Equation 2.21!).

If we now consider the limit for $n$ to $\infty$, we see three cases appearing:

if $\alpha \in (0,1)$, then $\lim_{n \to \infty} \E[X_n]=0$;
if $\alpha=1$, then $\lim_{n \to \infty} \E[X_n]=1$;
if $\alpha>1$, then $\lim_{n \to \infty} \E[X_n]=\infty$.

For part iv, recall that integrability was defined in Definition 3.22 and entails that $\E[\lvert Y \rvert]<\infty$. Since $Y$ is non-negative, we have that $\E[\lvert Y \rvert]=\E[Y]$, and by def (cf. Definition 3.22) \[\E[Y]=\int_{(0,1)} Y \dP.\] Now recall that we chose $\P$ to be the Lebesgue measure, and following the hint given in the question given we can still apply Proposition 3.19 to write this as a good old Riemann integral: \[\E[Y]=\int_{(0,1)} Y \dP=\int_0^1 Y(\omega) \d\omega=\int_0^1 \omega^{-\alpha} \d\omega\] (just to avoid any possible confusion: recall that the use of “$\omega$” in the Riemann integral has no particular meaning, it’s a dummy variable, cf. Remark 1.5 — use whatever symbol you prefer!).

Working out the Riemann integral we see that for $\alpha \in (0,1)$ we get $\E[Y]=1/(1-\alpha)$ while for $\alpha \geq 1$ we get $\E[Y]=\infty$. So we can conclude that $Y$ is integrable if and only if $\alpha \in (0,1)$.

For part v, looking at both the conditions in the DCT (cf. Theorem 3.27), we see that the almost sure convergence indeed holds (as seen in part ii), to the constant random variable $X:=0$. The other condition is that an integrable random variable $Z$ (using a different symbol, since $Y$ is already in use) exists so that $\lvert X_n \rvert \leq Z$ a.s. for all $n=1,2,\ldots$.

Now, there is obviously an implicit yet rather shouty hint in the fact that we were asked to do part iv: does the random variable $Y$ from there work to satisfy that second condition maybe?? Fix some $n=1,2,\ldots$. Since $X_n$ is non-negative, $\lvert X_n \rvert=X_n$. Further we claim that $X_n \leq Y$ indeed holds, even pointwise. To see this, not that for any $\omega \in (0,1/n)$ we have that \[Y(\omega) \geq Y \left( \frac{1}{n} \right)=\left( \frac{1}{n} \right)^{-\alpha}=n^\alpha=X_n(\omega)\] (just using that $Y$ is a decreasing function etc. — can always make a sketch of their graphs!) while for any $\omega \in [1/n,1)$ \[Y(\omega) \geq 0=X_n(\omega).\]

So, the only remaining condition to check is that $Y$ is integrable. This brings us to:

if $\alpha \in (0,1)$, then part iv tells us that $Y$ is integrable and hence it follows from the DCT that \[\lim_{n \to \infty} \E[X_n] = \E[X]=\E[0]=0,\] the final step by Exercise 3.5. Indeed, consistent with the comp we did in part iii!
if $\alpha \geq 1$, then part iv tells us that $Y$ is not integrable. So $Y$ cannot play the role of the integrable bound $Z$ in the DCT. Is there some other bound we could use instead? Maybe. Or, actually, no there cannot exist an integrable bound! Because if it did, then the DCT would apply and it would again follow that \[\lim_{n \to \infty} \E[X_n] = 0.\] But we know from part iii that this is not true!

Nice example I think of how the integrable bound in the DCT is indeed really necessary to ensure that the conclusion of the DCT holds! :).

For part vi, let’s try to apply the DCT (cf. Theorem 3.27) to this case. First we need to find an almost sure limit random variable of this sequence. You can readily check that the exact same argument in part ii applies and shows that we have \[Y_n \stackrel{\text{a.s.}}{\longrightarrow} Y \quad \text{as } n \to \infty.\] where $Y:=0$.

Now for condition ii in Theorem 3.27: finding an integrable bound. The crucial benefit we have here is that the function $\varphi$, despite that we know very little about it, is bounded (pointwise, because no qualifier present). That is, there exists some $C>0$ so that $\lvert \varphi(\omega) \rvert \leq C$ for all $\omega \in (0,1)$. Recall that in condition ii in Theorem 3.27 we need one single bound i.e. it is not allowed to depend on $n$. If you stare at this for a moment, then you’ll realise that the obvious choice is to try $Z:=\lvert \varphi \rvert$ i.e. the random variable $X$ defined as $Z(\omega)=\lvert \varphi(\omega) \rvert$ for all $\omega \in (0,1)$. To verify that this choice indeed works:

$\lvert Y_n \rvert \leq Z$ a.s. for all $n=1,2,\ldots$: fix some $n=1,2,\ldots$. You can readily check that for any $\omega \in (0,1)$, $\lvert Y_n(\omega) \rvert \leq Z(\omega)$ (for instance by considering $\omega \in (0,1/n)$ and $\omega \in [1/n,1)$ separately again). So the inequality holds even pointwise, and hence also almost surely (cf. Exercise 3.3 i).
$Z$ is integrable, i.e. (cf. Definition 3.22) $\E[\lvert Z \rvert]<\infty$ or equivalently $\E[Z]<\infty$ as $Z$ is non-negative: since $\lvert \varphi(\omega) \rvert \leq C$ for all $\omega \in (0,1)$ i.e. $Z \leq C$ pointwise and hence also almost surely, using the non-negativity property (cf. Theorem 3.24 ii, in particular Equation 3.43) as well as Exercise 3.5, we see that indeed $\E[Z] \leq \E[C]=C<\infty$. (Alternatively you could go via the Riemann integral again, as in part iii, and make use of the boundedness of $\varphi$ in that Riemann integral.)

So all conditions in the DCT have now been ticked off and we can conclude that \[\lim_{n \to \infty} \E[Y_n] = \E[Y]=\E[0]=0,\] the final step by Exercise 3.5 again.

Exercise 3.10 [*/**] On some probability space $(\Omega,\mathcal{F},\P)$, let $X_1 \geq X_2 \geq \ldots$ be a non-increasing sequence of non-negative random variables with the property that $\lim_{n \to \infty} X_n=0$ as $n \to \infty$ (all of this pointwise).

Suppose that $\E[X_1]<\infty$. Using the DCT, show that \[\lim_{n \to \infty} \E[X_n]=0.\]
In general, i.e. without making any further assumptions, what can you say about $\lim_{n \to \infty} \E[X_n]$?

Note: this has a nice link with the (non-examinable) discussion in Section 3.1, as it confirms that within the class of integrable random variables/functions, the Lebesgue integral also satisfies the remaining outstanding desirable/defining property of an integral that we hadn’t seen back yet, namely desirable/defining property iii in Definition 3.1 (the other two defining properties appear in Theorem 3.24)!

Hint (click to view)

For part i, checking the conditions of the DCT (cf. Theorem 3.27), the only real question is what we should be choosing as our integrable bound (condition ii in the DCT). If you look at the given properties of the sequence, it kind of stares you in the face that $X_1$ is the “largest” one (also in absoulte value). So why not pick that one? Yes of course you can, nobody ever said you couldn’t use a particular element from the sequence! As long as it is only one of them, so that you don’t introduce some dependence on $n$ in your choice which is of course Verboten. ;).

For part ii, first convince yourself that from Definition 3.22 (and/or construction of the Lebesgue integral) we have that $\E[X_n] \in [0,\infty]=[0,\infty) \cup \{\infty\}$ for all $n=1,2,\ldots$. If you think about it for a moment, then this gives rise to two possible cases:

$\E[X_n]=\infty$ for all $n=1,2,\ldots$,
$\E[X_n]<\infty$ for some $n=1,2,\ldots$.

In both cases, you can conclude what the limit $\lim_{n \to \infty} \E[X_n]$ is in one or two lines only!

Full solution (click to view)

For part i, setting $X=0$, we’re given that $X_n \to X$ as $n \to \infty$ pointwise and hence also almost surely (as we also used several times in Exercise 3.9 above). To apply the DCT (cf. Theorem 3.27), it remains to identify an integrable random variable $Y$ so that $\lvert X_n \rvert \leq Y$ a.s. for all $n=1,2,\ldots$. Note that as the sequence is conveniently non-increasing, it’s first element is “largest”. So why not try that one to bound all of them with? Indeed, set $Y:=X_1$. Then for any $n=1,2,\ldots$, also using that $X_n \geq 0$ \[\lvert X_n \rvert = X_n \leq X_1=Y\] (all pointwise and hence also almost surely). Further we need to check that $Y$ is integrable i.e. that $\E[\lvert Y \rvert]<\infty$ (cf. Definition 3.22). But this also follows readily, using that $Y=X_1$ is non-negative: \[\E[\lvert Y \rvert]=\E[Y]=\E[X_1]<\infty,\] by assumption.

So we can apply the DCT and conclude that indeed \[\lim_{n \to \infty} \E[X_n]=\E[X]=\E[0]=0,\] the final step by Exercise 3.5 (yet again).

For part ii, let’s investigate what we can say about $\E[X_n]$ from the info we’re given. Since $X_n$ is non-negative, we know from Definition 3.22 (going back to Definition 3.12) that $\E[X_n]$ definitely exists and that $\E[X_n] \in [0,\infty]=[0,\infty) \cup \{\infty\}$. Further, an application of Equation 3.43 shows that the sequence $\E[X_1], \E[X_2], \ldots$ is non-increasing sequence of real numbers (and/or infinities).

This gives rise to two possible cases:

if $\E[X_n]=\infty$ for all $n=1,2,\ldots$, then clearly \[\lim_{n \to \infty} \E[X_n]=\infty.\]
otherwise, if there exists an $n=1,2,\ldots$ for which $\E[X_n]<\infty$, then we are essentially back in the situation of part i. Indeed a limit is of course only about the “tail” of a sequence and we can just throw away the first however many terms we want! That is, if we define a new sequence $Y_1, Y_2, \ldots$ with $Y_1:=X_n$, $Y_2:=X_{n+1}, \ldots$ then it holds that \[\lim_{n \to \infty} \E[X_n]=\lim_{n \to \infty} \E[Y_n].\] And we can apply the conclusion of part i to the sequence $Y_1, Y_2, \ldots$, which yields \[\lim_{n \to \infty} \E[X_n]=0.\]

So only two extremes are possible: either the limit is zero or the limit is infinity!

Exercise 3.11 [**/***] On some probability space $(\Omega,\mathcal{F},\P)$, let $X_1, X_2, \ldots$ be a sequence of non-negative random variables, and let $A_1, A_2, \ldots$ be a sequence of events.

For the below, recall that an infinite sum is defined as the limit of finite/partial sums which may or may not exist, i.e. for a sequence $a_1, a_2, \ldots$ of real numbers, \[\sum_{k=1}^\infty a_k := \lim_{n \to \infty} \sum_{k=1}^n a_k \tag{3.62}\] provided that the limit indeed exists. Note that if $a_1, a_2, \ldots$ are non-negative, then these partial sums form a non-decreasing sequence and the limit is guaranteed to exist, though it may be $\infty$.

Define the random variable \[Y(\omega)=\sum_{k=1}^\infty X_k(\omega) \quad \text{for all } \omega \in \Omega.\] Using the MCT, show that \[\E [Y] = \sum_{k=1}^\infty \E[X_k].\]

Hint: although the linearity property in Theorem 3.24 is formulated for the sum of two terms only, it does automatically extend to any (finite!) number of terms — just apply the property multiple times. Note that it’s not a given that it also extends to infinitely many terms, and that’s exactly why there is something to prove here!
With $\#B$ denoting the number of elements in a set $B$, define the random variable \[N(\omega)=\#\{ i \in \{1,2,\ldots\} \, | \, \omega \in A_i \} \quad \text{for all } \omega \in \Omega\] i.e. the number of events that occur if the experiment is performed and the outcome is $\omega$. Show that \[\E[N]=\sum_{k=1}^\infty \P(A_k).\]

Hint (click to view)

For part i, we’re asked to prove an expression for $\E[Y]$ using the MCT (cf. Theorem 3.25). So we’ll need to come up with a sequence of random variables $Y_1, Y_2, \ldots$ that has $Y$ as their pointwise limit. Can you think of one? What if you have another look at Equation 3.62??

For part ii, there is one key trick this question, and that is to realise that we can write \[N=\sum_{i=1}^\infty \mathbf{1}_{A_i},\] using the indicator functions of the events $A_1, A_2, \ldots$ (cf. Subsection 2.4.2). Uisng part i, the result then follows readily. But how do we justify this expression? Well just imagine executing the experiment, and that we observe outcome $\omega$. What does the value $\mathbf{1}_{A_i}(\omega)$ have to do with whether or not the event $A_i$ happens for this outcome??

Full solution (click to view)

For part i, we’re asked to prove an expression for $\E[Y]$ using the MCT (cf. Theorem 3.25). But we don’t even have a sequence of random variables! So we’ll need to invent one, and ideally one that actually helps solving the question. ;). That means that we’re looking for a nice sequence that has $Y$ as its limit. Given Equation 3.62, let’s set for all $n=1,2,\ldots$ and $\omega \in \Omega$ \[Y_n(\omega)=\sum_{k=1}^n X_k(\omega).\] Then indeed $0 \leq Y_1 \leq Y_2 \leq \ldots$ (pointwise) since the $X_n$’s are non-negative with pointwise limit $Y$ (by Equation 3.62). So the MCT yields that \[\E[Y]=\lim_{n \to \infty} \E[Y_n].\] Further, following the hint given in the question, we get from the linearity property Theorem 3.24 i that \[\E[Y_n]=\sum_{k=1}^n \E[X_k]\] and hence \[\E[Y]=\lim_{n \to \infty} \sum_{k=1}^n \E[X_k]=\sum_{k=1}^\infty \E[X_k],\] the final one by Equation 3.62.

For part ii, the trick is to realise that we can write, using the indicator functions (cf. Subsection 2.4.2) of the events $A_1, A_2, \ldots$ \[N=\sum_{i=1}^\infty \mathbf{1}_{A_i}.\] To see this, imagine that the experiment gets performed and we observe an outcome $\omega$. Then in the whole sequence of indicator functions $\mathbf{1}_{A_1}(\omega), \mathbf{1}_{A_2}(\omega), \ldots$ is a sequence of $0$’s and $1$’s, with a $1$ for each event that occurs. Hence to count the number of events that occur, we need to count the number of $1$’s that we have in that sequence. But, indeed, the number of $1$’s is exactly equal to the sum of the whole sequence!

Now we can apply the result of part i to get \[\E[N]=\sum_{i=1}^\infty \E \left[ \mathbf{1}_{A_i} \right],\] and it only remains to plug in Equation 2.21 (which we’re still allowed to use, recall from Proposition 3.23).

Exercise 3.12 [***] Have another look at the statement of the MCT. It requires that your sequence $X_1, X_2, \ldots$ of random variables on $(\Omega,\mathcal{F},\P)$ has the property that $0 \leq X_1 \leq X_2 \leq \ldots$, where each inequality is pointwise. Consider the following two alternative conditions:

$0 \leq X_1 \leq X_2 \leq \ldots$ holds a.s.,
each inequality $0 \leq X_1$, $X_1 \leq X_2$, $\ldots$ holds a.s.

Show that both conditions are in fact equivalent.
Show that if condition 1 holds, then the (appropriately slightly adjusted) conclusion of the MCT still holds.

Note: this one is quite tricky I think, especially part ii — god speed you courageous warrior!

Hint (click to view)

For part i, first carefully write out both conditions (see Subsection 2.3.1). Then you maybe already see some light at the end of the tunnel. One of the two implications is pretty immediate. For the other, never forget that Exercise 1.12 is your loyal friend in such trying times!

For part ii, I don’t want to give too much away for those of you who relish the challenge. First of all, explicitly construct a random variable $X$ so that \[X_n \stackrel{\text{a.s.}}{\longrightarrow} X \quad \text{as } n \to \infty.\] Then the battle is to try and show that \[\lim_{n \to \infty} \E[X_n]=\E[X].\] For this, try to find a way to adjust your sequence of random variables $X_1, X_2, \ldots$ a little bit so that you can still make use of the MCT (the “original” one i.e. as formulated in Theorem 3.25)!

Full solution (click to view)

For part i, note that if we write out the two conditions, then they look as follows (consult Subsection 2.3.1 if needs be):

There exists an almost sure event $A$ so that \[0 \leq X_1(\omega) \leq X_2(\omega) \leq \ldots \quad \text{for all } \omega \in A.\]
There exist almost sure events $A_1, A_2, \ldots$ so that \[\begin{align*} 0 \leq X_1(\omega) &\quad \text{for all } \omega \in A_1, \\ X_1(\omega) \leq X_2(\omega) &\quad \text{for all } \omega \in A_2, \\ \vdots \qquad & \qquad \vdots \end{align*} \]

First assume that condition 1 holds. Then trivially, for any $\omega \in A$, since \[0 \leq X_1(\omega) \leq X_2(\omega) \leq \ldots\] also each of the inequalities $0 \leq X_1(\omega)$, $X_1(\omega) \leq X_2(\omega)$, $\ldots$ hold. So condition 2 also holds, with the choices $A_1:=A$, $A_2:=A$, $\ldots$ which are obviously all almost sure events since $A$ is.

Now assume that condition 2 holds. Define the event \[A=\bigcap_{i=1}^\infty A_i.\] Fix some $\omega \in A$. Then $\omega \in A_i$ for each $i=1,2,\ldots$ (cf. Equation 1.5). Hence the inequalities $0 \leq X_1(\omega)$, $X_1(\omega) \leq X_2(\omega)$, $\ldots$ all hold, and therefore also $0 \leq X_1(\omega) \leq X_2(\omega) \ldots.$ So it remains to show that $A$ is an almost sure event. But for this we can reach back to our Swiss Army knife that is Exercise 1.12.

For part ii, assume that condition 1 holds, i.e. that there exists an almost sure event $A$ so that \[0 \leq X_1(\omega) \leq X_2(\omega) \leq \ldots \quad \text{for all } \omega \in A.\] Let’s first investigate what the limit situation looks like. Fix an $\omega \in A$. Then we’re looking at a non-negative, non-decreasing sequence of real numbers (with possible infinities in there if we allow for that) which is guaranteed to have a limit, in $[0,\infty]:=[0,\infty) \cup \{\infty\}$.

So define the mapping $X:\Omega \to \R \cup \{\infty\}$ as \[X(\omega)=\begin{cases} \lim_{n \to \infty} X_n(\omega) & \text{if } \omega \in A \\ 0 & \text{if } \omega \not\in A, \end{cases} \] where the value $0$ on $A^c$ is an arbitrary choice. To upgrade $X$ to a random variable it needs to be measurable (cf. Definition 2.13), but that is covered by Proposition 2.11 v.

Ok, that sorts the limit situation. Observe that we have from Definition 2.34 that \[X_n \stackrel{\text{a.s.}}{\longrightarrow} X \quad \text{as } n \to \infty. \tag{3.63}\] That gives us some peace of mind as well, because obviously this limit $X$ is now no longer unique (for starters, we could have chosen any other value than $0$ on $A^c$) but we do know from Exercise 2.14 that any other a.s. limit of $X_1, X_2, \ldots$ is a.s. equal to $X$.

Now, we’re also asked to show that the conclusion of the MCT still holds, i.e. that \[\lim_{n \to \infty} \E[X_n]=\E[X]. \tag{3.64}\] Bwah… The only tools we have for this kind of shizzle are the MCT and DCT. We could look at the DCT, after all Equation 3.63 is the convergence mode the DCT relies on, but then we’d have to impose extra restrictions on our sequence to ensure that we have an integrable bound and stuff — not optimal. So that leaves the MCT — I mean the original MCT as formulated in Theorem 3.25. But the conditions there are not satisfied either?? No that’s right, but we’re actually close enough with a bit of smarts!

Here’s the idea. Define a new sequence of random variables, say $Y_1, Y_2, \ldots$ given by \[Y_n(\omega)=\begin{cases} X_n(\omega) & \text{if } \omega \in A \\ 0 & \text{if } \omega \not\in A \end{cases} \] for all $n=1,2,\ldots$ (for the measurability question, given that we’re trying to be ultra precise here, observe that we can also write $Y_n=\mathbf{1}_A X_n$ i.e. the product of two mappings that we know to be random variables, and hence we are safe by Proposition 2.11 iv).

Observe that for any $\omega \in \Omega$, we have that $Y_n(\omega) \to X(\omega)$ i.e. $Y_n \to X$ as $n \to \infty$ pointwise. Further clearly $0 \leq Y_1 \leq Y_2 \leq \ldots$ pointwise. This means that we can now apply the “original” MCT i.e. Theorem 3.25 to conclude that \[\lim_{n \to \infty} \E[Y_n]=\E[X].\] And since $Y_n(\omega)=X_n(\omega)$ for all $\omega \in A$, we have that $Y_n=X_n$ a.s. and hence also $\E[Y_n]=\E[X_n]$ (cf. Equation 3.44). Subbing that in, we indeed arrive at Equation 3.64!